lundi 29 juin 2015

Task not serializable exception in spark while used in JEE project

Following code is working fine when used in java console application. A and C classes both are serialable. When same code is used in JEE aRDD and bRDD are working good but performing the flatMap transformation on aRDD is throwing Task not serialable exception.

            JavaPairRDD<String,Iterable<A>> aRDD= CassandraJavaUtil.javaFunctions(sc).
                    cassandraTable("testing", "cf_taxlots",CassandraJavaUtil.mapRowTo(DailyTaxlot.class, colMap)).
                    where("openclosetag=? and rundate IN ?","Open",dates).groupBy(row -> row.getTaxlotId());

            JavaPairRDD<String,Iterable<A>> bRDD = CassandraJavaUtil.javaFunctions(sc).
                    cassandraTable("testing", "cf_taxlots",CassandraJavaUtil.mapRowTo(DailyTaxlot.class, colMap)).
                    where("openclosetag=? and rundate IN ?","Close",dateRange).groupBy(row -> row.getTaxlotId());

            JavaRDD<C> finalRDD = aRDD.flatMap( new FlatMapFunction<Tuple2<String,Iterable<A>>, C>() {
                public Iterable<C> call(Tuple2<String,Iterable<C>> tuple) throws Exception
                { ....... });

What is wrong with flatMap transformation on aRDD? Why its working on the java console application? Any pointers??

Aucun commentaire:

Enregistrer un commentaire