Spark process failing to receive data from the Kafka queue in yarn-client mode -


i trying run following code using yarn-client mode in getting slow readprocessor error mentioned below code works fine in local mode. pointer appreciated.

line of code receive data kafka queue:

javapairreceiverinputdstream<string, string> messages =  kafkautils.createstream(jssc, string.class, string.class, stringdecoder.class, stringdecoder.class, kafkaparams, kafkatopicmap, storagelevel.memory_only());  javadstream<string> lines = messages.map(new function<tuple2<string, string>, string>() {       public string call(tuple2<string, string> tuple2) {                               log.info(" &&&&&&&&&&&&&&&&&&&& input json stream data  " +  tuple2._2);         return tuple2._2();       }     }); 

error details:

016-02-05 11:44:00 warn dfsclient:975 - slow readprocessor read fields took 30 011ms (threshold=30000ms); ack: seqno: 1960 reply: 0 reply: 0 reply: 0 downstrea macktimenanos: 1227280, targets: [datanodeinfowithstorage[10.0.0.245:50010,ds-a5 5d9212-3771-4936-bbe7-02035e7de148,disk], datanodeinfowithstorage[10.0.0.243:500 10,ds-231b9915-c2e2-4392-b075-8a52ba1820ac,disk], datanodeinfowithstorage[10.0.0 .244:50010,ds-6b8b5814-7dd7-4315-847c-b73bd375af0e,disk]] 2016-02-05 11:44:00 info blockmanager:59 - removing rdd 1954 2016-02-05 11:44:00 info mappartitionsrdd:59 - removing rdd 1955 persisten 


Comments

Popular posts from this blog

sublimetext3 - what keyboard shortcut is to comment/uncomment for this script tag in sublime -

java - No use of nillable="0" in SOAP Webservice -

ubuntu - Laravel 5.2 quickstart guide gives Not Found Error -