Spark process failing to receive data from the Kafka queue in yarn-client mode -
i trying run following code using yarn-client mode in getting slow readprocessor error mentioned below code works fine in local mode. pointer appreciated.
line of code receive data kafka queue:
javapairreceiverinputdstream<string, string> messages = kafkautils.createstream(jssc, string.class, string.class, stringdecoder.class, stringdecoder.class, kafkaparams, kafkatopicmap, storagelevel.memory_only()); javadstream<string> lines = messages.map(new function<tuple2<string, string>, string>() { public string call(tuple2<string, string> tuple2) { log.info(" &&&&&&&&&&&&&&&&&&&& input json stream data " + tuple2._2); return tuple2._2(); } });
error details:
016-02-05 11:44:00 warn dfsclient:975 - slow readprocessor read fields took 30 011ms (threshold=30000ms); ack: seqno: 1960 reply: 0 reply: 0 reply: 0 downstrea macktimenanos: 1227280, targets: [datanodeinfowithstorage[10.0.0.245:50010,ds-a5 5d9212-3771-4936-bbe7-02035e7de148,disk], datanodeinfowithstorage[10.0.0.243:500 10,ds-231b9915-c2e2-4392-b075-8a52ba1820ac,disk], datanodeinfowithstorage[10.0.0 .244:50010,ds-6b8b5814-7dd7-4315-847c-b73bd375af0e,disk]] 2016-02-05 11:44:00 info blockmanager:59 - removing rdd 1954 2016-02-05 11:44:00 info mappartitionsrdd:59 - removing rdd 1955 persisten
Comments
Post a Comment