python - saving pyspark rdd to hbase raises attribute error -


i trying write spark rdd using pyspark hbase table. rdd looks following using print rdd.take(rdd.count()) command 

[decimal('0.39326837'), decimal('0.03643601'), decimal('0.06031798'), decimal('0.08885452')]

when try write rdd hbase table using function saverecord 

def saverecord(tx_fee_rdd):       host = 'localhost'  #sys.argv[1]       table = 'tx_fee_table'  #needs created before hand in hbase shell       conf = {"hbase.zookeeper.quorum": host,             "hbase.mapred.outputtable": table,             "mapreduce.outputformat.class": "org.apache.hadoop.hbase.mapreduce.tableoutputformat",             "mapreduce.job.output.key.class": "org.apache.hadoop.hbase.io.immutablebyteswritable",             "mapreduce.job.output.value.class": "org.apache.hadoop.io.writable"}     keyconv = "org.apache.spark.examples.pythonconverters.stringtoimmutablebyteswritableconverter"     valueconv = "org.apache.spark.examples.pythonconverters.stringlisttoputconverter"     #row key id,id, cfamily=tx_fee_col,column_name = tx_fee, column_value=x      datamap = tx_fee_rdd.map(lambda x: ("tx_fee_col","tx_fee",x ) )       datamap.saveasnewapihadoopdataset(conf=conf,keyconverter=keyconv,valueconverter=valueconv)     tx_fee_rdd.foreach(saverecord)

i following error

attributeerror: 'decimal' object has no attribute 'map'

how deal ?

following @zeros323 suggestions, getting following error

traceback (most recent call last):   file "/home/ubuntu/unix_practice/bcrpc/bitcoin-inspector-webserver/bitcoin/bctxfee_text3.py", line 66, in <module>     saverecord(tx_fee_rdd)   file "/home/ubuntu/unix_practice/bcrpc/bitcoin-inspector-webserver/bitcoin/bctxfee_text3.py", line 29, in saverecord     datamap.saveasnewapihadoopdataset(conf=conf,keyconverter=keyconv,valueconverter=valueconv)     file "/usr/local/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1348, in saveasnewapihadoopdataset   file "/usr/local/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__   file "/usr/local/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value py4j.protocol.py4jjavaerror: error occurred while calling z:org.apache.spark.api.python.pythonrdd.saveashadoopdataset. : org.apache.spark.sparkexception: rdd element of type [ljava.lang.object; cannot used     @ org.apache.spark.api.python.serdeutil$.pythontopairrdd(serdeutil.scala:237)     @ org.apache.spark.api.python.pythonrdd$.saveashadoopdataset(pythonrdd.scala:801)     @ org.apache.spark.api.python.pythonrdd.saveashadoopdataset(pythonrdd.scala)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:498)     @ py4j.reflection.methodinvoker.invoke(methodinvoker.java:231)     @ py4j.reflection.reflectionengine.invoke(reflectionengine.java:379)     @ py4j.gateway.invoke(gateway.java:259)     @ py4j.commands.abstractcommand.invokemethod(abstractcommand.java:133)     @ py4j.commands.callcommand.execute(callcommand.java:79)     @ py4j.gatewayconnection.run(gatewayconnection.java:207)     @ java.lang.thread.run(thread.java:745)

foreach operates on individual records hence receives decimal objects not rdds. cannot map on these not mention use saveasnewapihadoopdataset method.

if want use saveasnewapihadoopdataset function should operate directly on rdd:

saverecord(tx_fee_rdd)

another possible issue following part:

datamap = tx_fee_rdd.map(lambda x: ("tx_fee_col","tx_fee",x ) )

saveasnewapihadoopdataset expect pairs not triplets. may not work decimal objects. see hbase_outputformat.py example details.


Comments

Post a Comment

Popular posts from this blog

sublimetext3 - what keyboard shortcut is to comment/uncomment for this script tag in sublime -

Image
i have script tag in code, keyboard shortcut comment/uncomment script tag in sublime. after selecting script open , close tag. <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> i pressing ctrl + shift + / , getting this /*<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script>*/ also pressed ctrl + / , got this // <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> // </script> instead of <!-- <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> --> script open , close tag in 2 lines not in 1 line , not have spaces too. for mac,high light lines want comment. hold 'command , / ' commented. for windows: high line or select lines ,then hold ctrl + / if not work: ...
Read more

post - imageshack API cURL -

i credentials correct. when send image through form this:(action="http://api.imageshack.com/v2/images/"). working fine when tried send $ch = curl_init($url); failed. code: enter code here <form action="" method="post" enctype="multipart/form-data"> select image upload: <input type="file" name="file" id="file" multiple> <input type="text" name="auth_token" id="auth_token" value="xxxxxxxxxxxxxxxx"> <input type="text" name="api_key" id="api_key" value="xxxxxxxxxxxxxxx"> <input type="submit" value="upload image" > </form> $url = ' http://api.imageshack.com/v2/images/ '; $temp = $_files["file"]['tmp_name']; echo $temp."<br>" ; $ch = curl_init($url); curl_setopt($ch, curlopt_post, true); curl...
Read more

dataset - MPAndroidchart returning no chart Data available -

i having problems mpandroidchart. adding temperature , humidity data chart, displays "no chart data available". here can see bug is? super grateful! //create data //set size of data int size = list.size(); if (size > 24){ size = 24;} //create lists form temp , hum arraylist<entry> temps = new arraylist<entry>(); arraylist<entry> hums = new arraylist<entry>(); //fill list temp values (int = 0; < size; i++){ entry value = new entry(math.round(list.get(list.size()-size + i).temp), (size-i)); temps.add(value); } //fill list hum values (int = 0; < size; i++){ entry value = new entry(math.round(list.get(list.size()-size + i).hum), (size-i)); hums.add(value); } ...
Read more