scala - Broadcast not happening while joining dataframes in Spark 1.6 -


below sample code running. when spark job runs, dataframe joins happening using sortmergejoin instead of broadcastjoin.

def joineddf (sqlcontext: sqlcontext,               txntable:   dataframe,               countriesdfbroadcast: broadcast[dataframe]):                dataframe = {                     txntable.as("df1").join((countriesdfbroadcast.value).withcolumnrenamed("cntry_id", "dw_cntry_id").as("countries"),                     $"df1.user_cntry_id" === $"countries.dw_cntry_id", "inner")               } joineddf(sqlcontext, txntable, countriesdfbroadcast).write.parquet("temp")

the broadcastjoin not happening when specify broadcast() hint in join statement.

the optimizer hashpartitioning dataframe , causing data skew.

has seen behavior?

i running on yarn using spark 1.6 , hivecontext sqlcontext. spark job runs on 200 executors. , data size of txntable 240gb , datasize of countriesdf 5mb.

both way how broadcast dataframe , how access incorrect.

  • standard broadcasts cannot used handle distributed data structures. if want perform broadcast join on dataframe should use broadcast functions marks given dataframe broadcasting:

    import org.apache.spark.sql.functions.broadcast  val countriesdf: dataframe = ??? val tmp: dataframe = broadcast(   countriesdf.withcolumnrenamed("cntry_id", "dw_cntry_id").as("countries") )   txntable.as("df1").join(   broadcast(tmp), $"df1.user_cntry_id" === $"countries.dw_cntry_id", "inner")

    internally collect tmp without converting internal , broadcast afterwards.

  • join arguments eagerly evaluated. possible use sparkcontext.broadcast distributed data structure broadcast value evaluated locally before join called. thats' why function work @ doesn't perform broadcast join.


Comments

Post a Comment

Popular posts from this blog

sublimetext3 - what keyboard shortcut is to comment/uncomment for this script tag in sublime -

Image
i have script tag in code, keyboard shortcut comment/uncomment script tag in sublime. after selecting script open , close tag. <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> i pressing ctrl + shift + / , getting this /*<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script>*/ also pressed ctrl + / , got this // <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> // </script> instead of <!-- <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> --> script open , close tag in 2 lines not in 1 line , not have spaces too. for mac,high light lines want comment. hold 'command , / ' commented. for windows: high line or select lines ,then hold ctrl + / if not work: ...
Read more

java - No use of nillable="0" in SOAP Webservice -

i have received xsd 3rd party supplier generated java based system; used create soap endpoint receive data transfers. xsd not make use of nillable attribute defined within w3c/xsd namespace, example: <xs:complextype name="customer"> <xs:sequence> <xs:element minoccurs="1" maxoccurs="1" name="id" type="xs:integer"/> <xs:element minoccurs="0" maxoccurs="1" name="titleid" type="xs:integer"/> <xs:element minoccurs="0" maxoccurs="1" name="firstname" type="xs:string"/> <xs:element minoccurs="0" maxoccurs="1" name="familyname" type="xs:string"/> <xs:element minoccurs="0" maxoccurs="1" name="previousname" type="xs:string"/> <xs:element minoccurs="0" maxoccurs="1" nam...
Read more

ubuntu - Laravel 5.2 quickstart guide gives Not Found Error -

edit: think problem due not setting apache properly, these 2 links helped me. solution: http://laravel.io/forum/06-11-2014-not-found-but-route-exists https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-14-04-lts end edit i'm following laravel 5.2 quickstart guide. https://laravel.com/docs/5.2/quickstart i'm running ubuntu 14.04.2 the website runs initially, when click on add task button run 404 not found error. as per guide run following commands setup complete quickstart project. git clone https://github.com/laravel/quickstart-basic quickstart cd quickstart composer install php artisan migrate additionally run following commands because i'm on ubuntu server: sudo chown -r www-data.www-data quickstart sudo chmod -r 755 quickstart sudo chmod 777 quickstart/storage the routes.php file looks like: <?php use app\task; use illuminate\http\request; route::group(['middleware' => ['web']]...
Read more