Elasticsearch: Index 2GB of documents at once -


i use the mapper-attachments-plugin index , analyze pdfs in elasticsearch index (1 node, 5 shards). works fine problem initial import of 1.800 pdfs (about 2gb) blob table in mysql database.

i use native java client (transport mode), documents database, encode them base64 , send them elasticsearch. run outofmemory errors after 300 documents. when give more memory import application, elasticsearch master node blocked after.

is there better way import documents? perhaps "bulk-index-client" or so?


Comments

Post a Comment

Popular posts from this blog

php - mysql multiple keyword search in any order -

this question exact duplicate of: php keyword search engine no results [duplicate] i have simple mysql database keyword search functional. results search not being returned if keywords not in same order entered in database. example searching "dog cat lion" return result, searching "dog lion cat" return no results. on how fix issue appreciated. here code. <!doctype html> <html> <head> <meta charset="utf-8"> <title>part search</title> </head> <body> <?php $username = "xxxx"; $password = "xxxx"; $hostname = "localhost"; //connection database mysql_connect($hostname, $username, $password); mysql_select_db("wesco"); $search = mysql_real_escape_string(trim($_post['searchterm'])); $find_parts = mysql_query("select * `parts` `keywords` '%$search%'"); while($row = mysql_fetch_as...
Read more

routing - AngularJS State management ->load multiple states in one page -

Image
i developing site in angularjs ui-router. have situation on button click need change , load 3 state page , 3 looks image .please me how can load multiple pages. in 1 page you need configure router config this: $stateprovider .state("state1", { url: "/url1", controller: "firstcontroller", views: { view1: { templateurl: "templates/template1.html" }, view2: { templateurl: "templates/template2.html" }, view3: { templateurl: "templates/template3.html" } } }).state("state2", { url: "/url2", controller: "secondcontroller", views: { view1: { templateurl: "templates/template4.html" }, view2: { templateurl: "templates/template5.html" }, view3: { templateurl: "templates/template6.html" } }).state("state3", { ...
Read more

unity3d - Problems porting a GLSL shadertoy shader to unity -

Image
i'm trying port shadertoy.com shader ( atmospheric scattering sample , interactive demo code) unity. shader written in glsl , have start editor c:\program files\unity\editor>unity.exe -force-opengl make render shader (otherwise "this shader cannot run on gpu" error comes up), that's not problem right now. problem porting shader unity. the functions scattering etc. identical , "runnable" in ported shader, thing mainimage() functions manages camera, light directions , ray direction itself. has ofcourse changed sothat unity's camera position, view direction , light sources , directions used. the main function of original looks this: void mainimage( out vec4 fragcolor, in vec2 fragcoord ) { // default ray dir vec3 dir = ray_dir( 45.0, iresolution.xy, fragcoord.xy ); // default ray origin vec3 eye = vec3( 0.0, 0.0, 2.4 ); // rotate camera mat3 rot = rot3xy( vec2( 0.0, iglobaltime * 0.5 ) ); dir = rot * dir; eye...
Read more