Apache Drill not using max RAM -

i'm running apache drill 1.0(and on 1.4) locally on ubuntu machine has 16gb of ram. when work large tab delimited file(52 million rows, 7gb), , perform

select distinct columns[0] `table.tsv`  

,performance seems not improve @ second time same query ran (both took 53 seconds). second time same query ran, takes less half time compared first query. seems drill not using allocated memory.

my conf/drill-env.sh file looks like:

drill_max_direct_memory="14g" drill_heap="14g"  export drill_java_opts="-xms$drill_heap -xmx$drill_heap -xx:maxdirectmemorysize=$drill_max_direct_memory -xx:maxpermsize=14g -xx:reservedcodecachesize=1g -ddrill.exec.enable-epoll=true" 

i did within drill

alter system set `planner.memory.max_query_memory_per_node`=12884901888 

however, when check memory usage using smem, it's using 5gb of ram.

if cut table size 1 million row, can see first query completed in 3.6seconds , second time same query ran, took 1.8 seconds

what missing?

you have 16 gb of ram, it's not possible drill use 14 gb heap , 14 gb direct memory. these types of memory not overlap.

i suggest leave 2 gb os, have 14 gb left, assign 12 gb direct memory , 2 gb heap.

you'll find option named planner.width.max_per_node value of 70% of cores count. increase amount see fit.

you may want read the answers question.