python - Memory efficient way to keep paired lines (string match) -


i have large text files need sort through , remove unpaired lines. paired lines consecutive lines have same 32 characters @ beginning of each line. have script written in python while loop iterates through lines, compares first 32 characters of lines , (i+1), outputs lines in pairs. however, method memory intensive , slow since each file can several gigabytes. there faster , more efficient method recommend? reference, working on slurm linux server.

this uses 2 favorite python modules, itertools , collections. use itertools.groupby walk lines in file, grouping common prefix. use next() pull first element lines iterator, , 0-length deque consume remainder of lines iterator.

from itertools import groupby collections import deque consume = deque(maxlen=0).extend operator import itemgetter  prefix_slice = slice(0,32) open('bigfile.txt') infile):     _,lines in groupby(infile, key=itemgetter(prefix_slice)):         print next(lines).rstrip('\n')         # have consume iterator on remaining lines before         # advancing next groupby key         consume(lines)

this hold single line in memory @ time, plus 32-character prefix comparison following lines. (will collapse consecutive lines share common 32-character prefix, not pairs.)


Comments

Post a Comment

Popular posts from this blog

sublimetext3 - what keyboard shortcut is to comment/uncomment for this script tag in sublime -

Image
i have script tag in code, keyboard shortcut comment/uncomment script tag in sublime. after selecting script open , close tag. <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> i pressing ctrl + shift + / , getting this /*<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script>*/ also pressed ctrl + / , got this // <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> // </script> instead of <!-- <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> --> script open , close tag in 2 lines not in 1 line , not have spaces too. for mac,high light lines want comment. hold 'command , / ' commented. for windows: high line or select lines ,then hold ctrl + / if not work: ...
Read more

post - imageshack API cURL -

i credentials correct. when send image through form this:(action="http://api.imageshack.com/v2/images/"). working fine when tried send $ch = curl_init($url); failed. code: enter code here <form action="" method="post" enctype="multipart/form-data"> select image upload: <input type="file" name="file" id="file" multiple> <input type="text" name="auth_token" id="auth_token" value="xxxxxxxxxxxxxxxx"> <input type="text" name="api_key" id="api_key" value="xxxxxxxxxxxxxxx"> <input type="submit" value="upload image" > </form> $url = ' http://api.imageshack.com/v2/images/ '; $temp = $_files["file"]['tmp_name']; echo $temp."<br>" ; $ch = curl_init($url); curl_setopt($ch, curlopt_post, true); curl...
Read more

dataset - MPAndroidchart returning no chart Data available -

i having problems mpandroidchart. adding temperature , humidity data chart, displays "no chart data available". here can see bug is? super grateful! //create data //set size of data int size = list.size(); if (size > 24){ size = 24;} //create lists form temp , hum arraylist<entry> temps = new arraylist<entry>(); arraylist<entry> hums = new arraylist<entry>(); //fill list temp values (int = 0; < size; i++){ entry value = new entry(math.round(list.get(list.size()-size + i).temp), (size-i)); temps.add(value); } //fill list hum values (int = 0; < size; i++){ entry value = new entry(math.round(list.get(list.size()-size + i).hum), (size-i)); hums.add(value); } ...
Read more