Python search loop slow -


i running search on list of ads (adscrape). each ad dict within adscrape (e.g. ad below). searches through list of ids (database_ids) between 200,000 - 1,000,000 items long. want find ads in adscrape don't have id in database_ids.

my current code below. takes loooong time, , multiple seconds each ad scan through database_ids. there more efficient/faster way of running (finding items in big list, in big list)?

database_ids = ['id1','id2','id3'...] ad = {'body': u'\xa0suv', 'loc': u'sa', 'last scan': '06/02/16', 'eng': u'\xa06cyl 2.7l ', 'make': u'hyundai', 'year': u'2006', 'id': u'oag-ad-12371713', 'first scan': '06/02/16', 'odo': u'168911', 'active': 'y', 'adtype': u'dealer: used car', 'model': u'tucson auto 4x4 ', 'trans': u'\xa0automatic', 'price': u'9990'}  ad in adscrape:     ad['last scan'] = date     ad['active'] = 'y'     adscrape_ids.append(ad['id'])     if ad['id'] not in database_ids:         ad['first scan'] = date         print 'new ad:',ad         newads.append(ad)

`you can use list comprehensions code base given below. use existing database_ids list , adscrape dict given above.

code base: new_adds_ids = [ad ad in adscrape if ad['id'] not in database_ids]`


Comments

Post a Comment

Popular posts from this blog

routing - AngularJS State management ->load multiple states in one page -

Image
i developing site in angularjs ui-router. have situation on button click need change , load 3 state page , 3 looks image .please me how can load multiple pages. in 1 page you need configure router config this: $stateprovider .state("state1", { url: "/url1", controller: "firstcontroller", views: { view1: { templateurl: "templates/template1.html" }, view2: { templateurl: "templates/template2.html" }, view3: { templateurl: "templates/template3.html" } } }).state("state2", { url: "/url2", controller: "secondcontroller", views: { view1: { templateurl: "templates/template4.html" }, view2: { templateurl: "templates/template5.html" }, view3: { templateurl: "templates/template6.html" } }).state("state3", { ...
Read more

python - GRASS parser() error -

i'm trying use grass on python 2.7, i'm having problems @ setting script on idle, i'm getting error @ parser() function: here script: import os import sys gisbase = os.environ['gisbase'] = 'c:\program files (x86)\grass gis 7.0.1rc1' gisrc = 'c:\grassdata' gisdbase = 'c:\grassdata' location = 'newlocation' mapset = 'tc' ld_library_path = 'c:\program files (x86)\grass gis 7.0.1rc1\lib' path = 'c:\program files (x86)\grass gis 7.0.1rc1\etc';'c:\program files (x86)\grass gis 7.0.1rc1\etc\python';'c:\program files (x86)\grass gis 7.0.1rc1\lib';'c:\program files (x86)\grass gis 7.0.1rc1\bin';'c:\python27';'c:\program files (x86)\grass gis 7.0.1rc1\python27';'c:\program files (x86)\grass gis 7.0.1rc1\msys' pythonlib = 'c:\python27' pythonpath = 'c:\program files (x86)\grass gis 7.0.1rc1\etc\python' sys.path.append(os.path.join(os.environ['...
Read more

json - Gson().fromJson(jsonResult, Myobject.class) return values in 0's -

Image
i have json {"fechaupdateid":1,"fechasegundos":635902782941643690,"fechasegundosbc":635902779020935030} i execute command convert object fechaupdate dto = new gson().fromjson(json, fechaupdate.class); but got result this fechaupdate class public class fechaupdate { public int getfechaupdateid() { return fechaupdateid; } public void setfechaupdateid(int fechaupdateid) { this.fechaupdateid = fechaupdateid; } public long getfechasegundosbc() { return fechasegundosbc; } public void setfechasegundosbc(long fechasegundosbc) { this.fechasegundosbc = fechasegundosbc; } public long getfechasegundos() { return fechasegundos; } public void setfechasegundos(long fechasegundos) { this.fechasegundos = fechasegundos; } private int fechaupdateid ; private long fechasegundos ; private long fechasegundosbc; } solved ...
Read more