selenium - I am using Scrapy to crawl data, but the server block my IP -


enter image description here

as picture showed, using scrapy crawl data server, server seem block ip, curious server block ip of my mac or ip of router?

it router's public ip blocked.

in scenario there 2 networks.

one, public internet - server (hosting website crawl) connected.

two, private home network - mac connected.

your router acts gateway private home network internet , helps mac talk server.

to act "gateway" router have 2 ip addresses. 1 private ip address home network , 1 public ip address. public ip address visible server. in server's viewpoint public ip address crawl requests made.

hence it router's public ip blocked.

also please respect website's terms of service , crawl responsibly.

if don't want banned try following settings in settings.py:

  1. limit concurrent_requests
  2. set download_delay

reference : http://doc.scrapy.org/en/latest/topics/settings.html


Comments

Post a Comment

Popular posts from this blog

sublimetext3 - what keyboard shortcut is to comment/uncomment for this script tag in sublime -

Image
i have script tag in code, keyboard shortcut comment/uncomment script tag in sublime. after selecting script open , close tag. <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> i pressing ctrl + shift + / , getting this /*<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script>*/ also pressed ctrl + / , got this // <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> // </script> instead of <!-- <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> --> script open , close tag in 2 lines not in 1 line , not have spaces too. for mac,high light lines want comment. hold 'command , / ' commented. for windows: high line or select lines ,then hold ctrl + / if not work: ...
Read more

ios - MKMapView fails to load tiles with HTTP 410 error -

i have problem mkmapview . map fails load tiles when zoom in. -(void)mapviewdidfailloadingmap:(mkmapview *)mapview witherror:(nserror *)error error: domain=geoerrordomain code=-204 "(null)" userinfo={simpletilerequesterunderlyingerrors=( "error domain=geoerrordomain code=-204 \"(null)\" userinfo={httpstatus=410, nserrorfailingurlstringkey= http://gspe19.ls.apple.com/tile.vf?flags=1&style=20&size=2&scale=0&v=11037825&z=15&x=6205&y=12336&sid=0246704635757302674107153038443966765357&accesskey=1454685602_q3bvuyvhbdxsso0a_j0fk7eyq9b21npshv7grlzr4wfkkhxb4vo7%2blxcgsxj4zzhvhtalvwsypa3plu60cdrmrfwmwcybgrla9mchv%2fhorhotu9agi72vqp9ukzw%2b0gkqfrhpcw4xr%2f%2fttvgjz7wu4u4kna8k2rvvq%2foffhjq7oo4nyectvy0ur4i9d3sxf%2btn9dcxu8agdrjignb }", ... edit: seems related cache somehow, i'm not sure. problem disappears time after loading same map region in maps application. thanks in advance i analysed , d...
Read more

c# - How to utilize EF and LINQ to add filters and specify tables, columns, filters and order by dynamically -

Image
i have database structure (all parent , childs tables), what doing now want create ad hoc reporting page, let user select tables , columns in these tables. when user select tables , columns, can add filters (is in list or contains or filter etc..) columns. i send information json web service create, in web service planning use entityframework required dataset. what have done i able create html ui, web services, database layer repositories , uow, database etc.. question i mean can this, var result = context.parenta.include("child1.subchild1").include(....).where(.. but not sure how can specify columns want or add filters. for specifying column names can use select linq query : var result = context.parenta.include("child1.subchild1").include(....).where(..).select(s=> new {name = s.parentname , subname = s.subchild.name }); for adding filters need define them in clause .where(p=>p.name.contains("somevalue")) ...
Read more