爬取lagou时报错无法获取内容

Question

显示重定向，有2个问题 -

1. 是不是被禁爬虫了？该如何解封？

2. 另外再问下，用crawl模板生成的任务中，如何加入header？

Answer 1

bobby 2017-07-03 14:24:55

重启家里的路由器 然后ip就会变 直接在yield request(headers=header)

• 提问者 williamg #1

  还是不行，我加了header，加了代理，但出来的还是同样的情况 -
  2017-07-09 11:00:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/zhaopin/shenduxuexi/> (referer: https://www.lagou.com/)
  2017-07-09 11:00:02 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.lagou.com/zhaopin/jiqixuexi/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
  2017-07-09 11:00:02 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=112.195.97.53> from <GET https://www.lagou.com/zhaopin/jingzhuntuijian/>
  2017-07-09 11:00:03 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=112.195.97.53> from <GET https://www.lagou.com/zhaopin/sousuosuanfa/>
  2017-07-09 11:00:03 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=112.195.97.53> from <GET https://www.lagou.com/zhaopin/iOS/>

• bobby 回复 提问者 williamg #2

  你加我的qq 442421039 我看看