请稍等 ...
×

采纳答案成功！

向帮助你的同学说点啥吧！感谢那些助人为乐的人

拉勾网302问题

老师，我爬拉勾网的时候，加了UA和cookie，但是还是出现了302的问题。（用浏览器依然可以访问，应该是没封我的IP）
请问老师，这是为啥呀？
慕粉9451062 2019-01-29 21:59:11
源自：7-6   item loader方式解析职位
1023
收起

正在回答 回答被采纳积分+3

1回答

提问者 慕粉9451062 2019-01-30 09:21:09

补充console信息：

2019-01-30 09:32:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/> (referer: None)
2019-01-30 09:32:28 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5455295.html&t=1548811960&_ti=1> from <GET https://www.lagou.com/jobs/5455295.html>
2019-01-30 09:32:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5455295.html&t=1548811960&_ti=1> (referer: https://www.lagou.com/)
2019-01-30 09:32:29 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F2960414.html&t=1548811961&_ti=1> from <GET https://www.lagou.com/jobs/2960414.html>
2019-01-30 09:33:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5455295.html&t=1548811960&_ti=1>
{'crawl_time': datetime.datetime(2019, 1, 30, 9, 32, 29, 341141),
 'url': 'https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5455295.html&t=1548811960&_ti=1',
 'url_object_id': 'aef6e325d316968e50cd5cc87009e0f4'}
2019-01-30 09:33:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.lagou.com/jobs/2856806.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
2019-01-30 09:33:52 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F2960511.html&t=1548811961&_ti=1> from <GET https://www.lagou.com/jobs/2960511.html>
2019-01-30 09:33:54 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.lagou.com/jobs/5357786.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
2019-01-30 09:33:56 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.lagou.com/jobs/5474115.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2019-01-30 09:33:57 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 1 items (at 1 items/min)
2019-01-30 09:33:59 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.lagou.com/jobs/5528540.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2019-01-30 09:34:02 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.lagou.com/jobs/5197365.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2019-01-30 09:34:04 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.lagou.com/jobs/2844860.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2019-01-30 09:34:06 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.lagou.com/jobs/5497110.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2019-01-30 09:34:10 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F3165552.html&t=1548812060&_ti=1> from <GET https://www.lagou.com/jobs/3165552.html>
2019-01-30 09:34:13 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5481459.html&t=1548812060&_ti=1> from <GET https://www.lagou.com/jobs/5481459.html>
2019-01-30 09:34:15 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5317219.html&t=1548812060&_ti=1> from <GET https://www.lagou.com/jobs/5317219.html>
2019-01-30 09:34:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F3189780.html&t=1548812060&_ti=1> from <GET https://www.lagou.com/jobs/3189780.html>
2019-01-30 09:34:20 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F3111672.html&t=1548812069&_ti=1> from <GET https://www.lagou.com/jobs/3111672.html>
2019-01-30 09:34:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F2960511.html&t=1548811961&_ti=1> (referer: https://www.lagou.com/)
2019-01-30 09:34:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F2960414.html&t=1548811961&_ti=1> (referer: https://www.lagou.com/)
2019-01-30 09:34:26 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F2832873.html&t=1548812069&_ti=1> from <GET https://www.lagou.com/jobs/2832873.html>
2019-01-30 09:34:27 [scrapy.extensions.logstats] INFO: Crawled 4 pages (at 2 pages/min), scraped 1 items (at 0 items/min)
2019-01-30 09:34:29 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F3071732.html&t=1548812078&_ti=1> from <GET https://www.lagou.com/jobs/3071732.html>
2019-01-30 09:34:32 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5474115.html&t=1548812078&_ti=1> from <GET https://www.lagou.com/jobs/5474115.html>
2019-01-30 09:34:34 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F5357786.html&t=1548812078&_ti=1> from <GET https://www.lagou.com/jobs/5357786.html>
2019-01-30 09:34:36 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.lagou.com/utrack/trackMid.html?f=https%3A%2F%2Fwww.lagou.com%2Fjobs%2F2769395.html&t=1548812078&_ti=1> from <GET https://www.lagou.com/jobs/2769395.html>


0 回复 有任何疑惑可以回复我~
收起回答
  • bobby #1 
    是的， 这里被重定向了， 你可以试试重新登录 看看是否还是会重现这个问题呢
    回复 有任何疑惑可以回复我~ 2019-02-01 15:10:38
  • 提问者 慕粉9451062 回复 bobby #2 
    好的 老师
    回复 有任何疑惑可以回复我~ 2019-02-01 15:29:52
取消 回复

相似问题

完全没办法匹配拉勾网URL了
1038 0 7
302跳转重定向问题
1717 0 4
已经各种设置，拉勾网访问仍旧出现302
667 0 1
拉勾网模拟登入
1123 0 4
拉勾网模拟登录后还是302怎么办
802 0 1

登录后可查看更多问答，登录/注册

问题已解决，确定采纳
还有疑问，暂不采纳
Scrapy打造搜索引擎 畅销4年的Python分布式爬虫课
  • 参与学习       5794    人
  • 解答问题       6278    个

带你彻底掌握Scrapy，用Django+Elasticsearch搭建搜索引擎

了解课程

本课精华内容

问答 作业

Missing argument grant_type

4.8k 30

MysqlTwistedPipline的对象没有cursor属性

2.5k 18

知乎answer提取不到

1.1k 18

为什么知乎数据导入不到MySQL数据库中？

1.4k 15

运行 scrapy crawl jobbole 报错

2.8k 15

查看更多本课问答
意见反馈 帮助中心 APP下载
官方微信