将selenium写入到middleware和spider里还需要添加user-agent吗？ip代理呢？

Question

from selenium import webdriver

import time

browser=webdriver.Chrome(executable_path="C:\Program Files (x86)\Google\Chrome\Application\chromedriver.exe")

browser.get("https://xueqiu.com/hq/screener/CN#category=SH&orderby=tweet7d&order=desc&page=1&tweet7d=ALL")

time.sleep(5)



for tr in browser.find_elements_by_css_selector("div.stockScreener-search-result-table.mainTable.overflowx > table > tbody > tr"):

    stock_name=tr.text.split(" ")[0]

    comm_num= tr.text.split(" ")[1]

    print (stock_name+":"+comm_num)

打算抓取雪球网的一些数据，单独写一个test。py代码如上能成功，但是按照老师说的写进去返回403，

Connected to pydev debugger (build 172.3544.44)
C:\Users\Tap\Desktop\stock\snowstock\main.py
C:\Users\Tap\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\crawler.py:134: ScrapyDeprecationWarning: SPIDER_LOADER_CLASS (previously named SPIDER_MANAGER_CLASS) does not fully implement scrapy.interfaces.ISpiderLoader interface. Please add all missing methods to avoid unexpected runtime errors.
  self.spider_loader = _get_spider_loader(settings)
C:/Users/Tap/Desktop/stock/snowstock\snowstock\spiders\stock_spider.py:5: ScrapyDeprecationWarning: Importing from scrapy.xlib.pydispatch is deprecated and will no longer be supported in future Scrapy versions. If you just want to connect signals use the from_crawler class method, otherwise import pydispatch directly if needed. See: https://github.com/scrapy/scrapy/issues/1762
  from scrapy.xlib.pydispatch import dispatcher
2017-09-06 15:08:25 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: snowstock)
2017-09-06 15:08:25 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'snowstock.spiders', 'BOT_NAME': 'snowstock', 'SPIDER_MODULES': ['snowstock.spiders']}
2017-09-06 15:08:25 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.corestats.CoreStats']
2017-09-06 15:08:27 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:51189/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome"}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "chromeOptions": {"extensions": [], "args": []}, "platform": "ANY"}}
2017-09-06 15:08:29 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-09-06 15:08:29 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'snowstock.middlewares.JSPageMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-09-06 15:08:29 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-09-06 15:08:29 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-09-06 15:08:29 [scrapy.core.engine] INFO: Spider opened
2017-09-06 15:08:29 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-09-06 15:08:29 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-09-06 15:08:29 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://xueqiu.com/hq/screener/CN#category=SH&orderby=tweet7d&order=desc&page=1&tweet7d=ALL> (referer: None)
2017-09-06 15:08:29 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://xueqiu.com/hq/screener/CN>: HTTP status code is not handled or not allowed
2017-09-06 15:08:29 [scrapy.core.engine] INFO: Closing spider (finished)
2017-09-06 15:08:29 [selenium.webdriver.remote.remote_connection] DEBUG: DELETE http://127.0.0.1:51189/session/245926818e111deada0191a7a0c0d445 {"sessionId": "245926818e111deada0191a7a0c0d445"}
spider closed
2017-09-06 15:08:29 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-09-06 15:08:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 222,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 378,
 'downloader/response_count': 1,
 'downloader/response_status_count/403': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 9, 6, 7, 8, 29, 584373),
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/403': 1,
 'log_count/DEBUG': 6,
 'log_count/INFO': 8,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 9, 6, 7, 8, 29, 228122)}
2017-09-06 15:08:32 [scrapy.core.engine] INFO: Spider closed (finished)

Process finished with exit code 0

Answer 1

bobby 2017-09-08 09:45:56

亲 selenium的话 如果用chromdriver的话目前的版本 想要动态设置ip代理和user-agent是不行的 如果你要动态设置可以考虑phantomjs， 你现在的意思是用用selenium集成到爬虫中抓取雪球网会返回403？ 如果是403一般可能是cookie引起的 你试试在setting中设置cookie_enabled为false试试

• 提问者 慕斯0800953 #1

  非常感谢老师！