添加cookie

Question

老师：在selenium，获取到拉勾网的cookie后，保存在本地后，我怎么用这个cookie，就是带到每一个请求里面，它是list 包裹的dict

[
{
    "domain": ".lagou.com",
    "expirationDate": 1655007008,
    "hostOnly": false,
    "httpOnly": false,
    "name": "__lg_stoken__",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "170afd24d251de6cb94fed9b8fe8ad45743f6",
    "id": 1
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654876800,
    "hostOnly": false,
    "httpOnly": false,
    "name": "__SAFETY_CLOSE_TIME__21662680",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "1",
    "id": 2
},
{
    "domain": ".lagou.com",
    "expirationDate": 1717917211,
    "hostOnly": false,
    "httpOnly": false,
    "name": "_ga",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "170afd24d251de6cb94fed9b8fe8ad45743f6",
    "id": 3
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654845764,
    "hostOnly": false,
    "httpOnly": false,
    "name": "_gat",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "1",
    "id": 4
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654931611,
    "hostOnly": false,
    "httpOnly": false,
    "name": "_gid",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "GA1.170afd24d251de6cb94fed9b8fe8ad45743f6",
    "id": 5
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "_putrc",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "97547EC170afd24d251de6cb94fed9b8fe8ad45743f60EADC",
    "id": 6
},
{
    "domain": ".lagou.com",
    "expirationDate": 1655450009.568349,
    "hostOnly": false,
    "httpOnly": false,
    "name": "gate_login_token",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "9aa4aa8db4",
    "id": 7
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "1654845212",
    "id": 8
},
{
    "domain": ".lagou.com",
    "expirationDate": 1686381211,
    "hostOnly": false,
    "httpOnly": false,
    "name": "Hm_lvt_4233ea3d81c6ccf756e6",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "16548340526117",
    "id": 9
},
{
    "domain": ".lagou.com",
    "expirationDate": 1657437210.665005,
    "hostOnly": false,
    "httpOnly": false,
    "name": "index_location_city",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "%E5%B9%BF%E5%B7%9E",
    "id": 10
},
{
    "domain": ".lagou.com",
    "expirationDate": 3802328856.435914,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LG_HAS_LOGIN",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "1",
    "id": 11
},
{
    "domain": ".lagou.com",
    "expirationDate": 3802328856.435821,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LG_LOGIN_USER_ID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "abcs",
    "id": 12
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "LGRID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "20220610151331-5dfe84fb-07b5-462c-9472-b1c351ff1da1",
    "id": 13
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654847011.24849,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LGSID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "20220610151244-48392d09-f248-49fc-94a3-2af0aaf9a69b",
    "id": 14
},
{
    "domain": ".lagou.com",
    "expirationDate": 1967514230.483062,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LGUID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "20220510114351-abdfed1d-9e4b-4f5d-9b8c-743e8d8f6c5d",
    "id": 15
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "login",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "true",
    "id": 16
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.854852,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_HOST",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "",
    "id": 17
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.85488,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_LAND",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "https%3A%2F%2Fwww.lagou.com%2Fguangzhou%2F",
    "id": 18
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.854867,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_SITE",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "",
    "id": 19
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.854836,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_UTM",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "",
    "id": 20
},
{
    "domain": ".lagou.com",
    "expirationDate": 1655450009.944884,
    "hostOnly": false,
    "httpOnly": false,
    "name": "privacyPolicyPopup",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "false",
    "id": 21
}]

Answer 1

提问者 若羲 2022-06-10 17:57:47

1. 在pycharm 找到scrapy的源码，看cookies的参数，初始化时，是可以传入list（dict）

2. 在读取cookie文件后，将数据反序列化，转为list，里面每一个就是一个dict  

3. 在中间件中(process_request) 方法中，把cookies 设置为上面读取出来的值 

4. 记得在settings.py中配置，该下载中间件，不然不会生效 

ps：在其他地方自己揣摸……

• bobby #1

  如何将list类型的cookie转换为dict放入scrapy的请求中，课程中模拟登录就是讲解的这个内容啊