{"id":43,"date":"2019-06-04T03:08:42","date_gmt":"2019-06-04T03:08:42","guid":{"rendered":"http:\/\/justin.ooo\/?p=43"},"modified":"2023-06-20T03:12:16","modified_gmt":"2023-06-20T03:12:16","slug":"creating-a-requests-session-from-a-selenium-webdriver","status":"publish","type":"post","link":"https:\/\/justin.ooo\/index.php\/2019\/06\/04\/creating-a-requests-session-from-a-selenium-webdriver\/","title":{"rendered":"Creating a ‘requests’ session from a selenium web driver"},"content":{"rendered":"\n

Python is frequently used for web scraping. Often times, the ‘requests’ library is sufficient. However, it is typically only used for basic requests. We can send a GET request to a website, but what if the actual page is loaded via javascript? Using a real browser\/web driver allows us to load the page completely. Instead of simply sending a request to a url, we can automatically execute scripts and download resources.<\/p>\n\n\n\n

In my case, I’m scraping a website which requires me to be logged in. However, the login page has a variety of security implementations that can’t easily be circumvented with simple HTTP requests. After realizing this, I decided to use a selenium webdriver<\/g><\/g> to complete the login. After logging in, I simply needed the session information (cookies) established by the login request to scrape the rest of the site.<\/p>\n\n\n\n

The selenium webdriver objects have a get_cookies<\/strong><\/a> function, which returns a list<\/strong> of dict<\/strong>s. Here is a list of the key<\/strong>s in each dictionary, alongside their type<\/strong> and a brief description<\/strong>:
<\/p>\n\n\n\n

name (string): The name of the cookie.\nvalue (string): The value of the cookie.\ndomain (string): The domain of the server the cookie is sent to.\npath (string): Document location in which cookie is sent. \nsecure (bool): Cookie is only sent to the server in encrypted requests.\nhttpOnly (bool): Prevents the cookie from being accessed through client side scripts.\nexpiry (int): Unix timestamp indicating when the cookie expires.<\/pre>\n\n\n\n
Knowing this, here is an example of the result of using json.dumps(list)<\/strong> to serialize the resulting list from get_cookies<\/strong>:<\/p>\n\n\n\n
[\n  {\n    \"name\": \"cookie1\",\n    \"value\": \"whatever\",\n    \"path\": \"\/\",\n    \"domain\": \".justin.ooo\",\n    \"secure\": true,\n    \"httpOnly\": false,\n    \"expiry\": 1590978394\n  },\n  {\n    \"name\": \"cookie2\",\n    \"value\": \"doesn't matter\",\n    \"path\": \"\/\",\n    \"domain\": \"justin.ooo\",\n    \"secure\": true,\n    \"httpOnly\": false,\n    \"expiry\": 1559528794\n  }\n]<\/code><\/pre>\n\n\n\n
That’s a very simple example, containing 2 meaningless cookies. We need to somehow derive a requests<\/g>.Session<\/g> object from this list of dictionaries. Unfortunately, the requests library does not store cookies in simple dicts. Instead, it uses http.cookiejar.Cookie<\/a><\/strong> objects.<\/p>\n\n\n\n
After some brief searching on the CPython git repo<\/a><\/strong>, we can find the http.cookiejar.Cookie<\/strong> class source code<\/a> & constructor<\/a><\/strong>. To effectively copy these cookies, we’ll need to instantiate an instance of the Cookie object from each of our cookie dicts<\/g>, and then set each of those cookies in the new session. This can be achieved similarly using the requests.cookies.create_cookie<\/a> function, however I chose to use the standard construc<\/g>tor. My solution is written as follows:<\/p>\n\n\n\n
def generate_cookie(cookie_raw):\n    \"\"\"\n    Creates a http.cookiejar.Cookie object, given raw cookie information as dict.\n    This dict must contain the following keys: name, value, domain, path, secure\n    Parameters:\n        cookie_raw (dict): The cookie information dictionary.\n    Returns:\n        http.cookiejar.Cookie: The generated cookie object.\n    \"\"\"\n    # expiry is optional, so default it to false if not set\n    if not 'expiry' in cookie_raw:\n        cookie_raw['expiry'] = False\n    # initialize Cookie object\n    cookie = http.cookiejar.Cookie(\n        0,                      # version\n        cookie_raw['name'],     # name\n        cookie_raw['value'],    # value\n        None,                   # port\n        False,                  # port_specified\n        cookie_raw['domain'],   # domain\n        True,                   # domain_specified\n        \"\",                     # domain_initial_dot\n        cookie_raw['path'],     # path\n        True,                   # path_specified,\n        cookie_raw['secure'],   # secure\n        cookie_raw['expiry'],   # expires\n        False,                  # discard\n        \"\",                     # comment\n        \"\",                     # comment_url\n        None,                   # rest\n        )\n    return cookie<\/code><\/pre>\n\n\n\n
This block of code generates a single http.cookiejar.Cookie<\/strong> object from a dict. Logically, we can complete our goal by simply<\/p>\n\n\n\n