Showing progress of GET/PUT using ‘requests’ & ‘clint’

Uploading/downloading large files can be tedious, especially when you’re unable to view the progress and status of the request. Using the requests library alongside clint, it’s easy to visually display progress in a console application. We’ll be able to specify the chunk size and monitor the download speed.

In the snippets below, you’ll see a function ‘progress.bar’ called. This is contained inside if the clint.textui module, and it’s the function that will actually show our progress while the ‘requests’ library handles our enumerable chunks of data.

Required imports:

import requests
import os
from clint.textui import progress

GET Requests (Downloading)

Steps:

1. Send GET request to the file we’d like to download.
2. Open a file object to the path we’re downloading to, allowing us to continuously append bytes to the end of the file.
3. Determine the number of chunks we’ll be downloading, based on the “Content-Length” response header.
4. Use the iter_content to convert the response into a chunk iterator.
5. Enumerate chunks, writing each one into the output file

def download_file(url, path):
    # STEP 1: send the request. note that stream must be True.
    response = requests.get(url, stream = True)
    if response.status_code != 200: return False
    # STEP 2: open the file object to a local path, so we can write each chunk of data.
    with open(path, 'wb') as file:
        # STEP 3: determine the number of chunks we'll be downloading, use the "Content-Length" header
        total_length = int(response.headers.get("Content-Length"))
        count = (total_length / 1024) + 1
        # STEP 4: use iter_content to get list of cunks, and use use 'clint' to log progress
        chunks = response.iter_content(1024)
        chunker = progress.bar(chunks,  expected_size = count, label = "downloading: ")
        # STEP 5: loop through chunks, write them to file
        for chunk in chunker:
            if not chunk: break
            file.write(chunk)
            file.flush()
    return True

PUT Requests (Uploading)

Steps:

1. Open a file object to the path we’ll be uploading data from, allowing us to read chunked content from the file.
2. Determine the number of chunks we’ll be uploading, by getting the size of the file on the disk.
3. Read the file into an enumerable list of chunks.
4. Send the PUT request to upload the chunked data.

def get_chunks(file, chunk_size = 1):
    while True:
        data = file.read(chunk_size)
        if not data: break
        yield data

def upload_file(local_path, url):
    # STEP 1: open a file object to read content from the local file path
    with open(local_path,'rb') as file:
        # STEP 2: determine the total number of chunks we're uploading from the file size
        total_length = os.path.getsize(local_path)
        count = (total_length / 1024) + 1
        # STEP 3: read the file into an enumerable container of chunks, using the get_chunks function
        chunks = get_chunks(file, 1024)
        chunker = progress.bar(chunks, expected_size = count, label = "uploading: ")
        # STEP 4: send the post request, with chunked iter
        requests.put(url, data = chunker)