{"id":80,"date":"2019-06-07T23:48:33","date_gmt":"2019-06-07T23:48:33","guid":{"rendered":"http:\/\/justin.ooo\/?p=80"},"modified":"2023-06-20T03:08:22","modified_gmt":"2023-06-20T03:08:22","slug":"showing-progress-of-get-put-using-requests-clint","status":"publish","type":"post","link":"https:\/\/justin.ooo\/index.php\/2019\/06\/07\/showing-progress-of-get-put-using-requests-clint\/","title":{"rendered":"Showing progress of GET\/PUT using ‘requests’ & ‘clint’"},"content":{"rendered":"\n

Uploading\/downloading large files can be tedious, especially when you’re unable to view the progress and status of the request. Using the requests<\/a> library alongside clint<\/g><\/a>, it’s easy to visually display progress in a console application. We’ll be able to specify the chunk size and monitor the download speed.<\/p>\n\n\n\n

In the snippets below, you’ll see a function ‘progress.bar’ called. This is contained inside if the clint<\/g>.textui<\/g><\/g> module<\/g>, and it’s the function that will actually show our progress while the ‘requests’ library handles our enumerable chunks of data.<\/p>\n\n\n\n

Required imports:<\/p>\n\n\n\n

import requests\nimport os\nfrom clint.textui import progress<\/code><\/pre>\n\n\n\n
GET Requests (Downloading)<\/h3>\n\n\n\n
Steps:<\/p>\n\n\n\n
    \n
  1. Send GET request to the file we’d like to download.<\/li>\n\n\n\n
  2. Open a file object to the path we’re downloading to, allowing us to continuously append bytes to the end of the file.<\/li>\n\n\n\n
  3. Determine the number of chunks we’ll be downloading, based on the “Content-Length” response header.<\/li>\n\n\n\n
  4. Use the iter_content to convert the response into a chunk iterator.<\/li>\n\n\n\n
  5. Enumerate chunks, writing each one into the output file<\/li>\n<\/ol>\n\n\n\n
    def download_file(url, path):\n    # STEP 1: send the request. note that stream must be True.\n    response = requests.get(url, stream = True)\n    if response.status_code != 200: return False\n    # STEP 2: open the file object to a local path, so we can write each chunk of data.\n    with open(path, 'wb') as file:\n        # STEP 3: determine the number of chunks we'll be downloading, use the \"Content-Length\" header\n        total_length = int(response.headers.get(\"Content-Length\"))\n        count = (total_length \/ 1024) + 1\n        # STEP 4: use iter_content to get list of cunks, and use use 'clint' to log progress\n        chunks = response.iter_content(1024)\n        chunker = progress.bar(chunks,  expected_size = count, label = \"downloading: \")\n        # STEP 5: loop through chunks, write them to file\n        for chunk in chunker:\n            if not chunk: break\n            file.write(chunk)\n            file.flush()\n    return True<\/code><\/pre>\n\n\n\n
    PUT Requests (Uploading)<\/h3>\n\n\n\n
    Steps:<\/p>\n\n\n\n
      \n
    1. Open a file object to the path we’ll be uploading data from, allowing us to read chunked content from the file.<\/li>\n\n\n\n
    2. Determine the number of chunks we’ll be uploading, by getting the size of the file on the disk.<\/li>\n\n\n\n
    3. Read the file into an enumerable list of chunks.<\/li>\n\n\n\n
    4. Send the PUT request to upload the chunked data.<\/li>\n<\/ol>\n\n\n\n
      <\/p>\n\n\n\n
      <\/p>\n\n\n\n
      def get_chunks(file, chunk_size = 1):\n    while True:\n        data = file.read(chunk_size)\n        if not data: break\n        yield data\n\ndef upload_file(local_path, url):\n    # STEP 1: open a file object to read content from the local file path\n    with open(local_path,'rb') as file:\n        # STEP 2: determine the total number of chunks we're uploading from the file size\n        total_length = os.path.getsize(local_path)\n        count = (total_length \/ 1024) + 1\n        # STEP 3: read the file into an enumerable container of chunks, using the get_chunks function\n        chunks = get_chunks(file, 1024)\n        chunker = progress.bar(chunks, expected_size = count, label = \"uploading: \")\n        # STEP 4: send the post request, with chunked iter\n        requests.put(url, data = chunker)<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
      Uploading\/downloading large files can be tedious, especially when you’re unable to view the progress and status of the request. Using the requests library alongside clint, it’s easy to visually display progress in a console application. We’ll be able to specify the chunk size and monitor the download speed. In the snippets below, you’ll see a function ‘progress.bar’ called. This is contained inside if the clint.textui module, and it’s the function […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[2,3],"tags":[],"_links":{"self":[{"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/posts\/80"}],"collection":[{"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/comments?post=80"}],"version-history":[{"count":11,"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/posts\/80\/revisions"}],"predecessor-version":[{"id":258,"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/posts\/80\/revisions\/258"}],"wp:attachment":[{"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/media?parent=80"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/categories?post=80"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/justin.ooo\/index.php\/wp-json\/wp\/v2\/tags?post=80"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}