Advanced downloading using Wget

Data Processing in Shell

Susan Sun

Data Person

Multiple file downloading with Wget

Save a list of file locations in a text file.

cat url_list.txt

https://websitename.com/datafilename001.txt
https://websitename.com/datafilename002.txt
...

Download from the URL locations stored within the file url_list.txt using -i.

wget -i url_list.txt
Data Processing in Shell

Setting download constraints for large files

Set upper download bandwidth limit (by default in bytes per second) with --limit-rate.

Syntax:

wget --limit-rate={rate}k {file_location}

Example:

wget --limit-rate=200k -i url_list.txt
Data Processing in Shell

Setting download constraints for small files

Set a mandatory pause time (in seconds) between file downloads with --wait.

Syntax:

wget --wait={seconds} {file_location}

Example:

wget --wait=2.5 -i url_list.txt
Data Processing in Shell

curl versus Wget

curl advantages:

  • Can be used for downloading and uploading files from 20+ protocols.
  • Easier to install across all operating systems.

Wget advantages:

  • Has many built-in functionalities for handling multiple file downloads.
  • Can handle various file formats for download (e.g. file directory, HTML page).
Data Processing in Shell

Let's practice!

Data Processing in Shell

Preparing Video For Download...