We'll learn about the wget command, which will allow us to download files from the internet.

First, read the DESCRIPTION section of its manual to get a basic understanding of it.

The basic usage of the command is like so:

wget |URL|

The command's input argument is a URL to download to the current directory.

This command has many useful flags, for example:

  • -i |filename|. If |filename| contains a list of URLs, wget will download all of them one by one.
  • -c |filename|. If |filename| is a file saved by a previous download that was interrupted for whatever reason, this flag continues the download.
  • -r. If the given URL is a webpage, this flag recursively downloads every URL linked to on the webpage, and all URLs linked to on every linked URL, etc'.
  • -lX. This flag, when used together with -r, specifies how deep the recursion should go.
  • -A "|pattern|". This flag will cause wget to download only URLs that follow the given pattern, for example '-A "*.pdf"' will cause wget to download only PDF files.
  • -R "|pattern|". This flag will cause wget to download only URLs that don't follow the given pattern, for example '-R "*.pdf"' will cause wget to download everything besides PDF files.

Combining these flags we can get some cool results, try it out!