A parallel and fast way to download multiple files

We can write a short script to download multiple files easily in command, e..g

for i in X Y Z; do wget http://www.site.com/folder/$i.url; done

If we want them to run in background (so that in a pseudo-parallel way), we can use -b option for wget.

But this is still not fast enough, and the parallel with wget -b won't give me any notice once it's done.

Here is my solution: axel + parallel

parallel -a urls.file axel

Let's say I want to download all brain sample bigwig files of H3K4me1 marks from the Roadmap Epigenomics project. Here is the code:

mark=H3K4me1
> url.$mark # to generate an empty file
for i in E071 E074 E068 E069 E072 E067 E073 E070 E082 E081;
do 
  echo http://egg2.wustl.edu/roadmap/data/byFileType/signal/consolidated/macs2signal/pval/$i-$mark.pval.signal.bigwig >> url.$mark
done
parallel -a url.$mark axel -n 5 

Regarding what's axel and how fast it is comparing to wget, please refer to this link: http://www.cyberciti.biz/tips/download-accelerator-for-linux-command-line-tools.html

wget, screen

• How to download all (specific) files from a remote server?

wget -r -l1 -nH ftp://usename:pass@ftp.domain.name/folder/ -P mydownload

-r recursive 

-lNUMBER maxima recursion depth for downloading, e.g -l1 is for current folder

-nH  --no-host-directories       don't create host directories.

-P,  --directory-prefix=PREFIX   save files to PREFIX/...

Or, you can use -m (mirror, shortcut for -N -r -l inf --no-remove-listing) -nH to download all

You can also use -A to specify the file extension you want to include, e.g. -A jpg is to download all jpg files.

Good reference for wget:
http://www.linuxquestions.org/questions/linux-networking-3/use-wget-to-download-multiple-files-with-wildcards-603486/
and might be useful:
http://www.linuxquestions.org/questions/red-hat-31/i-want-to-download-ftp-site-files-via-wget-and-socks5-proxy-server-247167/

• How to use screen to manage multiple session?

type "screen" or "Ctrl-a, c" to create a new window

Ctrl-a, n   next window

Ctrl-a, p   previous window

Ctrl-a, k   kill window

Ctrl-a, ?   help

Very good tutorial for the screen command: http://www.rackaid.com/resources/linux-screen-tutorial-and-how-to/