4

This question is a follow-up to How to download multiple files simultaniously with wget?

Similar to that question, I need to download many files. With the accepted answer above recommends the use of cat url-list | parallel -j8 wget {}. But this will require establishing one TCP connection and also one https/TLS negotiation per URL, which is a waste on both client and server side.

If/Since all URLs are on the same server, they could be queried in multiple GET /object.txt over a single TCP / HTTPs connection.

Question: any way to do this with wget ?

Improve this question

1 Answer 1

Reset to default
5

Wget does this by default (--no-http-keep-alive turns it off). If multiple successive URLs are on the same server, it reuses the same connection (assuming the server is willing).

You need to tell parallel to invoke wget fewer times. Pass -X to make it bunch the arguments in each invocation.

cat url-list | parallel -j8 -X wget
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.