Questions tagged [gnu-parallel]
GNU parallel is a command line utility to run programs in parallel
273 questions
1 vote
1 answer
118 views
Expected behaviour of GNU parallel --memfree <size> and --memsuspend <size> when size much bigger than RAM
While experimenting with GNU parallel I found that the following cases all hang with decreasing CPU usage on a Fedora 41 VM with 8GB RAM. Is this expected behaviour? parallel --halt now,fail=1 --...
- 113
1 vote
1 answer
480 views
How to make GNU parallel report progress in a way suitable for use outside of a terminal?
I want to run a command in parallel on a bunch of files as part of a Github CI workflow (on an ubuntu runner) in order to speed up the CI job. I would also like the parallel command to report its ...
- 113
1 vote
1 answer
206 views
Parallel processing of single huge .bz2 or .gz file
I would like to use GNU Parallel to process a huge .gz or .bz2 file. I know I can do: bzcat huge.bz2 | parallel --pipe ... But it would be nice if there was a way similar to --pipe-part that can read ...
- 37.6k
2 votes
1 answer
64 views
GNU parallel: substitute a string or nothing depending on argument value
I'm trying to use GNU parallel to run a command for all combinations of several behavior-changing flags: parallel 'cmd --foo {1} --bar {2} {3} out.foo={1}.bar={2}/{3/}' ::: 0 1 ::: 0 1 ::: in/* This ...
- 5,759
1 vote
2 answers
110 views
recursive call to gnu parallel
I have a very slow samba share that i can access through wsl on my windows laptop. On windows accessing the share doesn't take much time but on linux it is in order of seconds. I basically replicate ...
- 11
1 vote
1 answer
87 views
How do I ask gnu parallel to preserve stdin?
I'm using GNU parallel like this (:::: is a form of --arg-file): parallel -0Xuj1 my-command -- :::: <(find … -print0) But it seems like the command's standard input is managed by GNU parallel, ...
- 6,809
2 votes
2 answers
159 views
Parallel for-loop in bash with simultaneous sequential execution of another task with dependencies on the parallelized loop
In my bash script, I need to execute two different functions, taskA and taskB, which take an integer ($i) as an argument. Since taskB $i depends on the completion of taskA $i, the following ...
- 121
1 vote
1 answer
108 views
GNU parallel: how to call exported bash function with empty string argument?
Scenario: $ process(){ echo "[$1] [$2] [$3]" ; } ; export -f process $ process "x" "" "a.txt" [x] [] [a.txt] Here we see that the 2nd argument is empty string ...
- 757
2 votes
1 answer
312 views
Use GNU parallel with very long lines
I have a very large SQL dumpfile (30GB) that I need to edit (do some find/replace) before loading back into the database. Besides having a large size, the file also contains very long lines. Except ...
- 23
0 votes
1 answer
83 views
gnu parallel: how to control output of program?
Fast and simple. This command works locate -i mymovieormysong|parallel mplayer the song (or movie) play, but i cannot control mplayer with keyboard. How to do (if possible) this? Actually when i use ...
- 14.5k
1 vote
1 answer
98 views
Why does /usr/bin/time coupled with GNU parallel output results before command's output rather than after command's output?
Scenario: $ cat libs.txt lib.a lib1.a $ cat t1a.sh f1() { local lib=$1 stdbuf -o0 printf "job for $lib started\n" sleep 2 stdbuf -o0 printf "job for $...
- 757
2 votes
1 answer
77 views
GNU parallel: why does diagnostic output look like sequential execution rather than parallel execution?
Scenario: $ cat libs.txt lib.a lib1.a $ cat t1a.sh f1() { local lib=$1 stdbuf -o0 printf "job for $lib started\n" sleep 2 stdbuf -o0 printf "job for $...
- 757
4 votes
1 answer
1k views
Download files in parallel with wget over a single connection
This question is a follow-up to How to download multiple files simultaniously with wget? Similar to that question, I need to download many files. With the accepted answer above recommends the use of ...
- 41
1 vote
2 answers
1k views
How can I multithreadedly extract several 7z archives into their own folder, using the archive name as the folder name?
To extract all 7z archives into their own folder, using the archive name as the folder name, I run the command: 7z x "*.7z" -o* E.g., if one has the two files a.7z and b.7z, then folders a ...
- 5,583
0 votes
3 answers
179 views
Problem in writing a parallel version of my Bash code
I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel My current code is working fine for ...
- 11