Questions tagged [parallelism]
Performing tasks in parallel, to make use of multiple processors
279 questions
4 votes
4 answers
581 views
How to wait for background commands that were executed within a subshell?
Given this code: #!/bin/bash set -euo pipefail function someFn() { local input_string="$1" echo "$input_string start" sleep 3 echo "$input_string end" } function ...
- 16.7k
1 vote
1 answer
640 views
`pzstd` = parallelized Zstandard, how to watch progress in 4TB large file/disk?
I am brand new to zstd/pzstd, trying out its features, compression, benchmarking it, and so on. (I run Linux Mint 22 Cinnamon.) This computer has 32 GB RAM. The basic command appears to be working, ...
- 31.3k
0 votes
1 answer
215 views
How to verify size of pigz (parallel gzip) archive contents? [duplicate]
I created some pigz (parallel gzip) - home page - compressed archives of my SSD disk drives. (compiled version 2.8) I called one of them 4TB-SATA-disk--Windows10--2024-Jan-21.img.gz which says the ...
- 31.3k
1 vote
1 answer
136 views
Curious lscpu Output Related to CPU Calculation
I have been trying to understand lscpu's output and came across several threads dedicated to concepts of CPUs, physical cores, and threads. Based on those threads, to get the total number of CPUs (...
- 13
0 votes
1 answer
103 views
Starting another OS to run parallel with another system
So at work we have a few systems running on their own hardware that is identical to all of the systems. If one of the hardware fails, one of the running system will take over and start the system that ...
0 votes
3 answers
179 views
Problem in writing a parallel version of my Bash code
I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel My current code is working fine for ...
- 11
1 vote
2 answers
2k views
How can I run `$(nproc)`× parallel instances of `openssl dhparam` & when first instance exits with `0`, kill the other instances?
I'm running openssl dhparam -out dhparam4096.pem 4096 and it pegs a single core at 100% for the duration of the task (which can be considerable on some processors). I have 1 or more additional cores ...
- 163
3 votes
2 answers
1k views
Race condition not seen while two scripts write to a same file
I have two scripts running parallelly and they are echoing to the same file. One script is echoing +++++++++++++++ to the file while the other script is echoing =========== to the file. Below is the ...
- 101
1 vote
1 answer
189 views
optipng in parallel using shell
I would like to run optipng in parallel with my 8-thread CPU using the shell. I know the program itself is not multicore optimized, the only way will be to run 8 files with it. I have 500+ PNG images ...
- 31.3k
3 votes
1 answer
1k views
How do you fetch a large file over http in parallel?
Question: Since HTTP supports resuming at an offset, are there any tools (or existing options for commands like wget or curl) that will launch multiple threads to fetch the file in parallel with ...
- 545
0 votes
0 answers
811 views
Terminal closes itself when running many python processes
I was trying to run multiple machine learning experiments simultaneously to save time. My OS is Fedora 36, and I use emacs. To automate the experiment I used runs.py import subprocess subprocess.run(&...
- 103
4 votes
4 answers
473 views
Parallelize filling of a variable in Bash
I am trying to speed up a find command which accesses files on multiple harddrives by utilizing parallelization. Unfortunately, either the parallelization is ignored or the variable is not filled. ...
- 278
1 vote
1 answer
385 views
Does parallel limit cause some items to not be checked?
I have near 400 git repos on my machine. And this is a script that I use to find their status collectively: function Check() { gitFolder="$1" parent=$(dirname $gitFolder); ...
- 841
1 vote
1 answer
307 views
How to parallelize a bash for loop and use the iteration variable for file IO
I have a bash script that more or less looks like this: N=32 for i in $(seq -f "%06g" 0 ${LAST_NUM}) # LAST_NUM is an env variable do ((j=j%N)); ((j++==0)) && wait # Wait for all ...
- 11
2 votes
1 answer
2k views
GNU Parallel with -j -N still uses one CPU
How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
- 1,470