Running a limited number of scripts in parallel from Bash

Standard

Imagine you have a text file with a single parameter for another script on each line, but you want to speed things up. Instead of writing an overly complicated wrapper script, as I did a few times in the past, you could just use xargs. It comes equipped with everything needed for this task. The following example assumes, that for each parameter in parameters.txt the command MyFancyScript.py should be executed, with no more than 20 processes at the same time:

cat parameters.txt | xargs -n 1 -P 20 MyFancyScript.py

I guess it’s not hard to figure out that -P is the magic switch to allow multiple instances to be executed at the same time.