NWORKERS keeps CPU down but not memory

[mjuric]

Hi Mario,

I've found another case in which the NWORKERS environmental variable doesn't seem to work as I would expect. I'm importing data from 2MASS into an existing database - I previously imported half the data, and now I'm importing the second half.

I've set NWORKERS to 8 by invoking export NWORKERS=8. Then, after entering lsd-import text tmass 2MASS_data/psc_b*, it looks in htop like LSD has spawned ~125 processes.

I tried using NWORKERS=24 before, and a much larger number of processes were spawned, which filled the RAM and swap, and almost brought pan down before I managed to terminate the processes. It looks like NWORKERS has some effect on the number of spawned processes, but it doesn't set the number exactly.

The CPU usage is, however, constrained by NWORKERS. Although a very large number of processes are spawned, the number actually doing work at any given time seems to be NWORKERS. The problem is that all of the inactive processes take up memory, which can easily crash the machine.

Thanks,
-Greg