Number of executors

[krame505]

I've been watching the CPU and memory usage on coldpress for a while, and there seems to be a fair amount of margin available. Since the issue that made us drop it down to 5 turned out to actually be a different problem, is there any reason not to bump it back up to 6 (or more?)

[krame505]

Actually I think I just dead-locked Jenkins - all the executors are occupied with Silver or ableC builds, waiting for an executor to become available for their integration builds to finish. I guess I'm going to need to add an executor in order to recover from this - let me know if I should change it back.

[tedinski]

If it's that bogged down with those useless slots, you can up it to 7 or so temporarily.

But we should reduce back to 5 after. When someone is logged into the machine, it can consume too much memory sometimes when jenkins is running.

[krame505]

I made a change to the default parameters passed to extension make commands, limiting the max number of threads to 8 when building examples/tests. This doesn't significantly affect the runtime (since we achieve 100% CPU utilization anyway), but significantly reduces the amount of memory consumed in these memory-intensive stages, especially for extensions with large numbers of tests.

With this change, could we go back up to 6 executors? We potentially lower the max threads for each build even further without hurting the performance too much. Thoughts?

[tedinski]

Not yet, but let's leave this open.

I think this is the blocker: melt-umn/silver#232

I think part of the reason coldpress gets so bogged down may not be memory pressure, but IOPS overload on the poor spinning rust disk. If we significantly reduce the number of small files that get copied around, we could probably up the number of executors again.