segfault launching flux session with >=128 tasks per node

[garlick]

As noted in flux-framework/distribution#13, starting a Flux session with srun with >= 128 tasks per node causes a segfault. It can be reproduced with a small node count, e.g.

$ salloc -N 8
$ srun --overcommit -N8 -n1024  src/cmd/flux start /bin/true
srun: error: jade751: tasks 580,588,596,604,612,620,628,636,644,652,660,668,676,684,692,700,708,716,724,732,740,748,756,764,772,780,788,796,812,820,828,836,852,860,868,876,884,892,900,908,916,924,932,948,956,964,972,980,988,996,1004,1012,1020: Segmentation fault
srun: error: jade752: tasks 581,589,597,605,613,621,629,637,645,653,661,669,677,685,693,701,709,717,725,733,741,749,757,765,773,781,789,797,813,821,829,837,853,861,869,877,885,893,901,909,917,925,933,949,957,965,973,981,989,997,1005,1013,1021: Segmentation fault
srun: error: jade754: tasks 583,591,599,607,615,623,631,639,647,655,663,671,679,687,695,703,711,719,727,735,743,751,759,767,775,783,791,799,815,823,831,839,855,863,871,879,887,895,903,911,919,927,935,951,959,967,975,983,991,999,1007,1015,1023: Segmentation fault
srun: error: jade749: tasks 578,586,594,602,610,618,626,634,642,650,658,666,674,682,690,698,706,714,722,730,738,746,754,762,770,778,786,794,810,818,826,834,850,858,866,874,882,890,898,906,914,922,930,946,954,962,970,978,986,994,1002,1010,1018: Segmentation fault
srun: error: jade748: tasks 577,585,593,601,609,617,625,633,641,649,657,665,673,681,689,697,705,713,721,729,737,745,753,761,769,777,785,793,809,817,833,849,857,865,873,881,889,897,905,913,921,929,937,945,953,961,969,977,985,993,1001,1009,1017: Segmentation fault
srun: error: jade753: tasks 582,590,598,606,614,622,630,638,646,654,662,670,678,686,694,702,710,718,726,734,742,750,758,766,774,782,790,798,814,822,830,838,854,862,870,878,886,894,902,910,918,926,934,950,958,966,974,982,990,998,1006,1014,1022: Segmentation fault
srun: error: jade750: tasks 579,587,595,603,611,619,627,635,643,651,659,667,675,683,691,699,707,715,723,731,739,747,755,763,771,779,787,795,811,819,827,835,851,859,867,875,883,891,899,907,915,923,931,947,955,963,971,979,987,995,1003,1011,1019: Segmentation fault
srun: error: jade747: tasks 576,584,592,600,608,616,624,632,640,648,656,664,672,680,688,696,704,712,720,728,736,744,752,760,768,776,784,792,808,816,832,848,856,864,872,880,888,896,904,912,920,928,936,944,952,960,968,976,984,992,1000,1008,1016: Segmentation fault
srun: error: jade754: task 807: Segmentation fault
srun: error: jade751: task 804: Segmentation fault
srun: error: jade752: task 805: Segmentation fault
srun: error: jade747: task 800: Segmentation fault
srun: error: jade753: task 806: Segmentation fault
srun: error: jade750: task 803: Segmentation fault
srun: error: jade749: task 802: Segmentation fault
srun: error: jade748: task 801: Segmentation fault
srun: First task exited 30s ago
srun: tasks 0-575,824-825,840-847,938-943: running
srun: tasks 576-823,826-839,848-937,944-1023: exited abnormally
srun: Terminating job step 1187101.3
slurmd[jade748]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade747]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade749]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade750]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade752]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade753]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade751]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade754]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
slurmd[jade747]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade754]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade752]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade749]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade751]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade750]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade753]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
slurmd[jade748]: *** STEP 1187101.3 KILLED AT 2016-08-12T13:18:11 WITH SIGNAL 9 ***
$

[garlick]

Oh even one node reproduces.

$ salloc -N1
salloc: Granted job allocation 1187102
$ srun --overcommit -N1 -n128 src/cmd/flux start /bin/true
srun: error: jade457: tasks 72-94,96-103,105-127: Segmentation fault
srun: First task exited 30s ago
srun: tasks 0-71,95,104: running
srun: tasks 72-94,96-103,105-127: exited abnormally
srun: Terminating job step 1187102.1
slurmd[jade457]: *** STEP 1187102.1 KILLED AT 2016-08-12T13:22:46 WITH SIGNAL 9 ***
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
slurmd[jade457]: *** STEP 1187102.1 KILLED AT 2016-08-12T13:22:46 WITH SIGNAL 9 ***
$

[grondo]

Hm, I can reproduce this on opal with srun -N1 --overcommit -n128 hostname -- not a flux issue I think

[dongahn]

Just FYI -- At least I don't plan to scale the overcommit factor that much as part of my testing. The max factor for #14 is 32.

[garlick]

@grondo found running with --mpibind=off seems to fix this.

[garlick]

Closing since this isn't a Flux bug.

[lipari]

This bug is fixed in the latest version of mpibind which has yet to make its way onto jade.