Run deadlocks

[Sameer-hui]

Hello,
I attempted to run my instance on 128 nodes with 512 processes. Although nekrs is running continuously, there is no log file output and no errors. However, I was able to successfully run and obtain results on 32 nodes with 128 processes and 64 nodes with 256 processes.

log：

loading udf kernels ... 
done (2.47983s)
benchmarking hot kernels ...
WARNING: kernel timings differ by up to 1.20 across ranks!
advSub: N=7 cubN=10 Nfields=3 wordSize=64 GDOF/s=2.20823 GB/s=198.634 GFLOPS/s=1228.38 kernelVer=0
advSub: N=7 cubN=10 Nfields=1 wordSize=64 GDOF/s=2.00579 GB/s=196.395 GFLOPS/s=1178.04 kernelVer=8
Ax: N=7 wordSize=64 GDOF/s=4.09956 GB/s=538.512 GFLOPS/s=715.976 constCoeff=0 poisson=0 kernelVer=7
Ax: N=7 wordSize=64 GDOF/s=5.5356 GB/s=528.835 GFLOPS/s=925.462 constCoeff=1 poisson=1 kernelVer=0
Ax: N=7 wordSize=32 GDOF/s=10.1478 GB/s=484.728 GFLOPS/s=1696.55 constCoeff=1 poisson=1 kernelVer=0
fdm: N=9 wordSize=32 GDOF/s=15.7129 GB/s=284.514 GFLOPS/s=2608.04 kernelVer=3
Ax: N=3 wordSize=32 GDOF/s=5.05884 GB/s=383.722 GFLOPS/s=767.445 constCoeff=1 poisson=1 kernelVer=2
fdm: N=5 wordSize=32 GDOF/s=13.9629 GB/s=337.791 GFLOPS/s=1761.34 kernelVer=3
Ax: Ndim=3 N=7 wordSize=64 GDOF/s=7.75223 GB/s=462.874 GFLOPS/s=1353.91 constCoeff=0 poisson=0 kernelVer=1
JIT compiling kernels (this may take awhile if they are not in cache) ...
done (246.745s)

initializing linAlg ...
wdotp: 1.827e-04s  (local: 1.567e-04s)

 reading test.re2                                                                                                                            
 hdr:#v004         2089704  3         2089704   1 hdr                                
 reading mesh 
   readp_re2_mesh:pack/cr/unpack : 0.19E-01 0.12E-02 0.91E-03
   readp_re2_mesh:byte_read_mpi  : 0.56E-01
 reading boundary faces               319812 for ifield   1
   readp_re2_bc:pack/cr/unpack : 0.13E-01 0.32E-02 0.38E-03
   readp_re2_bc:byte_read_mpi  : 0.40E-01
 done :: read .re2 file    0.15     sec

The log file stops outputting here, even though the program is still running, and there are no error messages.

par

[OCCA]
backend = HIP
deviceNumber= LOCAL-RANK

[MESH]
connectivityTol = 0.1

[GENERAL] 
#verbose=true
polynomialOrder = 7
#startFrom = "restart.fld"
stopAt = endTime
endTime = 5 
#dt = 4e-04
dt = targetCFL=1 + max=1e-04 + initial=1e-05
timeStepper = tombo2

#subCyclingSteps = 1

writeControl = simulationTime
writeInterval = 1

regularization = hpfrt + nModes=1 + scalingCoeff=10

[PRESSURE]
residualTol = 1e-04
#preconditioner = jac
initialGuess = projectionAconj+nVector=10

[VELOCITY]
boundaryTypeMap = v,O,W
density = 1.0
viscosity = 0.1
residualTol = 1e-06

[TEMPERATURE]
boundaryTypeMap = t,O,I
residualTol = 1e-06
rhoCp = 1200.0
conductivity = 0.025

I apologize if my expression is unclear. I would be happy to offer more details if necessary. Thank you for any guidance you can provide.

Yours sincerely,
sameer