Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Glusterfs brick processes are running very high CPU #2671

Closed
amgads opened this issue Jul 28, 2021 · 23 comments
Closed

Glusterfs brick processes are running very high CPU #2671

amgads opened this issue Jul 28, 2021 · 23 comments

Comments

@amgads
Copy link
Contributor

amgads commented Jul 28, 2021 •
edited
Loading

Description of problem:

We have a dynamic volume with 2 bricks processes (replica:2) running very high cpu exceeding 100%. The dynamic volume in K8s is used by two different pods. That was not the case with GlusterFS V6.5 -- when upgraded to release 7.8, we have seen this issue. This is a field issue so it's escalated.

The issue doesn't show with other pods/volumes. Collected the following profile data on the volume while load is running:
Appreciate your urgent attention:

gluster volume profile vol_7147326886af2706ee79f403c2998a3f start incremental;sleep 60
gluster volume profile vol_7147326886af2706ee79f403c2998a3f info
gluster volume profile vol_7147326886af2706ee79f403c2998a3f stop

gluster volume profile vol_7147326886af2706ee79f403c2998a3f start peek;sleep 60
gluster volume profile vol_7147326886af2706ee79f403c2998a3f info
gluster volume profile vol_7147326886af2706ee79f403c2998a3f stop

[root@vudmchcl01-control-01 MP1_lite]# gluster volume profile vol_7147326886af2706ee79f403c2998a3f start incremental
Starting volume profile on vol_7147326886af2706ee79f403c2998a3f has been successful
[root@vudmchcl01-control-01 MP1_lite]# sleep 65
[root@vudmchcl01-control-01 MP1_lite]# gluster volume profile vol_7147326886af2706ee79f403c2998a3f info
Brick: 172.27.2.35:/var/lib/heketi/mounts/vg_cedb863c562c710481e1bcd0ec691f42/brick_067a0b32180c9dd70d688c82ffaa6f6d/brick

Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 1 2 2
No. of Writes: 208 528 828290

Block Size: 16b+ 32b+ 64b+
No. of Reads: 60 916 3309
No. of Writes: 472257 693024 2489610

Block Size: 128b+ 256b+ 512b+
No. of Reads: 18937 7293 32175
No. of Writes: 2305389 538291 537133

Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 13423 18690 9187
No. of Writes: 558496 518944 410859

Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 20841 14409 16682
No. of Writes: 241198 102657 24286

Block Size: 65536b+ 131072b+
No. of Reads: 16709 83562
No. of Writes: 1725 7
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop

  0.00       0.00 us       0.00 us       0.00 us         728205      FORGET
  0.00       0.00 us       0.00 us       0.00 us        4746383     RELEASE
  0.00       0.00 us       0.00 us       0.00 us      671939117  RELEASEDIR
  0.00     835.33 us      92.39 us    8054.27 us            154     XATTROP
  0.00   39448.42 us     255.01 us  127163.69 us              6 REMOVEXATTR
  0.00   10603.69 us      77.98 us  279508.30 us             59    SETXATTR
  0.00     565.74 us      11.60 us   64885.12 us           9393    GETXATTR
  0.00    2139.27 us      45.66 us   77631.40 us          13870     READDIR
  0.00     243.24 us      17.31 us  149592.64 us         128471      STATFS
  0.01     239.35 us      24.00 us  133859.00 us         717817       FSTAT
  0.01     843.03 us      18.97 us  486483.65 us         263286        READ
  0.01   32835.74 us     131.74 us 3277624.94 us           7383      RENAME
  0.01     676.06 us      46.75 us  775224.60 us         423241       RMDIR
  0.02     227.04 us      20.58 us  423596.23 us        1957172        STAT
  0.04     234.73 us      41.19 us  431759.17 us        4416685        OPEN
  0.05     311.53 us       6.98 us  307541.25 us        4066656     ENTRYLK
  0.08     338.41 us       4.66 us  331411.00 us        5832165       FLUSH
  0.10     283.52 us       7.77 us  194616.77 us        8857316    FINODELK
  0.10    8901.19 us      36.39 us 1977575.53 us         295434      UNLINK
  0.15   12565.12 us     183.02 us 1751495.88 us         314880      CREATE
  0.18     483.36 us      33.44 us  414346.64 us        9692652       WRITE
  0.21     601.27 us      59.19 us  304489.24 us        8855891    FXATTROP
  0.22   12762.71 us     218.41 us 1416618.34 us         442805       MKDIR
  3.63     465.61 us      20.77 us 1851188.63 us      200383736      LOOKUP
  4.92     188.91 us       0.60 us 1198808.79 us      669909070     OPENDIR
 10.42     401.34 us       9.79 us 1327358.56 us      668262580    READDIRP
 30.19     340.48 us      29.97 us  598871.04 us     2282299154     SETATTR
 49.66     279.47 us       7.12 us  851452.14 us     4573341502     INODELK

Duration: 1388209 seconds

Data Read: 13984404450 bytes
Data Written: 11967854926 bytes

Interval 4 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 1
No. of Writes: 235 117 189

Block Size: 64b+ 128b+ 256b+
No. of Reads: 2 8 9
No. of Writes: 772 637 121

Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 25 12 9
No. of Writes: 129 167 118

Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 3 10 6
No. of Writes: 98 58 23

Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 14 8 189
No. of Writes: 5 0 0

%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop

  0.00       0.00 us       0.00 us       0.00 us            365      FORGET
  0.00       0.00 us       0.00 us       0.00 us           1455     RELEASE
  0.00       0.00 us       0.00 us       0.00 us         160492  RELEASEDIR
  0.00      72.55 us      72.52 us      72.58 us              2    GETXATTR
  0.00    8618.05 us    8618.05 us    8618.05 us              1    SETXATTR
  0.00     390.67 us      36.10 us    4602.81 us             31      STATFS
  0.00     123.72 us      35.10 us    1210.90 us            124       FSTAT
  0.01    8561.62 us     181.20 us   28340.64 us              5      RENAME
  0.02     233.24 us      36.82 us   10013.48 us            534        STAT
  0.04     719.96 us     120.64 us   10408.28 us            302       RMDIR
  0.04     216.94 us      52.91 us    8884.91 us           1237        OPEN
  0.11     279.17 us      13.14 us   19520.10 us           2427    FINODELK
  0.12    2313.19 us      27.30 us   65852.03 us            308        READ
  0.13     456.83 us      11.07 us   37927.76 us           1683       FLUSH
  0.16     367.86 us      12.15 us   10459.40 us           2608     ENTRYLK
  0.22     499.35 us      48.42 us   29579.63 us           2604       WRITE
  0.22    7817.25 us     276.22 us  108128.82 us            171      CREATE
  0.24    7026.55 us     145.53 us   75815.83 us            206      UNLINK
  0.24     600.40 us      82.56 us   27351.85 us           2427    FXATTROP
  0.35    8946.30 us     248.65 us  224750.23 us            235       MKDIR
  3.50     518.23 us      37.40 us  141889.91 us          40788      LOOKUP
  4.97     194.30 us      18.09 us   66899.15 us         154411     OPENDIR
 10.86     401.17 us      12.51 us   70969.09 us         163253    READDIRP
 29.78     340.25 us      37.42 us   54100.86 us         527960     SETATTR
 48.98     279.18 us       8.36 us   68422.45 us        1058209     INODELK

Duration: 335 seconds

Data Read: 26437792 bytes
Data Written: 2771470 bytes

Brick: 172.27.2.22:/var/lib/heketi/mounts/vg_e2e8b43d72ff0c7b572a8c1cb6494913/brick_50cc3f6362af27c0003e2f994916f9a8/brick

Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 2 3 3
No. of Writes: 208 528 828290

Block Size: 16b+ 32b+ 64b+
No. of Reads: 61 891 3386
No. of Writes: 472257 693024 2489610

Block Size: 128b+ 256b+ 512b+
No. of Reads: 18958 7148 31957
No. of Writes: 2305389 538291 537133

Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 13246 18522 9402
No. of Writes: 558496 518944 410858

Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 21209 14361 16881
No. of Writes: 241197 102657 24286

Block Size: 65536b+ 131072b+
No. of Reads: 16837 83932
No. of Writes: 1725 7
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop

  0.00       0.00 us       0.00 us       0.00 us         728592      FORGET
  0.00       0.00 us       0.00 us       0.00 us        4746385     RELEASE
  0.00       0.00 us       0.00 us       0.00 us      671939405  RELEASEDIR
  0.00   11480.63 us     128.00 us   28455.99 us              5 REMOVEXATTR
  0.00     502.31 us      57.70 us   25892.43 us            262     XATTROP
  0.00    3194.72 us      71.32 us   76561.62 us             58    SETXATTR
  0.00     308.23 us      10.93 us   37278.24 us           9484    GETXATTR
  0.00    1053.52 us     113.75 us   25935.31 us          13866     READDIR
  0.00     145.44 us      19.02 us   84677.64 us         128471      STATFS
  0.01     171.68 us      22.81 us  121298.45 us         718381       FSTAT
  0.01     535.48 us      17.59 us  829713.34 us         263760        READ
  0.01     462.97 us      43.31 us  816085.23 us         423252       RMDIR
  0.01   30365.04 us     123.52 us 2136739.26 us           7383      RENAME
  0.04     144.19 us      18.86 us  141258.60 us        4184277        STAT
  0.04     157.91 us       7.68 us  104635.51 us        4067348     ENTRYLK
  0.05     172.92 us      41.78 us  167082.07 us        4416688        OPEN
  0.05     144.91 us       4.35 us  117962.56 us        5832165       FLUSH
  0.09     161.88 us       7.47 us  179205.52 us        8857324    FINODELK
  0.13    7427.47 us      22.23 us 1627711.26 us         295458      UNLINK
  0.18     305.95 us      31.14 us  251819.61 us        9692652       WRITE
  0.19    9653.73 us     176.28 us 1519794.98 us         314880      CREATE
  0.21     387.48 us      59.01 us  217967.40 us        8855892    FXATTROP
  0.27    9957.14 us     209.27 us 2004439.88 us         442745       MKDIR
  4.00     326.14 us      18.06 us 2699538.43 us      200380351      LOOKUP
  5.29     129.03 us       0.61 us  908711.82 us      669909595     OPENDIR
 12.75     309.17 us       9.89 us 1210217.99 us      673114509    READDIRP
 32.32     231.17 us      28.58 us  434766.82 us     2282299958     SETATTR
 44.35     158.16 us       7.14 us  339313.55 us     4576990929     INODELK

Duration: 1388208 seconds

Data Read: 14055514436 bytes
Data Written: 11967837616 bytes

Interval 4 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 235 117 189

Block Size: 64b+ 128b+ 256b+
No. of Reads: 5 15 4
No. of Writes: 772 637 121

Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 24 14 4
No. of Writes: 129 167 118

Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7 20 13
No. of Writes: 98 58 23

Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1 8 64
No. of Writes: 5 0 0

%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop

  0.00       0.00 us       0.00 us       0.00 us            408      FORGET
  0.00       0.00 us       0.00 us       0.00 us           1455     RELEASE
  0.00       0.00 us       0.00 us       0.00 us         160485  RELEASEDIR
  0.00     171.09 us     171.09 us     171.09 us              1    SETXATTR
  0.00     363.76 us     363.76 us     363.76 us              1    GETXATTR
  0.00     330.98 us     137.64 us     524.32 us              2     XATTROP
  0.00     163.22 us      25.66 us    1745.76 us             31      STATFS
  0.00      94.01 us      31.51 us     872.33 us            141       FSTAT
  0.02     386.88 us      27.79 us   20857.63 us            190        READ
  0.03     326.98 us      61.81 us    1677.13 us            302       RMDIR
  0.04     151.34 us      25.79 us    6377.01 us            964        STAT
  0.06     161.11 us      51.74 us    4608.57 us           1237        OPEN
  0.06     127.08 us       9.47 us    5066.20 us           1683       FLUSH
  0.10     142.09 us      12.56 us   10036.93 us           2428    FINODELK
  0.10   72022.40 us     233.69 us  349922.05 us              5      RENAME
  0.13     173.95 us      10.91 us   12541.25 us           2614     ENTRYLK
  0.23     317.48 us      41.49 us   20713.45 us           2603       WRITE
  0.24     358.72 us      75.33 us   11169.99 us           2427    FXATTROP
  0.32    6756.12 us     230.18 us   91396.32 us            171      CREATE
  0.36    6365.17 us     130.23 us   89179.46 us            206      UNLINK
  0.49    7570.18 us     262.84 us  148262.72 us            234       MKDIR
  3.79     337.71 us      33.16 us  110642.99 us          40631      LOOKUP
  5.25     123.21 us      18.41 us   41828.25 us         154423     OPENDIR
 12.30     305.29 us      12.76 us   77316.40 us         145939    READDIRP
 32.60     223.67 us      35.38 us   95414.41 us         528014     SETATTR
 43.90     150.15 us       9.66 us   75168.80 us        1059141     INODELK

Duration: 335 seconds

Data Read: 9743802 bytes
Data Written: 2771470 bytes

[root@vudmchcl01-control-01 MP1_lite]#
[root@vudmchcl01-control-01 MP1_lite]# gluster volume profile vol_7147326886af2706ee79f403c2998a3f stop
Stopping volume profile on vol_7147326886af2706ee79f403c2998a3f has been successful
[root@vudmchcl01-control-01 MP1_lite]#

[root@vudmchcl01-control-01 MP1_lite]# gluster volume info vol_7147326886af2706ee79f403c2998a3f

Volume Name: vol_7147326886af2706ee79f403c2998a3f
Type: Replicate
Volume ID: 88b830ec-7ba8-468d-a3ad-62aac69ba7cf
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 172.27.2.35:/var/lib/heketi/mounts/vg_cedb863c562c710481e1bcd0ec691f42/brick_067a0b32180c9dd70d688c82ffaa6f6d/brick
Brick2: 172.27.2.22:/var/lib/heketi/mounts/vg_e2e8b43d72ff0c7b572a8c1cb6494913/brick_50cc3f6362af27c0003e2f994916f9a8/brick
Options Reconfigured:
cluster.eager-lock: on
user.heketi.id: 7147326886af2706ee79f403c2998a3f
storage.health-check-timeout: 20
storage.health-check-interval: 60
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
[root@vudmchcl01-control-01 MP1_lite]#

[root@vudmchcl01-control-01 MP1_lite]# gluster volume status vol_7147326886af2706ee79f403c2998a3f
Status of volume: vol_7147326886af2706ee79f403c2998a3f
Gluster process TCP Port RDMA Port Online Pid

Brick 172.27.2.35:/var/lib/heketi/mounts/vg
_cedb863c562c710481e1bcd0ec691f42/brick_067
a0b32180c9dd70d688c82ffaa6f6d/brick 49207 0 Y 14281
Brick 172.27.2.22:/var/lib/heketi/mounts/vg
_e2e8b43d72ff0c7b572a8c1cb6494913/brick_50c
c3f6362af27c0003e2f994916f9a8/brick 49206 0 Y 28758
Self-heal Daemon on localhost N/A N/A Y 20275
Self-heal Daemon on 172.27.2.18 N/A N/A Y 18237
Self-heal Daemon on 172.27.2.22 N/A N/A Y 17829

Task Status of Volume vol_7147326886af2706ee79f403c2998a3f

There are no active volume tasks

[root@vudmchcl01-control-01 MP1_lite]#

[root@vudmchcl01-control-01 MP1_lite]# gluster volume heal vol_7147326886af2706ee79f403c2998a3f info
Brick 172.27.2.35:/var/lib/heketi/mounts/vg_cedb863c562c710481e1bcd0ec691f42/brick_067a0b32180c9dd70d688c82ffaa6f6d/brick
Status: Connected
Number of entries: 0

Brick 172.27.2.22:/var/lib/heketi/mounts/vg_e2e8b43d72ff0c7b572a8c1cb6494913/brick_50cc3f6362af27c0003e2f994916f9a8/brick
/logs/vudmcl01ch01/2021_07_24T04_00_00
Status: Connected
Number of entries: 1

[root@vudmchcl01-control-01 MP1_lite]#

Expected results:
Running at 10% CPU

- The operating system / glusterfs version: CentOs-7

Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration
@amgads
Copy link
Contributor Author

amgads commented Jul 28, 2021

Any know issue with high CPU for release 7.8+?
Any other data is needed?

@mohit84
Copy link
Contributor

mohit84 commented Jul 31, 2021

I don't think any issue was reported in release-7.8+ specific to high CPU consumption, Can you please confirm what is workload in your environment ? Is it possible to capture continuous pstack while a brick process is consuming high CPU and share the brick logs also to debug it more.

@amgads
Copy link
Contributor Author

amgads commented Jul 31, 2021

Thanks Mohit -- this is a customer system, so I'll ask to collect the data in the lab which has the same issue.

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

@mohit84 Thanks for looking into the issue. We got the data from the customer lab:
1). The volume has 2 bricks running in k8s -- Here's "gluster heal info" output:

gluster volume heal vol_bc456156109bd5bffe9d060d21d7265f info

Brick 172.27.2.32:/var/lib/heketi/mounts/vg_5f52c63bee2bb0909dc1c81ff40683e1/brick_9ddff301dec7365e0a3e820540ee3b2b/brick
Status: Connected
Number of entries: 0

Brick 172.27.2.42:/var/lib/heketi/mounts/vg_7bc0c66ed7b35088cedfa56e802005e1/brick_dfefa1d7202794b7877a070d958b6b64/brick
Status: Connected
Number of entries: 0

2). For Brick-1, here's the pstack output:
while true; do pstack 17756 | tee pstack_1_dump.txt; echo "###########################NEXT PSTACK DUMP#######################################"| tee pstack_1_dump.txt ;sleep 5;done
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae560477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ba21271b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8da661e in server4_0_releasedir () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6b9f56b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#9 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b887f285 in __getdents64 () from /lib64/libc.so.6
#1 0x00007fb6b887ee81 in readdir64 () from /lib64/libc.so.6
#2 0x00007fb6aba8cdcb in posix_fill_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6aba8d747 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007fb6aba8dfa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#6 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#7 0x00007fb6aabb47ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#8 0x00007fb6aa99fa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#9 0x00007fb6aa761a8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#10 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#11 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#12 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#13 0x00007fb6a9f0dd69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#14 0x00007fb6ba2576b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#16 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#17 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b88ca65c in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x00007fb6b884647c in _L_lock_5314 () from /lib64/libc.so.6
#2 0x00007fb6b883f078 in _int_free () from /lib64/libc.so.6
#3 0x00007fb6ba1e0ee3 in iobref_destroy () from /lib64/libglusterfs.so.0
#4 0x00007fb6b9f560aa in rpcsvc_request_destroy () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f562f5 in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae560477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ba21271b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8da661e in server4_0_releasedir () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6b9f56b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#9 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8da661e in server4_0_releasedir () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6b9f56b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#9 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8db4c26 in server4_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905ca02 in io_stats_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d0913 in marker_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23969c in default_readdirp_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f051cf in up_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa76a435 in pl_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007fb6aa9a348f in posix_acl_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#14 0x00007fb6aabb9336 in br_stub_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#15 0x00007fb6aba8d899 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#16 0x00007fb6aba8dfa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#19 0x00007fb6aabb47ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#20 0x00007fb6aa99fa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa761a8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#23 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#24 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#25 0x00007fb6a9f0dd69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba2576b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b88b3430 in writev () from /lib64/libc.so.6
#1 0x00007fb6ba1de869 in sys_writev () from /lib64/libglusterfs.so.0
#2 0x00007fb6ae55b0c9 in __socket_rwv () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#3 0x00007fb6ae55dbfe in __socket_ioq_churn_entry.isra.13 () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ae55dfa2 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#5 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#6 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#7 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#9 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#10 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#14 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#19 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#20 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#21 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b8ffd5ea in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff9f9e in _L_unlock_738 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff9f10 in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcd5 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b88ad365 in _lxstat () from /lib64/libc.so.6
#1 0x00007fb6ba1ddab4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007fb6aba66dc7 in posix_handle_path () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6aba8d303 in posix_readdirp_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007fb6aba8dca3 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007fb6aba8dfa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#6 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#7 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#8 0x00007fb6aabb47ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#9 0x00007fb6aa99fa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#10 0x00007fb6aa761a8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#12 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#13 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#14 0x00007fb6a9f0dd69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#15 0x00007fb6ba2576b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#18 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8fffd3c in _L_cond_lock_847 () from /lib64/libpthread.so.0
#2 0x00007fb6b8fffbd1 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#3 0x00007fb6b8ffaeb4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffb0f6 in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf83c9 in do_iot_schedule () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6a9cf9866 in iot_inodelk () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#3 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#4 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#5 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#6 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#7 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#8 0x00007fb6a904aa7a in io_stats_inodelk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a8dbd23d in server4_inodelk_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#10 0x00007fb6a8d7dbe6 in server_resolve_done () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#11 0x00007fb6a8d7dc75 in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#12 0x00007fb6a8d7e7cc in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#13 0x00007fb6a8d7dcbe in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#14 0x00007fb6a8d7e52b in server_resolve_inode () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#15 0x00007fb6a8d7e7f8 in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#16 0x00007fb6a8d7dc9e in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#17 0x00007fb6a8d7e874 in resolve_and_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#18 0x00007fb6a8db81e1 in server4_0_inodelk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#19 0x00007fb6b9f56b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b88ad820 in __openat_nocancel () from /lib64/libc.so.6
#1 0x00007fb6b887ed85 in __opendirat () from /lib64/libc.so.6
#2 0x00007fb6aba7ca4c in posix_opendir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6ba2402cf in default_opendir () from /lib64/libglusterfs.so.0
#4 0x00007fb6ba2402cf in default_opendir () from /lib64/libglusterfs.so.0
#5 0x00007fb6aabb3b8a in br_stub_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#6 0x00007fb6aa99d7c5 in posix_acl_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#7 0x00007fb6aa760d1e in pl_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#8 0x00007fb6ba2402cf in default_opendir () from /lib64/libglusterfs.so.0
#9 0x00007fb6ba2402cf in default_opendir () from /lib64/libglusterfs.so.0
#10 0x00007fb6ba2402cf in default_opendir () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f127f2 in up_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6ba259d8b in default_opendir_resume () from /lib64/libglusterfs.so.0
#13 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#14 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#15 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b88ad365 in _lxstat () from /lib64/libc.so.6
#1 0x00007fb6ba1ddab4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007fb6aba66dc7 in posix_handle_path () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6aba6005e in posix_istat () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007fb6aba78ffa in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#6 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#7 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#8 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#9 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#10 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#11 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#12 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#14 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#16 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#17 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8fffd3c in _L_cond_lock_847 () from /lib64/libpthread.so.0
#2 0x00007fb6b8fffbd1 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#3 0x00007fb6b8ffaeb4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8fffd3c in _L_cond_lock_847 () from /lib64/libpthread.so.0
#2 0x00007fb6b8fffbd1 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#3 0x00007fb6b8ffaeb4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ba1c36a9 in inode_unref () from /lib64/libglusterfs.so.0
#4 0x00007fb6a8d7f328 in server_loc_wipe () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#5 0x00007fb6a8d7f42d in free_state () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#6 0x00007fb6a8d79ebe in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae560477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ba21271b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b88b3430 in writev () from /lib64/libc.so.6
#1 0x00007fb6ba1de869 in sys_writev () from /lib64/libglusterfs.so.0
#2 0x00007fb6ae55b0c9 in __socket_rwv () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#3 0x00007fb6ae55dbfe in __socket_ioq_churn_entry.isra.13 () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ae55dfa2 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#5 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#6 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#7 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#9 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#10 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#11 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#12 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#13 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#14 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#15 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#16 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#17 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#18 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#19 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#20 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#21 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#22 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#23 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#24 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#25 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#26 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#27 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#29 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#30 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#31 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae560477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ba21271b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b88ad607 in chmod () from /lib64/libc.so.6
#1 0x00007fb6ba1de2f9 in sys_chmod () from /lib64/libglusterfs.so.0
#2 0x00007fb6aba79dce in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#4 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#5 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#6 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#7 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#8 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#9 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#10 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#11 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#13 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#14 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#15 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b88b3430 in writev () from /lib64/libc.so.6
#1 0x00007fb6ba1de869 in sys_writev () from /lib64/libglusterfs.so.0
#2 0x00007fb6ae55b0c9 in __socket_rwv () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#3 0x00007fb6ae55dbfe in __socket_ioq_churn_entry.isra.13 () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ae55dfa2 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#5 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#6 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#7 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#9 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#10 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#14 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#19 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#20 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#21 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6ba1dc276 in gf_mem_set_acct_info () from /lib64/libglusterfs.so.0
#1 0x00007fb6a8da9a1e in dict_to_xdr () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#2 0x00007fb6a8db18bf in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#3 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#4 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#5 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#6 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#7 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#8 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#9 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#10 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#11 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#12 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#13 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#14 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#15 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#16 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#17 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#18 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#19 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#20 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#21 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#22 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#23 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#24 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#25 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b887f285 in __getdents64 () from /lib64/libc.so.6
#1 0x00007fb6b887ee81 in readdir64 () from /lib64/libc.so.6
#2 0x00007fb6aba8cdcb in posix_fill_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6aba8d747 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007fb6aba8dfa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#6 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#7 0x00007fb6aabb47ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#8 0x00007fb6aa99fa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#9 0x00007fb6aa761a8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#10 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#11 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#12 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#13 0x00007fb6a9f0dd69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#14 0x00007fb6ba2576b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#16 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#17 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7fb6b177b700 (LWP 17757)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1c19a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fb6b0f7a700 (LWP 17758)):
#0 0x00007fb6b8ffe3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056088fb43013 in glusterfs_sigwaiter ()
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fb6b0779700 (LWP 17759)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6ba1dc386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fb6aff78700 (LWP 17760)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fb6af777700 (LWP 17761)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6ba1f06a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fb6ba1f1310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fb6aef76700 (LWP 17762)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6ba229844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fb6accac700 (LWP 17763)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae560477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ba21271b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fb6ac4ab700 (LWP 17764)):
#0 0x00007fb6b88bcf43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb6ba2123d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fb6a88fe700 (LWP 17793)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a94ab9f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fb6a87fd700 (LWP 17794)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb6a36ff700 (LWP 17797)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aaff9a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb6a2efe700 (LWP 17798)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb6a26fd700 (LWP 17799)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb6a1efc700 (LWP 17800)):
#0 0x00007fb6b88b39a3 in select () from /lib64/libc.so.6
#1 0x00007fb6aaff9cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb6a16fb700 (LWP 17803)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63a40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb6a0efa700 (LWP 17804)):
#0 0x00007fb6b888385d in nanosleep () from /lib64/libc.so.6
#1 0x00007fb6b88836f4 in sleep () from /lib64/libc.so.6
#2 0x00007fb6aba63382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb69bfff700 (LWP 17805)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba5e754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb69b7fe700 (LWP 17806)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6aba63c43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007fb6aba63eb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb69affd700 (LWP 17825)):
#0 0x00007fb6b8ffaa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6b9f56bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb69a7fc700 (LWP 19241)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8da661e in server4_0_releasedir () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6b9f56b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#9 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb6a8102700 (LWP 28236)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcaa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb6a80c1700 (LWP 28559)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ba1c36a9 in inode_unref () from /lib64/libglusterfs.so.0
#4 0x00007fb6a8d7f328 in server_loc_wipe () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#5 0x00007fb6a8d7f42d in free_state () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#6 0x00007fb6a8d79ebe in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb6a8080700 (LWP 28798)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb6a06f9700 (LWP 29192)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb6a06b8700 (LWP 30031)):
#0 0x00007fb6b88b3430 in writev () from /lib64/libc.so.6
#1 0x00007fb6ba1de869 in sys_writev () from /lib64/libglusterfs.so.0
#2 0x00007fb6ae55b0c9 in __socket_rwv () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#3 0x00007fb6ae55dbfe in __socket_ioq_churn_entry.isra.13 () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6ae55dfa2 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#5 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#6 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#7 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a8db19a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#9 0x00007fb6a905d381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#10 0x00007fb6a98d6c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#11 0x00007fb6ba23afec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#12 0x00007fb6a9f05793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#13 0x00007fb6aa12437c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#14 0x00007fb6aa774823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#15 0x00007fb6aa9a37c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#16 0x00007fb6aafe4d62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#17 0x00007fb6aba793fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#18 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#19 0x00007fb6aafea657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#20 0x00007fb6ba23fd07 in default_setattr () from /lib64/libglusterfs.so.0
#21 0x00007fb6aa9a5000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#22 0x00007fb6aa7648a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#23 0x00007fb6aa54b9bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#24 0x00007fb6aa33b67b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#25 0x00007fb6aa1293b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#26 0x00007fb6a9f0e7f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#27 0x00007fb6ba258fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#29 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#30 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#31 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb6a0677700 (LWP 30099)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb6a0636700 (LWP 30100)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb6a05f5700 (LWP 30136)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fb6a05b4700 (LWP 25718)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6a9cf7cd8 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#4 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb6a0573700 (LWP 26434)):
#0 0x00007fb6b8ffd5ea in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff9f9e in _L_unlock_738 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff9f10 in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00007fb6ae55dcd5 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007fb6b9f5458a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007fb6b9f5626a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007fb6a8d79d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007fb6a8dbce90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a9065349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6ba234ab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007fb6aa785f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007fb6aa7868e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007fb6aa5457b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007fb6aa33b114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007fb6ba23d374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007fb6ba252b5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007fb6ba1d8f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb6a0532700 (LWP 26450)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb6a04f1700 (LWP 26765)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb6a04b0700 (LWP 26766)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb6a046f700 (LWP 26767)):
#0 0x00007fb6b8ffade2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb6a9cf7d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb6a042e700 (LWP 26917)):
#0 0x00007fb6b8ffd54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb6b8ff8e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007fb6b8ff8d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fb6ba1c3729 in inode_table_prune () from /lib64/libglusterfs.so.0
#4 0x00007fb6ba1c4228 in inode_link () from /lib64/libglusterfs.so.0
#5 0x00007fb6ba1dd8c5 in gf_link_inode_from_dirent () from /lib64/libglusterfs.so.0
#6 0x00007fb6ba1dd92e in gf_link_inodes_from_dirent () from /lib64/libglusterfs.so.0
#7 0x00007fb6a8db4bf4 in server4_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007fb6a905ca02 in io_stats_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007fb6a98d0913 in marker_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007fb6ba23969c in default_readdirp_cbk () from /lib64/libglusterfs.so.0
#11 0x00007fb6a9f051cf in up_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007fb6aa76a435 in pl_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007fb6aa9a348f in posix_acl_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#14 0x00007fb6aabb9336 in br_stub_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#15 0x00007fb6aba8d899 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#16 0x00007fb6aba8dfa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#18 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#19 0x00007fb6aabb47ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#20 0x00007fb6aa99fa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007fb6aa761a8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#23 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#24 0x00007fb6ba23f257 in default_readdirp () from /lib64/libglusterfs.so.0
#25 0x00007fb6a9f0dd69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007fb6ba2576b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#27 0x00007fb6ba1d94d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007fb6a9cf7dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007fb6b8ff6ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb6b88bc96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb6ba6b44c0 (LWP 17756)):
#0 0x00007fb6b8ff8017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb6ba211938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056088fb3f37b in main ()

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

3). For the 2nd brick -- here's the pstack dump

while true; do pstack 15748 | tee pstack_2_dump.txt; echo "###########################NEXT PSTACK DUMP#######################################"| tee pstack_2_dump.txt ;sleep 5;done

Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb602d1d in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f67fc7e61fb in gf_mem_set_acct_info () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7b1abb in dict_set_lk () from /lib64/libglusterfs.so.0
#3 0x00007f67fc7b1e32 in dict_setn () from /lib64/libglusterfs.so.0
#4 0x00007f67fc7b527c in dict_set_uint64 () from /lib64/libglusterfs.so.0
#5 0x00007f67e7b2dbb1 in xdr_to_dict () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#6 0x00007f67e7b3f6c0 in server4_0_lookup () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67fc560b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#8 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67e7de4210 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#1 0x00007f67ec6e1c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#2 0x00007f67fc844fec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#3 0x00007f67ecd10793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#4 0x00007f67ecf2f37c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#5 0x00007f67ed57f823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#6 0x00007f67ed7ae7c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#7 0x00007f67eddefd62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#8 0x00007f67ee8843fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#9 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#10 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#11 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#12 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#13 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#15 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#16 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#17 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#18 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#19 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#20 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#21 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb609d3c in _L_cond_lock_847 () from /lib64/libpthread.so.0
#2 0x00007f67fb609bd1 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#3 0x00007f67fb604eb4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67ed590bbf in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#4 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#5 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#6 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#7 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#8 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#9 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#10 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#11 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#12 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#13 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67ed590bbf in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#4 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#5 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#6 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#7 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#8 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#9 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#10 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#11 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#12 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#13 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67ed590bbf in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#4 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#5 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#6 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#7 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#8 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#9 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#10 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#11 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#12 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#13 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67fae1060e in vfprintf () from /lib64/libc.so.6
#1 0x00007f67faeddd18 in __vsprintf_chk () from /lib64/libc.so.6
#2 0x00007f67faeddc6d in __sprintf_chk () from /lib64/libc.so.6
#3 0x00007f67fc7c692e in lkowner_utoa () from /lib64/libglusterfs.so.0
#4 0x00007f67ed591804 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#5 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#6 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#7 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#8 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#9 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#10 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#11 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#12 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#13 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#14 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb6075ea in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x00007f67fb603f9e in _L_unlock_738 () from /lib64/libpthread.so.0
#2 0x00007f67fb603f10 in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00007f67ecb033db in do_iot_schedule () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#4 0x00007f67ecb04866 in iot_inodelk () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#6 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#7 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#8 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#9 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#10 0x00007f67e7dd1a7a in io_stats_inodelk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#11 0x00007f67e7b4423d in server4_inodelk_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#12 0x00007f67e7b04be6 in server_resolve_done () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#13 0x00007f67e7b04c75 in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#14 0x00007f67e7b057cc in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#15 0x00007f67e7b04cbe in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#16 0x00007f67e7b0552b in server_resolve_inode () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#17 0x00007f67e7b057f8 in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#18 0x00007f67e7b04c9e in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#19 0x00007f67e7b05874 in resolve_and_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#20 0x00007f67e7b3f1e1 in server4_0_inodelk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#21 0x00007f67fc560b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#22 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67e7b43e91 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#1 0x00007f67e7dec349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#2 0x00007f67fc83eab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#3 0x00007f67ed590f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#4 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#5 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#6 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#7 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#8 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#9 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#10 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#11 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#12 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#13 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67faeb7607 in chmod () from /lib64/libc.so.6
#1 0x00007f67fc7e82f9 in sys_chmod () from /lib64/libglusterfs.so.0
#2 0x00007f67ee884dce in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#4 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#5 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#6 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#7 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#8 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#9 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#10 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#11 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#13 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#14 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#15 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb6075ea in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x00007f67fb603f9e in _L_unlock_738 () from /lib64/libpthread.so.0
#2 0x00007f67fb603f10 in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00007f67ecb033db in do_iot_schedule () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#4 0x00007f67ecb0359c in iot_setattr () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#6 0x00007f67ec6da0fe in marker_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#7 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#8 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#9 0x00007f67ec091ede in quota_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/quota.so
#10 0x00007f67e7dd4221 in io_stats_setattr () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#11 0x00007f67e7b38bf3 in server4_setattr_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#12 0x00007f67e7b04be6 in server_resolve_done () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#13 0x00007f67e7b04c75 in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#14 0x00007f67e7b057cc in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#15 0x00007f67e7b04cbe in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#16 0x00007f67e7b0552b in server_resolve_inode () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#17 0x00007f67e7b057f8 in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#18 0x00007f67e7b04c9e in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#19 0x00007f67e7b05874 in resolve_and_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#20 0x00007f67e7b3d6a0 in server4_0_setattr () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#21 0x00007f67fc560b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#22 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb603f0e in pthread_mutex_unlock () from /lib64/libpthread.so.0
#1 0x00007f67ed59120b in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#2 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#3 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#4 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#5 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#6 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#7 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#8 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#9 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#10 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#11 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67faec5517 in llistxattr () from /lib64/libc.so.6
#1 0x00007f67fc7e90b9 in sys_llistxattr () from /lib64/libglusterfs.so.0
#2 0x00007f67ee86bdb4 in posix_xattr_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee898217 in posix_entry_xattr_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67ee8983ed in posix_readdirp_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007f67ee898ca3 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#6 0x00007f67ee898fa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#7 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#8 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#9 0x00007f67ed9bf7ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#10 0x00007f67ed7aaa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#11 0x00007f67ed56ca8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#13 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#14 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#15 0x00007f67ecd18d69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#16 0x00007f67fc8616b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#17 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#18 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#19 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b67caa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67e7b389a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7de4381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007f67ec6e1c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007f67fc844fec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007f67ecd10793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007f67ecf2f37c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007f67ed57f823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007f67ed7ae7c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007f67eddefd62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007f67ee8843fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b6a477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc81c71b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb6050f6 in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb033c9 in do_iot_schedule () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67ecb0359c in iot_setattr () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#3 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#4 0x00007f67ec6da0fe in marker_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#5 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#6 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#7 0x00007f67ec091ede in quota_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/quota.so
#8 0x00007f67e7dd4221 in io_stats_setattr () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007f67e7b38bf3 in server4_setattr_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#10 0x00007f67e7b04be6 in server_resolve_done () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#11 0x00007f67e7b04c75 in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#12 0x00007f67e7b057cc in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#13 0x00007f67e7b04cbe in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#14 0x00007f67e7b0552b in server_resolve_inode () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#15 0x00007f67e7b057f8 in server_resolve () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#16 0x00007f67e7b04c9e in server_resolve_all () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#17 0x00007f67e7b05874 in resolve_and_resume () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#18 0x00007f67e7b3d6a0 in server4_0_setattr () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#19 0x00007f67fc560b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#20 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67faebd430 in writev () from /lib64/libc.so.6
#1 0x00007f67fc7e8869 in sys_writev () from /lib64/libglusterfs.so.0
#2 0x00007f67f0b650c9 in __socket_rwv () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#3 0x00007f67f0b67bfe in __socket_ioq_churn_entry.isra.13 () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67f0b67fa2 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#5 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#6 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#7 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7b389a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#9 0x00007f67e7de4381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#10 0x00007f67ec6e1c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#11 0x00007f67fc844fec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#12 0x00007f67ecd10793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#13 0x00007f67ecf2f37c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#14 0x00007f67ed57f823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#15 0x00007f67ed7ae7c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#16 0x00007f67eddefd62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#17 0x00007f67ee8843fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#18 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#19 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#20 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#21 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#22 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#23 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#24 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#25 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#26 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#27 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#28 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#29 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#30 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#31 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67faeb7365 in _lxstat () from /lib64/libc.so.6
#1 0x00007f67fc7e7ab4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007f67ee86b0ac in posix_istat () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee883ffa in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#5 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#6 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#7 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#8 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#9 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#10 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#11 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#12 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#13 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#14 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#15 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#16 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fae1447e in vfprintf () from /lib64/libc.so.6
#1 0x00007f67faeddd18 in __vsprintf_chk () from /lib64/libc.so.6
#2 0x00007f67faeddc6d in __sprintf_chk () from /lib64/libc.so.6
#3 0x00007f67fc7c692e in lkowner_utoa () from /lib64/libglusterfs.so.0
#4 0x00007f67ed591804 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#5 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#6 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#7 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#8 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#9 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#10 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#11 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#12 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#13 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#14 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67faeb7365 in _lxstat () from /lib64/libc.so.6
#1 0x00007f67fc7e7ab4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007f67ee86b655 in posix_pstat () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee887f9a in posix_opendir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#5 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#6 0x00007f67ed9beb8a in br_stub_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#7 0x00007f67ed7a87c5 in posix_acl_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#8 0x00007f67ed56bd1e in pl_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#9 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#10 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#11 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#12 0x00007f67ecd1d7f2 in up_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#13 0x00007f67fc863d8b in default_opendir_resume () from /lib64/libglusterfs.so.0
#14 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#15 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#16 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67faec54ea in lgetxattr () from /lib64/libc.so.6
#1 0x00007f67fc7e9129 in sys_lgetxattr () from /lib64/libglusterfs.so.0
#2 0x00007f67ee8691bb in _posix_xattr_get_set_from_backend () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee86ca01 in _posix_xattr_get_set () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67fc7b3b57 in dict_foreach_match () from /lib64/libglusterfs.so.0
#5 0x00007f67fc7b3d38 in dict_foreach () from /lib64/libglusterfs.so.0
#6 0x00007f67ee86bdcb in posix_xattr_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#7 0x00007f67ee8770ba in posix_lookup () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#8 0x00007f67fc848315 in default_lookup () from /lib64/libglusterfs.so.0
#9 0x00007f67fc848315 in default_lookup () from /lib64/libglusterfs.so.0
#10 0x00007f67ed9c1777 in br_stub_lookup () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#11 0x00007f67ed7a70d9 in posix_acl_lookup () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#12 0x00007f67ed56c3ec in pl_lookup () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007f67fc848315 in default_lookup () from /lib64/libglusterfs.so.0
#14 0x00007f67fc848315 in default_lookup () from /lib64/libglusterfs.so.0
#15 0x00007f67fc848315 in default_lookup () from /lib64/libglusterfs.so.0
#16 0x00007f67ecd1b883 in up_lookup () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#17 0x00007f67fc85efda in default_lookup_resume () from /lib64/libglusterfs.so.0
#18 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67faeb7365 in _lxstat () from /lib64/libc.so.6
#1 0x00007f67fc7e7ab4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007f67ee86b655 in posix_pstat () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee89852b in posix_readdirp_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67ee898ca3 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007f67ee898fa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#6 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#7 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#8 0x00007f67ed9bf7ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#9 0x00007f67ed7aaa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#10 0x00007f67ed56ca8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#12 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#13 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#14 0x00007f67ecd18d69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#15 0x00007f67fc8616b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#16 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#17 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#18 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fae10a52 in vfprintf () from /lib64/libc.so.6
#1 0x00007f67faedde55 in __vsnprintf_chk () from /lib64/libc.so.6
#2 0x00007f67fc7e6958 in gf_vasprintf () from /lib64/libglusterfs.so.0
#3 0x00007f67fc7e6a64 in gf_asprintf () from /lib64/libglusterfs.so.0
#4 0x00007f67fc7b222c in data_from_int64 () from /lib64/libglusterfs.so.0
#5 0x00007f67fc7b4c06 in dict_set_int64 () from /lib64/libglusterfs.so.0
#6 0x00007f67e7b2dbd1 in xdr_to_dict () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67e7b2f45f in server4_0_readdirp () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67fc560b1c in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#9 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b67caa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67e7b3bc26 in server4_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7de3a02 in io_stats_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007f67ec6db913 in marker_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007f67fc84369c in default_readdirp_cbk () from /lib64/libglusterfs.so.0
#11 0x00007f67ecd101cf in up_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007f67ed575435 in pl_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007f67ed7ae48f in posix_acl_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#14 0x00007f67ed9c4336 in br_stub_readdirp_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#15 0x00007f67ee898899 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#16 0x00007f67ee898fa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#18 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#19 0x00007f67ed9bf7ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#20 0x00007f67ed7aaa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007f67ed56ca8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#23 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#24 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#25 0x00007f67ecd18d69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007f67fc8616b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#27 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67faec557a in lsetxattr () from /lib64/libc.so.6
#1 0x00007f67fc7e9039 in sys_lsetxattr () from /lib64/libglusterfs.so.0
#2 0x00007f67ee89cbc3 in posix_store_mdata_xattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee89cffe in posix_set_mdata_xattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67ee89debc in posix_set_ctime () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007f67ee884148 in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#6 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#7 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#8 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#9 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#10 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#12 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#13 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#14 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#15 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#16 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#17 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#18 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb609d3c in _L_cond_lock_847 () from /lib64/libpthread.so.0
#2 0x00007f67fb609bd1 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#3 0x00007f67fb604eb4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67ecb02cd8 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#4 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 37 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 36 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b6a477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc81c71b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b6a477 in socket_event_handler () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc81c71b in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#5 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86df42 in posix_fs_health_check () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee86e39a in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b67caa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67e7b389a2 in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7de4381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007f67ec6e1c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#10 0x00007f67fc844fec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#11 0x00007f67ecd10793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#12 0x00007f67ecf2f37c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#13 0x00007f67ed57f823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007f67ed7ae7c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007f67eddefd62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#16 0x00007f67ee8843fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#18 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#19 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#20 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#23 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#24 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#25 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#27 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b67caa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67e7b43e90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7dec349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007f67fc83eab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007f67ed590f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67faebd430 in writev () from /lib64/libc.so.6
#1 0x00007f67fc7e8869 in sys_writev () from /lib64/libglusterfs.so.0
#2 0x00007f67f0b650c9 in __socket_rwv () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#3 0x00007f67f0b67bfe in __socket_ioq_churn_entry.isra.13 () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67f0b67fa2 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#5 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#6 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#7 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7b35f24 in server4_opendir_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#9 0x00007f67e7de64d0 in io_stats_opendir_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#10 0x00007f67fc845dfc in default_opendir_cbk () from /lib64/libglusterfs.so.0
#11 0x00007f67fc845dfc in default_opendir_cbk () from /lib64/libglusterfs.so.0
#12 0x00007f67ecd13275 in up_opendir_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#13 0x00007f67ed573018 in pl_opendir_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#14 0x00007f67fc845dfc in default_opendir_cbk () from /lib64/libglusterfs.so.0
#15 0x00007f67fc845dfc in default_opendir_cbk () from /lib64/libglusterfs.so.0
#16 0x00007f67ee887cad in posix_opendir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#17 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#18 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#19 0x00007f67ed9beb8a in br_stub_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#20 0x00007f67ed7a87c5 in posix_acl_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#21 0x00007f67ed56bd1e in pl_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#22 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#23 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#24 0x00007f67fc84a2cf in default_opendir () from /lib64/libglusterfs.so.0
#25 0x00007f67ecd1d7f2 in up_opendir () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#26 0x00007f67fc863d8b in default_opendir_resume () from /lib64/libglusterfs.so.0
#27 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#28 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#29 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b67caa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67e7b43e90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7dec349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007f67fc83eab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007f67ed590f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67faebd430 in writev () from /lib64/libc.so.6
#1 0x00007f67fc7e8869 in sys_writev () from /lib64/libglusterfs.so.0
#2 0x00007f67f0b650c9 in __socket_rwv () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#3 0x00007f67f0b67bfe in __socket_ioq_churn_entry.isra.13 () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67f0b67fa2 in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#5 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#6 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#7 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7b43e90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#9 0x00007f67e7dec349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#10 0x00007f67fc83eab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#11 0x00007f67ed590f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#14 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#15 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#17 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#18 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#19 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#20 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#21 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fae5e3de in strchrnul () from /lib64/libc.so.6
#1 0x00007f67fae10691 in vfprintf () from /lib64/libc.so.6
#2 0x00007f67faedde55 in __vsnprintf_chk () from /lib64/libc.so.6
#3 0x00007f67faedddb8 in __snprintf_chk () from /lib64/libc.so.6
#4 0x00007f67ee871d66 in posix_handle_path () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007f67ee8982bb in posix_readdirp_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#6 0x00007f67ee898ca3 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#7 0x00007f67ee898fa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#8 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#9 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#10 0x00007f67ed9bf7ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#11 0x00007f67ed7aaa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#12 0x00007f67ed56ca8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#13 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#14 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#15 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#16 0x00007f67ecd18d69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#17 0x00007f67fc8616b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#18 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67fcae34c0 (LWP 17203)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fba1b8bc in handle_fildes_io () from /lib64/librt.so.1
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 36 (Thread 0x7f67f3d85700 (LWP 15749)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7cb9a5 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f67f3584700 (LWP 15750)):
#0 0x00007f67fb6083c1 in sigwait () from /lib64/libpthread.so.0
#1 0x000056459f809013 in glusterfs_sigwaiter ()
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f67f2d83700 (LWP 15751)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67fc7e6386 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f67f2582700 (LWP 15752)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f67f1d81700 (LWP 15753)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc7fa6a0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f67fc7fb310 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f67f1580700 (LWP 15754)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67fc833844 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f67ef2b6700 (LWP 15755)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f67e6ab5700 (LWP 15756)):
#0 0x00007f67faec6f43 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f67fc81c3d0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f67e6dee700 (LWP 15789)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ec2b69f4 in index_worker () from /usr/lib64/glusterfs/7.8/xlator/features/index.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f67e5fb1700 (LWP 15800)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ede04a8b in changelog_ev_connector () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f67e57b0700 (LWP 15801)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f67e4faf700 (LWP 15802)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f67dffff700 (LWP 15803)):
#0 0x00007f67faebd9a3 in select () from /lib64/libc.so.6
#1 0x00007f67ede04cca in changelog_ev_dispatch () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f67df7fe700 (LWP 15806)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86ea40 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f67deffd700 (LWP 15807)):
#0 0x00007f67fae8d85d in nanosleep () from /lib64/libc.so.6
#1 0x00007f67fae8d6f4 in sleep () from /lib64/libc.so.6
#2 0x00007f67ee86e382 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f67de7fc700 (LWP 15808)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee869754 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f67ddffb700 (LWP 15809)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ee86ec43 in posix_fsyncer_pick () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#2 0x00007f67ee86eeb5 in posix_fsyncer () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f67dd7fa700 (LWP 15882)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f67e6af7700 (LWP 16879)):
#0 0x00007f67e7b30a19 in dict_to_xdr () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#1 0x00007f67e7b388bf in server4_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#2 0x00007f67e7de4381 in io_stats_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#3 0x00007f67ec6e1c52 in marker_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/marker.so
#4 0x00007f67fc844fec in default_setattr_cbk () from /lib64/libglusterfs.so.0
#5 0x00007f67ecd10793 in up_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#6 0x00007f67ecf2f37c in leases_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#7 0x00007f67ed57f823 in pl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#8 0x00007f67ed7ae7c7 in posix_acl_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#9 0x00007f67eddefd62 in changelog_setattr_cbk () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#10 0x00007f67ee8843fb in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#11 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#12 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#13 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#14 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#15 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#16 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#17 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#18 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#19 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#20 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#21 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#22 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#23 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f67dcff9700 (LWP 16880)):
#0 0x00007f67fb604a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67fc560bf2 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f67e6ced700 (LWP 24854)):
#0 0x00007f67faeb7365 in _lxstat () from /lib64/libc.so.6
#1 0x00007f67fc7e7ab4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007f67ee86b655 in posix_pstat () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee884116 in posix_setattr () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#5 0x00007f67eddf5657 in changelog_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/changelog.so
#6 0x00007f67fc849d07 in default_setattr () from /lib64/libglusterfs.so.0
#7 0x00007f67ed7b0000 in posix_acl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#8 0x00007f67ed56f8a5 in pl_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#9 0x00007f67ed3569bf in worm_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#10 0x00007f67ed14667b in ro_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#11 0x00007f67ecf343b6 in leases_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/leases.so
#12 0x00007f67ecd197f9 in up_setattr () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#13 0x00007f67fc862fd0 in default_setattr_resume () from /lib64/libglusterfs.so.0
#14 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#15 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#16 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f67e45ac700 (LWP 25461)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f67e456b700 (LWP 25486)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb602e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007f67fb602d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f67f0b67caa in socket_submit_outgoing_msg () from /usr/lib64/glusterfs/7.8/rpc-transport/socket.so
#4 0x00007f67fc55e58a in rpcsvc_transport_submit () from /lib64/libgfrpc.so.0
#5 0x00007f67fc56026a in rpcsvc_submit_generic () from /lib64/libgfrpc.so.0
#6 0x00007f67e7b00d2f in server_submit_reply () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#7 0x00007f67e7b43e90 in server4_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/protocol/server.so
#8 0x00007f67e7dec349 in io_stats_inodelk_cbk () from /usr/lib64/glusterfs/7.8/xlator/debug/io-stats.so
#9 0x00007f67fc83eab4 in default_inodelk_cbk () from /lib64/libglusterfs.so.0
#10 0x00007f67ed590f04 in pl_common_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#11 0x00007f67ed5918e7 in pl_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007f67ed3507b4 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/worm.so
#13 0x00007f67ed146114 in ro_inodelk () from /usr/lib64/glusterfs/7.8/xlator/features/read-only.so
#14 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#15 0x00007f67fc847374 in default_inodelk () from /lib64/libglusterfs.so.0
#16 0x00007f67fc85cb5d in default_inodelk_resume () from /lib64/libglusterfs.so.0
#17 0x00007f67fc7e2f31 in call_resume_wind () from /lib64/libglusterfs.so.0
#18 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#19 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#20 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f67e452a700 (LWP 25506)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f67e44e9700 (LWP 25815)):
#0 0x00007f67fb60754d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f67fb609d3c in _L_cond_lock_847 () from /lib64/libpthread.so.0
#2 0x00007f67fb609bd1 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#3 0x00007f67fb604eb4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#5 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f67e44a8700 (LWP 26195)):
#0 0x00007f67faec5517 in llistxattr () from /lib64/libc.so.6
#1 0x00007f67fc7e90b9 in sys_llistxattr () from /lib64/libglusterfs.so.0
#2 0x00007f67ee86bdb4 in posix_xattr_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#3 0x00007f67ee898217 in posix_entry_xattr_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#4 0x00007f67ee8983ed in posix_readdirp_fill () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#5 0x00007f67ee898ca3 in posix_do_readdir () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#6 0x00007f67ee898fa7 in posix_readdirp () from /usr/lib64/glusterfs/7.8/xlator/storage/posix.so
#7 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#8 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#9 0x00007f67ed9bf7ab in br_stub_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/bitrot-stub.so
#10 0x00007f67ed7aaa72 in posix_acl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/access-control.so
#11 0x00007f67ed56ca8f in pl_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/locks.so
#12 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#13 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#14 0x00007f67fc849257 in default_readdirp () from /lib64/libglusterfs.so.0
#15 0x00007f67ecd18d69 in up_readdirp () from /usr/lib64/glusterfs/7.8/xlator/features/upcall.so
#16 0x00007f67fc8616b0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#17 0x00007f67fc7e34d5 in call_resume () from /lib64/libglusterfs.so.0
#18 0x00007f67ecb02dd0 in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#19 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f67e4467700 (LWP 26199)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f67e4426700 (LWP 26229)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f67e43e5700 (LWP 26231)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f67e43a4700 (LWP 2757)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f67e4363700 (LWP 2846)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f67e4322700 (LWP 2847)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f67e42e1700 (LWP 2848)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f67e42a0700 (LWP 3163)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f67e425f700 (LWP 3164)):
#0 0x00007f67fb604de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f67ecb02d2d in iot_worker () from /usr/lib64/glusterfs/7.8/xlator/performance/io-threads.so
#2 0x00007f67fb600ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f67faec696d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f67fccbe4c0 (LWP 15748)):
#0 0x00007f67fb602017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f67fc81b938 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000056459f80537b in main ()

@amgads
Copy link
Contributor Author

amgads commented Sep 8, 2021

@mohit84 For the logs -- they're two many -- I have them in zip files, How can I send them (no attachment here). Can I email directly to you?

@amgads
Copy link
Contributor Author

amgads commented Sep 21, 2021

@mohit84 Any idea from the pstack output?

@mohit84
Copy link
Contributor

mohit84 commented Sep 22, 2021

@mohit84 Any idea from the pstack output?

I am not able to figure out from pstack , i believe high CPU usage should be momentarily while processing a fop request.
afaik top always shows CPU usage per core level not aggregate level so i believe it should not an issue.
If you are sure this was not the case with gluster-6.5 then i have to check it.I think in 6.5 epoll thread(default) count was 1 The current stack is showing 2 poller threads are configured (default), Can you try to configure 1 event-poll thread but it will hurt performance slightly.If performance is not criteria then you can configure it and observe the CPU usage.

@amgads
Copy link
Contributor Author

amgads commented Sep 22, 2021

@mohit84 .. thanks for looking at this. Yes application didn’t see that high cpu on 6.5. Can you share how to configure 1 event-poll thread ?

@mohit84
Copy link
Contributor

mohit84 commented Sep 22, 2021

@mohit84 .. thanks for looking at this. Yes application didn’t see that high cpu on 6.5. Can you share how to configure 1 event-poll thread ?

gluster v set vol-name client.event-threads 1
gluster v set vol-name server.event-threads 1

@amgads
Copy link
Contributor Author

amgads commented Sep 22, 2021

Thanks @mohit84 will try that and let you know. Is there a way to make for all volumes? Something like a general configuration?

@mohit84
Copy link
Contributor

mohit84 commented Sep 22, 2021

Thanks @mohit84 will try that and let you know. Is there a way to make for all volumes? Something like a general configuration?

No

@amgads
Copy link
Contributor Author

amgads commented Sep 22, 2021

@mohit84 We tried setting those parameters to 1, but saw no drop in CPU -- any other thoughts?
gluster v set vol-name client.event-threads 1
gluster v set vol-name server.event-threads 1

@mohit84
Copy link
Contributor

mohit84 commented Sep 22, 2021

@xhernandez Can you suggest something how can we proceed ?

@mohit84
Copy link
Contributor

mohit84 commented Sep 23, 2021

@mohit84 We tried setting those parameters to 1, but saw no drop in CPU -- any other thoughts?
gluster v set vol-name client.event-threads 1
gluster v set vol-name server.event-threads 1

do u have environment of 6.5, If you have can you please share pstack of a brick process while your application is consuming volume.

@amgads
Copy link
Contributor Author

amgads commented Sep 24, 2021

@mohit84
The following is the pstack for the 1st brick (release 6.5)

Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7f2d416ff700 (LWP 31285)):
#0 0x00007f2d48f69e5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f2d4a130786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f2d40efe700 (LWP 31286)):
#0 0x00007f2d48f6a381 in sigwait () from /lib64/libpthread.so.0
#1 0x00005597f004abd3 in glusterfs_sigwaiter ()
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f2d406fd700 (LWP 31287)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d4a14bb28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f2d3fefc700 (LWP 31288)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f2d3f6fb700 (LWP 31289)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d4a15dfd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007f2d4a15ec40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f2d3eefa700 (LWP 31290)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d4a196794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f2d3cc31700 (LWP 31296)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f2d3c430700 (LWP 31297)):
#0 0x00007f2d48828e63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f2d4a17f370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f2d38657700 (LWP 31308)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3942e9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f2d38556700 (LWP 31309)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f2d336ff700 (LWP 31311)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3af7fb8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2d32efe700 (LWP 31312)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2d326fd700 (LWP 31313)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2d31efc700 (LWP 31314)):
#0 0x00007f2d4881f953 in select () from /lib64/libc.so.6
#1 0x00007f2d3af7fdca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2d315fa700 (LWP 31319)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea6f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2d30df9700 (LWP 31320)):
#0 0x00007f2d487ef80d in nanosleep () from /lib64/libc.so.6
#1 0x00007f2d487ef6a4 in sleep () from /lib64/libc.so.6
#2 0x00007f2d3b9ea012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2d2bfff700 (LWP 31321)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9e5564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2d2b7fe700 (LWP 31322)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d3b9ea8f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007f2d3b9eab65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2d3805d700 (LWP 31851)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2d2affd700 (LWP 31856)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2d2a7fc700 (LWP 31899)):
#0 0x00007f2d48f669f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d49ec5c02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2d304f7700 (LWP 958)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2d304b6700 (LWP 9544)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2d30475700 (LWP 9587)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2d30434700 (LWP 16695)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2d303f3700 (LWP 16721)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2d303b2700 (LWP 16722)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2d30371700 (LWP 16723)):
#0 0x00007f2d48f66da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f2d39c7bc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007f2d48f62e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f2d4882888d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2d4a61c4c0 (LWP 31284)):
#0 0x00007f2d48f63fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f2d4a17e878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x00005597f0047035 in main ()
###########################NEXT PSTACK DUMP#######################################

@amgads
Copy link
Contributor Author

amgads commented Sep 24, 2021

@mohit84 This is the pstack for the 2nd brick (release 6.5)
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa54af315 in _lxstat () from /lib64/libc.so.6
#1 0x00007fbfa6de2af4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007fbf986838d7 in posix_handle_path () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbf9867cdee in posix_istat () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#4 0x00007fbf9868b1f0 in posix_mkdir () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#5 0x00007fbf93dec642 in trash_mkdir () from /usr/lib64/glusterfs/6.5/xlator/features/trash.so
#6 0x00007fbf93ba78de in changelog_mkdir () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#7 0x00007fbfa6e3fe82 in default_mkdir () from /lib64/libglusterfs.so.0
#8 0x00007fbf93557762 in posix_acl_mkdir () from /usr/lib64/glusterfs/6.5/xlator/features/access-control.so
#9 0x00007fbfa6e3fe82 in default_mkdir () from /lib64/libglusterfs.so.0
#10 0x00007fbfa6e3fe82 in default_mkdir () from /lib64/libglusterfs.so.0
#11 0x00007fbf92ef1a85 in ro_mkdir () from /usr/lib64/glusterfs/6.5/xlator/features/read-only.so
#12 0x00007fbfa6e3fe82 in default_mkdir () from /lib64/libglusterfs.so.0
#13 0x00007fbf92acc18e in up_mkdir () from /usr/lib64/glusterfs/6.5/xlator/features/upcall.so
#14 0x00007fbfa6e564f7 in default_mkdir_resume () from /lib64/libglusterfs.so.0
#15 0x00007fbfa6dde495 in call_resume () from /lib64/libglusterfs.so.0
#16 0x00007fbf928acd10 in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#17 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa54af315 in _lxstat () from /lib64/libc.so.6
#1 0x00007fbfa6de2af4 in sys_lstat () from /lib64/libglusterfs.so.0
#2 0x00007fbf986838d7 in posix_handle_path () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbf986a96ff in posix_readdirp_fill () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#4 0x00007fbf986aa0b3 in posix_do_readdir () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#5 0x00007fbf986aa3b7 in posix_readdirp () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#6 0x00007fbfa6e41297 in default_readdirp () from /lib64/libglusterfs.so.0
#7 0x00007fbfa6e41297 in default_readdirp () from /lib64/libglusterfs.so.0
#8 0x00007fbf9376b77b in br_stub_readdirp () from /usr/lib64/glusterfs/6.5/xlator/features/bitrot-stub.so
#9 0x00007fbf93556092 in posix_acl_readdirp () from /usr/lib64/glusterfs/6.5/xlator/features/access-control.so
#10 0x00007fbf933166ef in pl_readdirp () from /usr/lib64/glusterfs/6.5/xlator/features/locks.so
#11 0x00007fbfa6e41297 in default_readdirp () from /lib64/libglusterfs.so.0
#12 0x00007fbfa6e41297 in default_readdirp () from /lib64/libglusterfs.so.0
#13 0x00007fbfa6e41297 in default_readdirp () from /lib64/libglusterfs.so.0
#14 0x00007fbf92ac2d58 in up_readdirp () from /usr/lib64/glusterfs/6.5/xlator/features/upcall.so
#15 0x00007fbfa6e596f0 in default_readdirp_resume () from /lib64/libglusterfs.so.0
#16 0x00007fbfa6dde495 in call_resume () from /lib64/libglusterfs.so.0
#17 0x00007fbf928acd10 in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#18 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#19 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf9867fc12 in posix_fs_health_check () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbf9868002a in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#4 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbfa70df4c0 (LWP 1767)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa601386c in handle_fildes_io () from /lib64/librt.so.1
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 29 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf9867fdc2 in posix_fs_health_check () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbf9868002a in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#4 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbfa70df4c0 (LWP 5201)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa601386c in handle_fildes_io () from /lib64/librt.so.1
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################
Thread 28 (Thread 0x7fbf9e395700 (LWP 2834)):
#0 0x00007fbfa5bffe5d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fbfa6dc6786 in gf_timer_proc () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fbf9db94700 (LWP 2835)):
#0 0x00007fbfa5c00381 in sigwait () from /lib64/libpthread.so.0
#1 0x000055de8d975bd3 in glusterfs_sigwaiter ()
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fbf9d393700 (LWP 2836)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbfa6de1b28 in pool_sweeper () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fbf9cb92700 (LWP 2837)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fbf9c391700 (LWP 2838)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6df3fd0 in syncenv_task () from /lib64/libglusterfs.so.0
#2 0x00007fbfa6df4c40 in syncenv_processor () from /lib64/libglusterfs.so.0
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fbf9bb90700 (LWP 2839)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbfa6e2c794 in runner () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fbf998c7700 (LWP 2840)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fbf990c6700 (LWP 2841)):
#0 0x00007fbfa54bee63 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fbfa6e15370 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fbf909ce700 (LWP 2842)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9205f9f4 in index_worker () from /usr/lib64/glusterfs/6.5/xlator/features/index.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fbf908cd700 (LWP 2843)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fbf8bfff700 (LWP 2844)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf93bb0b8b in changelog_ev_connector () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fbf8b7fe700 (LWP 2845)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fbf8affd700 (LWP 2846)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fbf8a7fc700 (LWP 2847)):
#0 0x00007fbfa54b5953 in select () from /lib64/libc.so.6
#1 0x00007fbf93bb0dca in changelog_ev_dispatch () from /usr/lib64/glusterfs/6.5/xlator/features/changelog.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fbf89ffb700 (LWP 2848)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf986806f0 in posix_disk_space_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fbf897fa700 (LWP 2849)):
#0 0x00007fbfa548580d in nanosleep () from /lib64/libc.so.6
#1 0x00007fbfa54856a4 in sleep () from /lib64/libc.so.6
#2 0x00007fbf98680012 in posix_health_check_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fbf88ff9700 (LWP 2850)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf9867b564 in posix_ctx_janitor_thread_proc () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fbf887f8700 (LWP 2851)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf986808f3 in posix_fsyncer_pick () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#2 0x00007fbf98680b65 in posix_fsyncer () from /usr/lib64/glusterfs/6.5/xlator/storage/posix.so
#3 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fbf87ff7700 (LWP 3226)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf877f6700 (LWP 3373)):
#0 0x00007fbfa5bfc9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbfa6b5bc02 in rpcsvc_request_handler () from /lib64/libgfrpc.so.0
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf901d2700 (LWP 26351)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf90191700 (LWP 27144)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf90150700 (LWP 32595)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf9010f700 (LWP 1836)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf900ce700 (LWP 19009)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbf9008d700 (LWP 7487)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf9004c700 (LWP 6745)):
#0 0x00007fbfa5bfcda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf928acc6d in iot_worker () from /usr/lib64/glusterfs/6.5/xlator/performance/io-threads.so
#2 0x00007fbfa5bf8e65 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fbfa54be88d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbfa72b24c0 (LWP 2833)):
#0 0x00007fbfa5bf9fd7 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fbfa6e14878 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2 0x000055de8d972035 in main ()
###########################NEXT PSTACK DUMP#######################################

@amgads
Copy link
Contributor Author

amgads commented Sep 28, 2021

@mohit84 We identified the issue. It's the way the application updating these files. I'm closing the ticket. Thanks for your support as usual.

@amgads amgads closed this as completed Sep 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mohit84 @amgads