Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Close all existing processes when restarting Manticore #2798

Open
1 of 6 tasks
JubilantJerry opened this issue Nov 28, 2024 · 3 comments
Open
1 of 6 tasks

Close all existing processes when restarting Manticore #2798

JubilantJerry opened this issue Nov 28, 2024 · 3 comments
Labels
bug waiting Waiting for the original poster (in most cases) or something else

Comments

@JubilantJerry
Copy link

JubilantJerry commented Nov 28, 2024

Proposal:

I've encountered strange behavior multiple times which ultimately was caused by old instances of Manticore search still staying alive even though I've already used sudo systemctl restart manticore. Usually the existence of both new and old Manticore processes manifests as MYSQL and HTTP clients failing to connect to the backend.

In the end, I had to first run sudo systemctl stop manticore to kill the new processes, then run top -u manticore to find the PID of the old processes, then using sudo kill -9 [pid] on the old processes (sometimes they don't respond unless SIGKILL is used).

I believe Manticore should check for the old processes itself and close them without needing manual user action. I think there may even be a risk of data loss if two instances of Manticore are simultaneously active.

Checklist:

To be completed by the assignee. Check off tasks that have been completed or are not applicable.

  • Implementation completed
  • Tests developed
  • Documentation updated
  • Documentation reviewed
  • Changelog updated
  • OpenAPI YAML updated and issue created to rebuild clients
@sanikolaev
Copy link
Collaborator

Hi @JubilantJerry

Manticore should check for the old processes itself and close them without needing manual user action

It already does. A new instance shouldn’t start if an old one is still running. What you’re describing sounds like a bug. Could you share your searchd log and provide more details about when this issue occurred? This will help us investigate further.

@sanikolaev sanikolaev added waiting Waiting for the original poster (in most cases) or something else bug labels Nov 28, 2024
@JubilantJerry
Copy link
Author

JubilantJerry commented Nov 28, 2024

It does seem the old instance of Manticore crashed and left unresponsive processes during a precaching step, which would indeed be a bug. Also I've created very large tables, so it might have been resource exhaustion of some kind. However, I wanted to bring attention more to the fact that it should not even be possible to have old and new instances of Manticore coexist, even if there had been a crash in the past. In other words, Manticore should either forcibly clean up old processes even if they are misbehaving, or refuse to start up until they are closed.

When looking at systemd logs with sudo journalctl -u manticore, it seems that my attempts to restart Manticore only send SIGTERM to the old processes, and they didn't truly get killed. Afterward, Manticore still starts up even with the old processes alive.

In searchd.log there's information about precisely what went wrong with the old process, but otherwise the new instance produces normal looking logs after the event. When both old and new instances of Manticore coexist, the client often hangs when trying to connect to port 9306 or 9308. During this hanging, nothing is being output in searchd.log, as if the backend is not aware of the client's connection at all.

The relevant sudo journalctl -u manticore contents:

10/28/2024 16:46:56 searchd[3336695]: Manticore 6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
10/28/2024 16:46:56 searchd[3336695]: Copyright (c) 2001-2016, Andrew Aksyonoff
10/28/2024 16:46:56 searchd[3336695]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
10/28/2024 16:46:56 searchd[3336695]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
10/28/2024 16:46:56 systemd[1]: manticore.service: Deactivated successfully.
10/28/2024 16:46:56 systemd[1]: manticore.service: Unit process 3335663 (searchd) remains running after unit stopped.
10/28/2024 16:46:56 systemd[1]: manticore.service: Unit process 3335664 (searchd) remains running after unit stopped.
10/28/2024 16:46:56 systemd[1]: Stopped Manticore Search Engine.
10/28/2024 16:46:56 systemd[1]: manticore.service: Consumed 52.227s CPU time.
10/28/2024 16:47:08 systemd[1]: Starting Manticore Search Engine...
10/28/2024 16:47:08 searchd[3336843]: [Mon Oct 28 16:47:08.621 2024] [3336843] using config file '/etc/manticoresearch/manticore.conf' (424 chars)...
10/28/2024 16:47:08 searchd[3336844]: Manticore 6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
10/28/2024 16:47:08 searchd[3336844]: Copyright (c) 2001-2016, Andrew Aksyonoff
10/28/2024 16:47:08 searchd[3336844]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
10/28/2024 16:47:08 searchd[3336844]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
10/28/2024 16:47:08 searchd[3336846]: starting daemon version '6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.>
10/28/2024 16:47:08 searchd[3336846]: listening on 127.0.0.1:9312 for sphinx and http(s)
10/28/2024 16:47:08 searchd[3336846]: listening on 127.0.0.1:9306 for mysql
10/28/2024 16:47:08 searchd[3336846]: listening on 127.0.0.1:9308 for sphinx and http(s)
10/28/2024 16:47:08 searchd[3336846]: Manticore 6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
10/28/2024 16:47:08 searchd[3336846]: Copyright (c) 2001-2016, Andrew Aksyonoff
10/28/2024 16:47:08 searchd[3336846]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
10/28/2024 16:47:08 searchd[3336846]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
10/28/2024 16:47:08 searchd[3336846]: precaching table 'temp'
10/28/2024 16:47:08 searchd[3336846]: precaching table 'tempx'
10/28/2024 16:47:08 searchd[3336846]: precaching table 'pile_all'
10/28/2024 16:47:20 searchd[3336846]: precaching table 'pile_super_all'
10/28/2024 16:47:24 searchd[3336843]: Manticore 6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
10/28/2024 16:47:24 searchd[3336843]: Copyright (c) 2001-2016, Andrew Aksyonoff
10/28/2024 16:47:24 searchd[3336843]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
10/28/2024 16:47:24 searchd[3336843]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
10/28/2024 16:47:24 systemd[1]: manticore.service: Supervising process 3336846 which is not our child. We'll most likely not notice when it exits.
10/28/2024 16:47:24 systemd[1]: Started Manticore Search Engine.
11/13/2024 15:35:09 systemd[1]: Stopping Manticore Search Engine...
11/13/2024 15:35:10 searchd[2721881]: [Wed Nov 13 15:35:10.015 2024] [2721881] using config file '/etc/manticoresearch/manticore.conf' (424 chars)...
11/13/2024 15:35:10 searchd[2721881]: [Wed Nov 13 15:35:10.016 2024] [2721881] stop: successfully sent SIGTERM to pid 3336846
11/13/2024 15:35:13 searchd[2721881]: Manticore 6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
11/13/2024 15:35:13 searchd[2721881]: Copyright (c) 2001-2016, Andrew Aksyonoff
11/13/2024 15:35:13 searchd[2721881]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
11/13/2024 15:35:13 searchd[2721881]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
11/13/2024 15:35:13 systemd[1]: manticore.service: Control process exited, code=exited, status=1/FAILURE
11/13/2024 15:35:13 systemd[1]: manticore.service: Failed with result 'exit-code'.
11/13/2024 15:35:13 systemd[1]: manticore.service: Unit process 3336845 (searchd) remains running after unit stopped.
11/13/2024 15:35:13 systemd[1]: manticore.service: Unit process 3336846 (searchd) remains running after unit stopped.
11/13/2024 15:35:13 systemd[1]: Stopped Manticore Search Engine.
11/13/2024 15:35:13 systemd[1]: manticore.service: Consumed 5d 6h 13min 24.256s CPU time.
11/13/2024 15:35:13 systemd[1]: manticore.service: Found left-over process 3336845 (searchd) in control group while starting unit. Ignoring.
11/13/2024 15:35:13 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
11/13/2024 15:35:13 systemd[1]: manticore.service: Found left-over process 3336846 (searchd) in control group while starting unit. Ignoring.
11/13/2024 15:35:13 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
11/13/2024 15:35:13 systemd[1]: Starting Manticore Search Engine...

[...Processes 3336845 and 3336846 continue to persist, in the meanwhile I observe strange behavior and restart Manticore many times]

11/28/2024 11:43:35 searchd[3966689]: Manticore 6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
11/28/2024 11:43:35 searchd[3966689]: Copyright (c) 2001-2016, Andrew Aksyonoff
11/28/2024 11:43:35 searchd[3966689]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
11/28/2024 11:43:35 searchd[3966689]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
11/28/2024 11:43:35 systemd[1]: manticore.service: Supervising process 3966705 which is not our child. We'll most likely not notice when it exits.
11/28/2024 11:43:35 systemd[1]: Started Manticore Search Engine.
11/28/2024 11:48:44 systemd[1]: Stopping Manticore Search Engine...
11/28/2024 11:48:44 searchd[4058331]: [Thu Nov 28 11:48:44.519 2024] [4058331] using config file '/etc/manticoresearch/manticore.conf' (426 chars)...
11/28/2024 11:48:44 searchd[4058331]: [Thu Nov 28 11:48:44.520 2024] [4058331] stop: successfully sent SIGTERM to pid 3966705
11/28/2024 11:48:45 searchd[4058331]: Manticore 6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
11/28/2024 11:48:45 searchd[4058331]: Copyright (c) 2001-2016, Andrew Aksyonoff
11/28/2024 11:48:45 searchd[4058331]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
11/28/2024 11:48:45 searchd[4058331]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
11/28/2024 11:48:45 systemd[1]: manticore.service: Deactivated successfully.
11/28/2024 11:48:45 systemd[1]: manticore.service: Unit process 3336845 (searchd) remains running after unit stopped.
11/28/2024 11:48:45 systemd[1]: manticore.service: Unit process 3336846 (searchd) remains running after unit stopped.
11/28/2024 11:48:45 systemd[1]: manticore.service: Unit process 3966704 (searchd) remains running after unit stopped.
11/28/2024 11:48:45 systemd[1]: manticore.service: Unit process 3966705 (searchd) remains running after unit stopped.
11/28/2024 11:48:45 systemd[1]: Stopped Manticore Search Engine.
11/28/2024 11:48:45 systemd[1]: manticore.service: Consumed 49.964s CPU time.
11/28/2024 11:48:45 systemd[1]: manticore.service: Found left-over process 3336845 (searchd) in control group while starting unit. Ignoring.
11/28/2024 11:48:45 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
11/28/2024 11:48:45 systemd[1]: manticore.service: Found left-over process 3336846 (searchd) in control group while starting unit. Ignoring.
11/28/2024 11:48:45 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
11/28/2024 11:48:45 systemd[1]: manticore.service: Found left-over process 3966704 (searchd) in control group while starting unit. Ignoring.
11/28/2024 11:48:45 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
11/28/2024 11:48:45 systemd[1]: manticore.service: Found left-over process 3966705 (searchd) in control group while starting unit. Ignoring.
11/28/2024 11:48:45 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
11/28/2024 11:48:45 systemd[1]: Starting Manticore Search Engine...

[Then I manually kill the two processes]

11/28/2024 12:58:51 systemd[1]: Starting Manticore Search Engine...
11/28/2024 12:58:51 searchd[1160824]: [Thu Nov 28 12:58:51.017 2024] [1160824] using config file '/etc/manticoresearch/manticore.conf' (426 chars)...
11/28/2024 12:58:51 searchd[1160838]: Manticore 6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
11/28/2024 12:58:51 searchd[1160838]: Copyright (c) 2001-2016, Andrew Aksyonoff
11/28/2024 12:58:51 searchd[1160838]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
11/28/2024 12:58:51 searchd[1160838]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
11/28/2024 12:58:51 searchd[1160840]: starting daemon version '6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.>
11/28/2024 12:58:51 searchd[1160840]: listening on 127.0.0.1:9312 for sphinx and http(s)
11/28/2024 12:58:51 searchd[1160840]: listening on 127.0.0.1:9306 for mysql
11/28/2024 12:58:51 searchd[1160840]: listening on 127.0.0.1:9308 for sphinx and http(s)
11/28/2024 12:58:51 searchd[1160824]: Manticore 6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052>
11/28/2024 12:58:51 searchd[1160824]: Copyright (c) 2001-2016, Andrew Aksyonoff
11/28/2024 12:58:51 searchd[1160824]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
11/28/2024 12:58:51 searchd[1160824]: Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
11/28/2024 12:58:51 systemd[1]: manticore.service: Supervising process 1160840 which is not our child. We'll most likely not notice when it exits.
11/28/2024 12:58:51 systemd[1]: Started Manticore Search Engine.

The relevant searchd.log contents:

[Mon Nov 11 10:54:42.138 2024] [3336941] rt: table pile_super_all: diskchunk 240(241), segments 32  saved in 27.387171 (28.466158) sec, RAM saved/new 5835880089/1547517322 ratio 0.790406 (soft limit 6789534066, conf limit 8589934592)
[Mon Nov 11 11:01:46.534 2024] [3336972] rt: table pile_all: diskchunk 449(450), segments 32  saved in 38.144894 (39.586574) sec, RAM saved/new 7176668560/1400110087 ratio 0.836756 (soft limit 7187676872, conf limit 8589934592)
[Mon Nov 11 11:06:42.703 2024] [3336920] rt: table pile_super_all: diskchunk 241(242), segments 32  saved in 35.411263 (36.903299) sec, RAM saved/new 6818380942/71862335 ratio 0.950000 (soft limit 8160437862, conf limit 8589934592)
[Mon Nov 11 11:09:28.015 2024] [3336871] rt: table pile_all: diskchunk 450(451), segments 32  saved in 39.476822 (41.025140) sec, RAM saved/new 7187324729/1394568706 ratio 0.837499 (soft limit 7194059187, conf limit 8589934592)
[Mon Nov 11 11:16:58.453 2024] [3336867] rt: table pile_all: diskchunk 451(452), segments 32  saved in 39.669818 (40.924104) sec, RAM saved/new 7193639656/1401436971 ratio 0.836949 (soft limit 7189336035, conf limit 8589934592)
[Mon Nov 11 11:16:59.685 2024] [3336867] rt: table pile_all: diskchunk 452(453), segments 32  saved in 7.754468 (20.213629) sec, RAM saved/new 1397624267/3812704 ratio 0.786949 (soft limit 6759839306, conf limit 8589934592)
[Mon Nov 11 11:24:53.057 2024] [3336881] rt: table pile_all: diskchunk 453(454), segments 32  saved in 38.513024 (39.768936) sec, RAM saved/new 6762487708/450303614 ratio 0.937569 (soft limit 8053654195, conf limit 8589934592)
[Mon Nov 11 11:25:40.177 2024] [3336859] rt: table pile_super_all: diskchunk 242(243), segments 32  saved in 40.357995 (42.000830) sec, RAM saved/new 8179883832/635418057 ratio 0.927919 (soft limit 7970761293, conf limit 8589934592)
[Mon Nov 11 11:25:40.652 2024] [3336859] rt: table pile_super_all: diskchunk 243(244), segments 32  saved in 2.251657 (40.113494) sec, RAM saved/new 408218146/227199911 ratio 0.642440 (soft limit 5518519870, conf limit 8589934592)
[Mon Nov 11 11:35:30.753 2024] [3336892] rt: table pile_all: diskchunk 454(455), segments 32  saved in 40.301148 (41.681615) sec, RAM saved/new 8053111394/597981616 ratio 0.930878 (soft limit 7996180373, conf limit 8589934592)
[Mon Nov 11 11:35:31.518 2024] [3336892] rt: table pile_all: diskchunk 455(456), segments 32  saved in 3.030919 (35.525632) sec, RAM saved/new 539101952/58879664 ratio 0.880878 (soft limit 7566683643, conf limit 8589934592)
[Mon Nov 11 11:37:25.946 2024] [3336951] rt: table pile_super_all: diskchunk 244(245), segments 32  saved in 29.081783 (30.294126) sec, RAM saved/new 5516667725/164817298 ratio 0.950000 (soft limit 8160437862, conf limit 8589934592)
[Mon Nov 11 11:43:31.311 2024] [3336853] rt: table pile_all: diskchunk 456(457), segments 32  saved in 38.927459 (39.312069) sec, RAM saved/new 7566688027/1088703153 ratio 0.874217 (soft limit 7509464780, conf limit 8589934592)
[Mon Nov 11 11:43:32.136 2024] [3336853] rt: table pile_all: diskchunk 457(458), segments 32  saved in 5.780681 (25.110691) sec, RAM saved/new 1031205144/57498009 ratio 0.824217 (soft limit 7079968051, conf limit 8589934592)
[Mon Nov 11 11:50:13.595 2024] [3336846] caught SIGTERM, shutting down
[Mon Nov 11 11:50:13.598 2024] [3336887] WARNING: conn 127.0.0.1:54858(4127), sock=126: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Mon Nov 11 11:50:13.600 2024] [3336949] WARNING: conn 127.0.0.1:54844(4125), sock=124: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Mon Nov 11 11:50:13.600 2024] [3336962] WARNING: conn 127.0.0.1:54056(4041), sock=40: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Mon Nov 11 11:50:13.600 2024] [3336872] WARNING: conn 127.0.0.1:44314(1477), sock=8963: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Mon Nov 11 11:50:13.690 2024] [3336961] WARNING: conn 127.0.0.1:54040(4038), sock=38: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[...Many lines of the above warning...]
[Mon Nov 11 11:50:16.638 2024] [3336953] rt: table temp: ramchunk saved in 0.002 sec
[Mon Nov 11 11:50:16.641 2024] [3336860] rt: table pile_test20241106: ramchunk saved in 0.001 sec
[Mon Nov 11 11:50:19.881 2024] [3336860] rt: table pile_super_test20241106: ramchunk saved in 3.239 sec
[Mon Nov 11 11:50:26.112 2024] [3336860] rt: table pile_all: ramchunk saved in 6.200 sec
[Mon Nov 11 11:50:35.424 2024] [3336860] rt: table pile_super_all: ramchunk saved in 7.143 sec
[Wed Nov 13 15:35:13.196 2024] [2721890] watchdog: main process 2721891 forked ok
[Wed Nov 13 15:35:13.199 2024] [2721891] Using local time zone '/etc/localtime'
[Wed Nov 13 15:35:13.199 2024] [2721891] FATAL: failed to lock '/var/lib/manticore/binlog/binlog.lock': 11 'Resource temporarily unavailable'
[Wed Nov 13 15:35:13.200 2024] [2721890] watchdog: main process 2721891 exited cleanly (exit code 1), shutting down
[Wed Nov 13 15:41:42.666 2024] [2727196] watchdog: main process 2727197 forked ok
[Wed Nov 13 15:41:42.667 2024] [2727197] Using local time zone '/etc/localtime'
[Wed Nov 13 15:41:42.674 2024] [2727197] starting daemon version '6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
[Wed Nov 13 15:41:42.674 2024] [2727197] listening on 127.0.0.1:9312 for sphinx and http(s)
[Wed Nov 13 15:41:42.674 2024] [2727197] listening on 127.0.0.1:9306 for mysql
[Wed Nov 13 15:41:42.674 2024] [2727197] listening on 127.0.0.1:9308 for sphinx and http(s)

My manticore.service file (seems to match dist/deb/manticore.service.in):

[Unit]
Description=Manticore Search Engine
After=network.target remote-fs.target nss-lookup.target
Documentation=https://manual.manticoresearch.com/, man:searchd(1)

[Service]
Type=forking
User=manticore
Group=manticore
RuntimeDirectory=manticore
# Run ExecStart with User=manticore / Group=manticore
ExecStart=/usr/bin/searchd --config /etc/manticoresearch/manticore.conf $_ADDITIONAL_SEARCHD_PARAMS      
ExecStop=/usr/bin/searchd --config /etc/manticoresearch/manticore.conf --stopwait
KillMode=process
KillSignal=SIGTERM
SendSIGKILL=no
LimitNOFILE=65536
LimitCORE=infinity
LimitMEMLOCK=infinity
Restart=on-failure
TimeoutStartSec=infinity
PIDFile=/var/run/manticore/searchd.pid

[Install]
WantedBy=multi-user.target
Alias=manticore.service
Alias=searchd.service

@sanikolaev
Copy link
Collaborator

The relevant searchd.log contents:

That's from November 11 to 13. Do you have the log from November 28?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug waiting Waiting for the original poster (in most cases) or something else
Projects
None yet
Development

No branches or pull requests

2 participants