kalite command remaining issues

[benjaoming]

Recall the task list from #2893

• Testing on Win
• Testing on OSX
• Testing on Android
• Testing on Linux (Ubuntu)
• Test case for .travis
• Bat script for Windows, currently do: python.exe bin/kalite
• Resolve how implementation should work with service / task / rc.d etc systems.
• Document SHELL environment variables
• Add a kalite reload command (not gonna call it setup, because then there's 4 commands beginning with S?)
• Add deprecation messages when running deprecated commands
• Update documentation
• Add a --port option to start and status
• Do not fail stop or restart with "Not Responding", just kill process
• Spontaneous occurrences of 'unknown command kaserve' and 'unknown command cronserver'

[benjaoming]

@aronasorman this does not warrant the ready label yet :)

[benjaoming]

I'm having an issue that the kaserve and cronserver commands are randomly missing. Traceback can look like this:

➜  ka-lite git:(kalite-command-fixes) ✗ bin/kalite restart
kalite stopped
Unknown command: 'kaserve'
Type 'kalite manage help' for usage.

Calls to clean_pyc have been removed so it seems not to be a problem of pyc files disappearing from one process while another is executed.

[benjaoming]

It may have something to do with Django not supporting true concurrency in django.core.management.get_commands() and thus we need to force a call to this function to build the list of commands before trying to execute any. Otherwise the first process will start building the list of commands, and the others will access it before it's done being built.

[benjaoming]

Yup, it was a problem of running two Django commands concurrently without a call to get_commands() first. Fixed in aede7dd

[aronasorman]

Oh, that was placed automatically by waffle.io whenever I move something to "this week" :).

[jamalex]

Love what I'm seeing!

Originally, I had been thinking the kalite status command would show more detail, but I can see the utility of keeping it simple like it is, and putting these other details into a separate command, like kalite info. This would include things like:

• The IP addresses at which KA Lite (if running) is listening (this could then be used for the GUI; see Cross-platform design of the tasktray icon launcher menu ka-lite-installers#6 (comment))
• The current version (number and hash) of KA Lite
• The device name and ID, and registration status (including Zone info)
• How much disk space is being used by KA Lite (and, when possible, how much is free)

(only the first one of these is really important -- and we at least used to have cross-platform ways to do this from Python, since we used to display it in the web interface, but numbers 2 and 3 should be easy/useful as well)

[benjaoming]

@jamalex So I guess this is overlapping a bit with kalite diagnose, but that some of it can be in status, too... like when running, it could output the IP address and port it's running on, that'd be cool.

[benjaoming]

Version is available btw with kalite --version

[benjaoming]

Going to investigate this a bit more in case it's a problem...

  ~  bin/kalite stop
Already stopped. Status was: Not responding
➜  ~  bin/kalite stop
kalite stopped

[benjaoming]

@cpauya forcing kalite to stop has been added to #2955 and should affect both start and restart

[jamalex]

@jamalex So I guess this is overlapping a bit with kalite diagnose, but that some of it can be in status, too... like when running, it could output the IP address and port it's running on, that'd be cool.

Yeah -- the use cases I had been thinking of were:

• status for the user to determine what state the server is in, and how they can access it if it's running (e.g. IPs/URLs)
• info (or diagnose? or does that do more, like run base tests?) for the user to run to output as much information as possible about the local installation, which they can copy-paste and send to us.

[benjaoming]

status now outputs URLs for the user when kalite is running and responding correctly :)

[MCGallaspy]

So our minimum for 0.13 release is to get start/stop running on all OSes.

[MCGallaspy]

On Windoze there are issues with adding kalite.bat to the path and executing it from any directory, but it seems to run fine (fingers crossed) if invoked in its same directory.

[MCGallaspy]

Seems like Windows just doesn't like some of the default options from kalite.bat start...? Testing it out, but because the server takes so long to start its quite a slow process.

[MCGallaspy]

Also kalite status seems to always think the server has stopped (even when it's running).

[MCGallaspy]

Indeed, the server seems to fail to start when the --daemonize option is included...

So the full command that windows hates is something like kalitectl.py manage kaserve --host=0.0.0.0 --daemonize --production --pidfile=foo --startup-lock-file=barf.

Any thoughts on this @benjaoming ?

[benjaoming]

It is unfortunately true that Windows doesn't support daemon behaviour,
actually whatever happens in the cherrypy, probably doesn't work on Windows
and should perhaps be conditionally turned off. This is true both for
chronograph and cherrypy AFAIK.

Windows has to go other ways with "services" or something to have a
daemon-like process.

I don't know why it would always think that the server has stopped, that
part used to be okay, and stopping it works fine, right?

On 20 February 2015 at 17:08, Michael Gallaspy [email protected]
wrote:

Indeed, the server seems to fail to start when the --daemonize option is
included...

So the full command that windows hates is something like kalitectl.py
manage kaserve --host=0.0.0.0 --daemonize --production --pidfile=foo
--startup-lock-file=barf.

—
Reply to this email directly or view it on GitHub
#2919 (comment)
.

[MCGallaspy]

I don't know why it would always think that the server has stopped, that
part used to be okay, and stopping it works fine, right?

No, stopping fails because it thinks it's already been stopped. I'll try to make it more robust. Regarding the --daemonize option, do you think its inclusion should be conditional on the operating system?

[MCGallaspy]

Yeah, something's awfully fishy about this start command on Windows. Server starts, but it fails to create a lock file or pid file, even when I take out the --daemonize option.

[benjaoming]

I would think that --daemonize should exist on windows but perhaps be a
no-op until something turns up.

The server will always fail to stop if there is no .pid file. That's
unfortunately fundamental to stopping it :) It used to create a .pid file
just fine in kalitectl... maybe it's a problem with the way that kaserve
works (didn't it also create a .pid file?).

On 23 February 2015 at 19:39, Michael Gallaspy [email protected]
wrote:

I don't know why it would always think that the server has stopped, that
part used to be okay, and stopping it works fine, right?

No, stopping fails because it thinks it's already been stopped. I'll try
to make it more robust. Regarding the --daemonize option, do you think
its inclusion should be conditional on the operating system?

—
Reply to this email directly or view it on GitHub
#2919 (comment)
.

[MCGallaspy]

This issue is frustrating me to no end, not least because it takes so long for the server to start up. I'm curious what command previous versions of KA Lite use to start up on Windows, as that might provide some clues as to the appropriate commands to be running here. Does it just call the start.bat script in the scripts directory?

[benjaoming]

Yes, previous method is still available in the scripts dir.

On 24 February 2015 at 23:45, Michael Gallaspy [email protected]
wrote:

This issue is frustrating me to no end, not least because it takes so long
for the server to start up. I'm curious what command previous versions of
KA Lite use to start up on Windows, as that might provide some clues as to
the appropriate commands to be running here. Does it just call the
start.bat script in the scripts directory?

—
Reply to this email directly or view it on GitHub
#2919 (comment)
.

[MCGallaspy]

@rtibbles okay, as a sanity check, just try to start and stop the server in the following ways:

1. Using /bin/windows/kalite.bat start and /bin/windows/kalite.bat stop using cmd.exe
2. Using python.exe manage.py kaserve using cmd.exe

[MCGallaspy]

I've found that running the command python.exe kalitectl.py start fails because this launches two separate processes, somehow. One begins to execute normally, but the other fails saying Refusing to start: Start up lock exists:... and both processes subsequently die. @benjaoming ?

[MCGallaspy]

So for me, running python.exe kalitectl.py start fails but python.exe kalitectl.py manage kaserve succeeds.

[MCGallaspy]

Regarding my above comment, the same thing happens... for some reason (that has nothing to do with the logic in kalitectl.py as far as I can tell) two identical processes are started, both executing kalitectl.py with the same arguments. In the case of kalitectl.py start, the second process finds that the first is running because of the existence of the lock file, and causes both to fail. In the case of the kalitectl.py manage kaserve, eventually one process succeeds in starting the server and who knows what happens with the other...?

[MCGallaspy]

I've been reduced to stepping through the kalitectl.py script line-by-line with a debugger, and I just have to reiterate (#3042) that import_all_child_modules is pure evil... innocuous-looking import statements can end up taking a minute or more to execute.

[benjaoming]

@MCGallaspy
I'm pretty sure that somewhere cherrypy or kaliteserve will do a fork, in order to create another process that can keep running while the main process exits. This is the most common way of doing daemon-mode, afaik... but I don't know if somehow, somewhere, this has been creating an extra process.. lack of windows experience.

Good thing you found out the problem in detail, because the start command was oddly slower on windows.

[aronasorman]

Noticed that there's a checklist for testing on Android. Do we want that as a blocker to close this issue? Maybe once we commit to supporting the Android platform we can have that as a blocker, but for now I think we can just ignore Android.

[aronasorman]

Also, looks like the only blocker for 0.13 is the Windows support? Is that correct @benjaoming?

[benjaoming]

Yes, correct, afaik fully satisfactory Windows support is the only
remaining issue, and I can't really handle it :)

But you are right that Android can be skipped for now, maybe if you can
open a separate issue tracking all Android issues, that would be good for
targetting full Android support in .14 or .15 ?

On 4 March 2015 at 12:39, Aron Fyodor Asor [email protected] wrote:

Also, looks like the only blocker for 0.13 is the Windows support? Is that
correct @benjaoming https://github.com/benjaoming?

—
Reply to this email directly or view it on GitHub
#2919 (comment)
.

[aronasorman]

Superseded by #2926. Closing.