Datalab is loading on different (incorrect) user workspace

[josecelaya]

Datalab is randomly loading under different users workspace within the serving VM for the datalab app engine (datalab_main).

I am attaching an image of datalab for which I list the workspace folder and you can see that it is on the workspace of my colleague.

We suspect this has been happening for a while since we have randomly disappearing files from workspaces. We also realize that these VMs have some background linux process to synchronize source code with git so perhaps that is the cause also of the missing files.

In any case, this is an issue and it is affecting our work now.

[ojarjur]

We have a theory about what the source of this bug could be.

The current code that launches a Jupyter server per user tells the Jupyter server to listen to a specific port. However, it does not confirm that the Jupyter server is listening on that port. Further, if the port that Jupyter is told to run on is not available, then Jupyter (by default) will try the next port in sequence until it finds a free one.

So, we could hit the following scenario:

1. A Jupyter server is launched for user A, and told to run on port N.
2. That server cannot connect to port N, so it connects to port N+1 instead.
3. Another Jupyter server is launched for user B, and told to run on port N+1.
4. Requests for user B get routed to the Jupyter server launched for user A.

The fix for that is to do multiple things:

1. Move the port retry logic out of Jupyter (by passing the '--port-retries=0' flag) and into the wrapping logic that launches Jupyter
2. Ensure that Jupyter successfully connected to the port we told it to before routing any requests to it.

I've created two different pull requests to apply that fix; one for the master branch and one for the 'datalab-managed' branch.

[ojarjur]

This bug should now be fixed, but there were two other cleanups that this has brought to our attention. I've split those out into their own issues here and here

[josecelaya]

thank you.
regards,
José

On Jun 30, 2016, at 5:32 PM, Omar Jarjur <[email protected]mailto:[email protected]> wrote:

Closed #884https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_GoogleCloudPlatform_datalab_issues_884&d=CwMCaQ&c=uGuXJ43KPkPWEl2imVFDmZQlhQUET7pVRA2PDIOxgqw&r=PknZLpsILkbpNJZKwt5YHQ&m=GYMa5jX9drItEgGI63HaMrJj0eCxzUh759wPsdKhOhI&s=mDG2toR-t5YEQIT7lx98IBKi0eG1wXl-oENgoGOh0n4&e=.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_GoogleCloudPlatform_datalab_issues_884-23event-2D709923331&d=CwMCaQ&c=uGuXJ43KPkPWEl2imVFDmZQlhQUET7pVRA2PDIOxgqw&r=PknZLpsILkbpNJZKwt5YHQ&m=GYMa5jX9drItEgGI63HaMrJj0eCxzUh759wPsdKhOhI&s=VQ5t0OSC1PmwI_McMgIOSDbUdCQbcigvB0xUZguTASQ&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe_AF6ZSIsfcMew1-2DFOrcLu1oLxsX5h4C9Xks5qRGAXgaJpZM4I-5FX1a&d=CwMCaQ&c=uGuXJ43KPkPWEl2imVFDmZQlhQUET7pVRA2PDIOxgqw&r=PknZLpsILkbpNJZKwt5YHQ&m=GYMa5jX9drItEgGI63HaMrJj0eCxzUh759wPsdKhOhI&s=8FNGsxSiMcj9KdjZSGMn1ufn6HTfI4dMWQ6xYRqME1E&e=.