Request python package PyQt5, PIL for class EECS 16A #1505
Comments
|
Hello @Quizenger! I've added pillow to eecs.datahub.berkeley.edu. PyQT5 doesn't work via a browser generally - however, we added a special 'desktop' view that you can try out at the eecs datahub. This gives you a full linux desktop inside the browser, although we haven't given it a lot of RAM yet. If you can tell me what PyQT5 was used for, we can try to see if it works there? |
|
Hi @yuvipanda! Thanks for adding Pillow! The desktop view idea sounds great! I tried testing it out (by generating an nbgitpuller link for my public repo) but it showed the PyQt5 module as missing and I couldn't find the desktop view. Am I supposed to try the changes at eecs datahub some other way? PyQt5 is used to create a few QtWidgets that finally lead to one window being launched that displays lots of matrices (visualizing matrix computations) in a timer-fired loop. My public repo link: https://github.com/Quizenger/testing_datahub/tree/master/ee16a_imaging_lab3/ee16a_imaging_lab3_nOMP |
|
@Quizenger I just installed pyqt5 as well. I would recommend using JupyterLab for your use case, since it also has the easy link to the desktop version. If you go to eecs.datahub.berkeley.edu directly, it should open up in jupyterlab and have a link to the desktop environment. I also wonder if you can use ipywidgets in some form instead of qt widgets? I couldn't see where qt is used in the link you sent me. If you can point me to it I could look at possible alternatives too. |
|
@yuvipanda Thank you for installing PyQt5! I tried using JupyterLab. The desktop environment seems great! However, the cell calling the python file that runs PyQt5 results in the kernel dying. Here's a link to the python file that seems to be causing this. Please let me know if you're unable to access it and please feel free to suggest ideas/alternatives. One of my teammates tried using ipywidgets but they were too laggy. The animation was too jittery to be able to learn from. |
|
I ran the same file in the terminal provided by JupyterLab. Here's the error! qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb. Aborted (core dumped) |
|
That's great, @Quizenger! I'll poke around. Can you share the ipywidgets code? Longer term I believe that is the way to go, since it is web native. Will outperform the hacky desktop view we have, I guarantee you. |
|
@Quizenger can you open the desktop view, open a terminal in there and try the code? |
|
@yuvipanda Just tried the code using a terminal in the desktop view. The same error! I'll have to ask around for the ipywidgets code, however, I'll definitely give ipywidgets another shot. |
|
@Quizenger ok, it works for me now:
You still need to run the python script from the terminal inside the desktop launcher manually, and I had to specify the .npy file name explicitly, but it works and seems fast enough. Can you confirm? If this works, I can see if we can auto-open this. But I think it should unblock you already? |
|
@yuvipanda Yes, it seems to work quite well! I confirm. Thanks! Opening via the JupyterNotebook using a run magic still causes the kernel to die. |
|
@yuvipanda Any updates on getting PyQt5 working on the Python Terminal (from the Launcher, not the desktop terminal, I confirmed the latter works) and from the jupyter notebook? I feel that getting it working from the Launcher Python Terminal will solve the jupyter notebook issue too. Thanks! |
|
@quizenger I spent a little time on it, but didn't get anywhere. How hard is it for people to open the desktop view manually? Is this a blocker for the summer course? |
|
Hi @yuvipanda, The issue isn't opening a terminal as much as it is punching in the code to get the visualization running. We must keep in mind that these students have no programming experience and potentially no idea of how to use a terminal. This isn't a blocker for the summer course. We require it for remote instruction for the fall semester as well. I'll get started on trying another implementation. This'll be either OpenCV (which may need to be installed) or iPyWidgets based. (Trying both to see which works better for teaching.) If you could give PyQt5 another try please. |
Along with '%gui qt5', this lets users launch GUI applications from notebook to the VNC desktop! Stolen from https://github.com/manics/jupyter-omeroanalysis-desktop/blob/napari-binder/napari.ipynb Ref #1505
|
I spent some more time on this today, and got it to work! Credit to @manics for https://github.com/manics/jupyter-omeroanalysis-desktop/blob/napari-binder/napari.ipynb, hopefully implemented in #1556. You need to manually open the desktop view in a different tab before this can work. You can use this link - anyone clicking this will open the desktop view on their own server. Maybe add this to the markdown in a cell before you need to open the viewer? If it doesn't work, we have two things to try:
Let me know if this works? We might be able to make this a pane in JupyterLab, but this should hopefully unblock you before that? |
|
I would also recommend using JupyterLab from the start (you can select that as an option in nbgitpuller.link), so we can inline this in the future. |
|
Hi @yuvipanda! Wow this looks like some great progress! Unfortunately, I tried this and it still caused the kernel to die. Same for the terminal in the launcher. Same error code as last time. If you tried running the notebook, did it work for you? |
|
For clarity sakes: Whle trying, I added %gui qt5 to the notebook, opened the desktop view separately using the second link and this was all in jupyterLab. |
|
@Quizenger It looks like your notebook server has been active from before I set up this change. Can you stop and start your server from the control panel (https://eecs.datahub.berkeley.edu/hub/home), and try again? |
|
@yuvipanda Wow! That works! I can confirm. In fact, adding the %gui qt5 breaks the kernel, but it's great without it. The allotted RAM also seems enough to me but we'll be stress testing it soon. Thank you so much for all your help! If you don't mind, could we keep this thread open till we have the lab fully deployed? Just in case we run into any more bugs (fingers crossed!) |
|
@Quizenger no worries, you can keep it open :) Memory testing is important, so please do it when you get the chance before the lab. Let us know how it goes! This was a very fun project to work on for me too, bring more fun issues :) |
|
@yuvipanda Thank you! Oh yes I agree. I really really appreciate your help! For sure, count on us to get you more fun issues and quirky requirements! |
|
@yuvipanda Hi! Edit: The section had 45 students who tried to access the lab folder at the same time and the progress bar was stuck for ~10 mins. |
|
@Quizenger looking at our graph of 'waiting pods', I see a big spike at about the time you mention:
A new node was required to come up at that point, causing this. I'll try and increase the headroom for the EECS hub to prevent this from happening. How did it go otherwise? |
|
@yuvipanda alright, that sounds good! Apart from that one glitch, it's been smooth-sailing! We'll have a mini stress test with all the Lab TAs and ASEs for the PyQt5 stuff during Lab training on Wednesday. I'll let you know how that goes. Thanks! |
|
@yuvipanda we had lab training yesterday. It went really well for the most part! No major errors! However we ran into some performance issues. For some users, the lab notebook would load with either LaTeX or html image tags taking ages to render. There were also a few instances if the kernel crashing randomly. Is there anything we could do about this? We expect 50 users for each of the lab sections mentioned before. |
|
@yuvipanda this is urgent! We have a lab currently going on with approximately 50 students and a lot of them have frozen screens. This is one of the labs that uses PyQt5 though it doesn't exceed more than 700/1024 MB of provided memory. Is there anything that could be done? |
|
@Quizenger hmm, looking at it now. |
|
@Quizenger btw, joining uctech.slack.com and getting to the #ucb-datahubs channel would help get things resolved quicker for the future. |
|
@Quizenger I just gave everyone much more RAM, can you try? You might have to start / stop servers now. |
They need it (currently) for desktop usage. We might dial it back later. Ref #1505
|
@yuvipanda thanks! It was better once the RAM was added. However, the initial pull from the public repo still causes a lot of servers to freeze up. |
|
@yuvipanda some feedback from TAs: Multiple sections: "Sometimes the server just,, hangs. it feels like its thrashing and you can't even click on anything so restarting isn't an option" "Is there a way we could potentially get a TA viewable load graph for hub ?" "I think the best solution is to have an admin account that can just shutdown any student servers (this is a bit scary power but would be really helpful)" Is there anything we could do to address these concerns? |
Turns out running a Linux Desktop takes a lot of RAM Ref #1505
|
Sorry to hear that :( With #1611 I am giving users even more RAM - it might be that 4G of RAM is an absolute minimum for running something with Linux desktop. I can see that from the graph here:
I'll update the limit to about 6G shortly, so we can see where this goes. I'm also adding more 'headroom' with #1612. This should prevent need for new nodes to come up as users are logging in, which is often the cause of 'hangs' - although I don't see evidence of that in our graphs (it'll show up as a spike of 'non-running pods', which are user servers waiting to start but haven't yet.)
Can you provide a screenshot of where it hangs? That would be very helpful, since there are two loading pages - one for pulling in the content and one for the server itself starting. You could also send it to me in private on slack, since we want to keep student user names off here.
Users can always go to https://eecs.datahub.berkeley.edu/hub/home to stop & start their server. Especially if they had already opened the desktop view (which takes up a bunch of RAM) this might be useful. I also filed yuvipanda/jupyter-desktop-server#20 to make this more accessible from the desktop view. Can you also gimme a sense of what these users are doing when this happens? Just using the notebook? Any particular notebook? Desktop view on / off?
Just gave you access to http://grafana.datahub.berkeley.edu/d/naRCw05Wz/cluster-health?orgId=1&refresh=1m&from=now-24h&to=now - you can login with your github account after accepting my invite to this organization. If you can gimme other users' github usernames, I can give them access too.
We actually provide this a lot already. Make a PR listing their names at https://github.com/berkeley-dsep-infra/datahub/blob/staging/deployments/eecs/config/common.yaml#L29? They'll get an 'Admin' tab in their control panel.
Thank you for patiently helping work through these issues! We're on the cutting edge here, and I'm grateful for your patience. |
Users are consistently using at least 4G. Increasing the guarantee earlier makes sure they get at least that. However, I want to set the limit higher to see how much they actually use. Ref #1505
Was crowding out users in gamma pool for EECS hub. We also increase the placeholders to prevent issues when users are starting up together Ref #1505
|
@yuvipanda thanks for looking into all these issues! Really appreciate it!
Thanks for the link! Useful when the server is not at all responsive and students can't reach the File menu.
It hangs at two places.
Doesn't matter whether desktop view is on or off. And not just a particular notebook. How do I figure out the number of frozen user pods on Grafana?
I am compiling a list of the users for our course. Will let you know as soon as it is ready.
That's great! I'll make the PR soon. |
Thank you for your patience, @Quizenger :)
So there are two progressbars - one where your server is launching, and then when the notebook contents are being pulled in from git. The first one stuck is what I've been trying to fix with my resource requests and moving pools around. If it's the second one, can you click the progress bar to expand the log output, and paste that?
This unfortunately looks like a client side issue :( You mentioned this happens to different people. Are they on somewhat underpowered machines? Either way, trying out classic notebook to see if that fixes it might be useful.
I usually look at this graph: http://grafana.datahub.berkeley.edu/d/naRCw05Wz/cluster-health?orgId=1&refresh=1m&from=now-24h&to=now&fullscreen&panelId=2. It's the graph of user pods that are 'waiting' to start, and should ideally be just 0.
Awesome, thanks.
👍 |
|
@yuvipanda Alright, so I'm not sure which of the two progress bars led to the issues. I'll get back to you with more concrete data whenever we have lab next. Hmmmm, you bring up a good point about underpowered machines. Let's see how classic Jupyter notebooks go. Thanks for all the help! |
|
@Quizenger heya! I'm going to slowly reduce the RAM, currently from 8G - I think the issues we were seeing were mostly unrelated to that. Nobody has used more than about 2G of RAM in the last couple of weeks...
#1662 reduces it to 4G limit (from 8G), with a 2G guarantee. I'll leave this for a while... |
|
@yuvipanda yes, that sounds good! Thanks! |
|
@Quizenger think this can be closed now? |
|
Hi @yuvipanda , Apologies for the delayed reply. We'll be using the EECS Datahub next semester as well. Should I update this file with the TAs and PR without opening an issue? If so, then yes, we can close the issue. https://github.com/berkeley-dsep-infra/datahub/blob/staging/deployments/eecs/config/common.yaml#L29 |
|
@Quizenger yeah, you can just open a PR when you need to. Am very glad this worked out! |
Which package do you want installed?
PyQt5 to display visual demonstrations using a GUI/Graphics window. This opens a separate window outside the jupyter notebook interface and we need it to be visible (I'm not sure if that's possible with DataHub)
PIL (Pillow fork) to open, save and modify images
Which hub do you want it installed on? For example r.datahub, datahub, data100.datahub, etc.
eecs.datahub
Which class will use this package?
EECS 16A
Include a link to appropriate entry in https://classes.berkeley.edu/ if available. If not, please mention class name & instructor.
https://classes.berkeley.edu/content/2020-summer-eecs-16a-001-lec-001
https://classes.berkeley.edu/content/2020-fall-eecs-16a-001-lec-001
Which semester will this package be used for?
Summer 2020 (urgent), Fall 2020
This helps us clean up unused packages after a term ends.
Any additional information we should know about?
The text was updated successfully, but these errors were encountered: