Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Julia builds #23

Closed
2 tasks done
choldgraf opened this issue Jun 2, 2017 · 62 comments · Fixed by #29
Closed
2 tasks done

Julia builds #23

choldgraf opened this issue Jun 2, 2017 · 62 comments · Fixed by #29
Labels
enhancement help wanted

Comments

@choldgraf
Copy link
Member

choldgraf commented Jun 2, 2017 •
edited by yuvipanda
Loading

We should support dependencies for Julia.

Major to-do items

  • Allow users to specify requirements with a REQUIRE file
  • Allow users to pre-compile packages before building the docker image

Julia handles dependencies in a text file similar to how Python does this.

Thoughts on that? @yuvipanda
@choldgraf choldgraf mentioned this issue Jun 2, 2017
Closed
yuvipanda added a commit that referenced this issue Jun 15, 2017
@yuvipanda
Add support for Julia buildpack
3d23d8a 
- Looks for REQUIRE file in a repo, assumes is Julia
- Version pinned to v0.5.2 now - in the future, pick one
  based on presence of Julia spec in REQUIRE

Fixes #23
@yuvipanda yuvipanda mentioned this issue Jun 15, 2017
Merged
yuvipanda added a commit that referenced this issue Jun 15, 2017
@yuvipanda
Add support for Julia buildpack
6a4aa2f 
- Looks for REQUIRE file in a repo, assumes is Julia
- Version pinned to v0.5.2 now - in the future, pick one
  based on presence of Julia spec in REQUIRE

Fixes #23
@choldgraf choldgraf reopened this Jun 16, 2017
@tkelman
Copy link

tkelman commented Jun 17, 2017

If you run julia -e 'using Gadfly' etc once while building the docker image it should precompile and reduce the initial load time

@sje30
Copy link

sje30 commented Jun 30, 2017

Hi,
I'm struggling to get started with Julia and binder. I've built a simple repo at: https://github.com/sje30/simplejl with my own dockerfile to get Julia 0.6.0 (rather than 0.5.2), but I'm getting a dead kernel when I put it into beta.mybinder.org. Any clues? Or should I go via the REQUIRE mechanism?

@dpsanders
Copy link

I tried adding a REQUIRE file with
julia 0.6, but the notebook launched with a Julia 0.5.2 kernel.

@sje30
Copy link

sje30 commented Jun 30, 2017

I copied/stole the Dockerfile from a source in the file; but perhaps I need more of the invocations from this Dockerfile instead?
https://github.com/jupyter/docker-stacks/blob/master/datascience-notebook/Dockerfile

@choldgraf
Copy link
Member Author

hmm - are you building on your own machine? And if so try running from master...we bumped the julia version recently: #30

@yuvipanda
Copy link
Collaborator

yuvipanda commented Jun 30, 2017 via email

@choldgraf
Copy link
Member Author

choldgraf commented Jun 30, 2017 •
edited
Loading

but to your broader point about setting up your own Dockerfile with Julia, see here for the "base" dockerfile etc that are used in the case of Julia builds...you could base your own dockerfile off of this.

@dpsanders
Copy link

Thanks. Does that Dockerfile live somewhere to use as a base in a FROM field?

@yuvipanda
Copy link
Collaborator

@sje30 @dpsanders so https://github.com/yuvipanda/simplejl works now (I did a bunch more deployments) on beta.mybinder.org.

I don't think the docker images we have are a good source of FROM, because they are subject to change pretty quickly + are meant to be used explicitly with s2i. We're working on making it easy to do other things (such as installing apt packages, running arbitrary scripts, etc) easy (see #34).

For a plain dockerfile to work with Julia, am not sure what the best approach is. We haven't quite documented the requirements for getting a plain Dockerfile to work with binder yet. It's mostly just making sure that:

  1. the notebook python package is installed, and available in $PATH
  2. An appropriate version of the jupyterhub python package is installed, and jupyterhub-singleuser is available in $PATH

Do you think a Julia enabled image would be a good addition to https://github.com/jupyer/docker-stacks? Or could it live somewhere where it's maintained by the Julia community instead?

We'd also want to add support for specifying versions of julia in the REQUIRE file - #34 should lead to a refactor that should help with this!

Finally, I must admit I know very little Julia! Any and all help welcome and gladly accepted!

thank you!

@tkelman
Copy link

tkelman commented Jul 1, 2017

We don't even maintain https://github.com/docker-library/julia/blob/master/Dockerfile, but if there were some way to paste that together with your jupyter dockerfiles (plus a Pkg.add("IJulia")) that would be neat.

If it helps, I live in Berkeley and would be happy to drop by BIDS for an hour or two any day if it would be useful to hack something out in person with some Julia expertise in the room.

@sje30
Copy link

sje30 commented Jul 1, 2017

I'd appreciate some help getting a Dockerfile that had Julia 0.6.0 + pyplot (or similar) working. My efforts so far floundering with the plotting!

@yuvipanda
Copy link
Collaborator

yuvipanda commented Jul 1, 2017 via email

@yuvipanda
Copy link
Collaborator

yuvipanda commented Jul 1, 2017 via email

@sje30
Copy link

sje30 commented Jul 1, 2017

thanks. I'd not seen the require file. Your version makes progress, but it needs a plotting engine to work still. e.g. PyPlot

thanks for adding 0.6.0 of Julia!

@sje30
Copy link

sje30 commented Jul 1, 2017

e.g. a minimal example for plotting

Using Plots
pyplot()
plot(rand(100))

I'll add it to the notebook when I get chance.

@yuvipanda
Copy link
Collaborator

yuvipanda commented Jul 1, 2017 via email

@sje30
Copy link

sje30 commented Jul 1, 2017

This is where some advice from @tkelman or @dpsanders might help.

pyplot() is part of the PyPlot package, which I could only install this morning on Dockerfile by

RUN apt-get install -y python-matplotlib

(see https://github.com/sje30/simplejl/blob/master/Dockerfile)

but then I think I got an error within the notebook saying that it could not find a suitable frontend.

@dpsanders : which plotting frontend would you recommend be added?

@yuvipanda
Copy link
Collaborator

Right, so if you were using pip it would be 'matplotlib' that'll need to be installed (we generally prefer to install python dependenceis with pip or conda rather than apt).

I'll work a bit on #34 today, which should help this situation a bit.

@dpsanders
Copy link

Pkg.add("PyPlot") should install everything automatically (using its own installation of Conda).

I recommend the GR and PlotlyJS backends.

@tkelman
Copy link

tkelman commented Jul 1, 2017

Though if we're going to use a jupyter backend that's set up already in the dockerfile through some other means, we wouldn't want to let Conda.jl bring in a whole separate instance.

@yuvipanda
Copy link
Collaborator

@tkelman we should totally hang out at Berkeley! I'll start an email thread to co-ordinate (anyone else in the area and wants to join is also welcome)

@yuvipanda
Copy link
Collaborator

Progress!

https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simpljl/master

You can see in https://github.com/yuvipanda/simplejl/ I now have a requirements.txt and a REQUIRE file - so multiple runtimes are supported! if you use a environment.yml file instead of requirements.txt you'll get a conda environment instead of venv. You can also install arbitrary apt packages by listing them in apt.txt but remember that since the python running is running from a virtualenv python packages (such as python3-matplotlib) installed via apt won't be available to the python runtime. Those should be installed via requirements.txt or environment.yml instead!

Try it out now?

There are still some rough edges in this, but I hope to have it running on the main binder beta site by end of the week.

@choldgraf
Copy link
Member Author

Me and @tkelman hacked on this today at BIDS!

woooot!

pre-compilation now works! We pre-compile everything that's opted into pre-compilation :)
We tracked down the issue with PyPlot to PyCall, and with GR to missing apt packages.

wooooooooooot!

We sent a PR to PyCall (JuliaPy/PyCall.jl#411), and getting that merged would let Julia's PyCall work with virtualenv. Until then, you can use an environment.yml to get Python / Julia combination stuff working.

wooooooooooooooooooot!

Note that I haven't deployed pre-compilation anywhere yet.

sounds good - do we have a good demo repo set up somewhere?

@choldgraf: @tkelman mentioned there's a Julia Users discourse mailing list, perhaps we could publicize this there too? (Once we deploy this support to beta, than just to staging)

+1 to that...let's set up a demo repo or two first and then for sure we should spread the word

@sje30
Copy link

sje30 commented Jul 11, 2017

Thanks guys! Amazing work. I'm happy to help with a minimal working example using PyPlots.

If its easy to fix GR, that would be great to get working too. (@jheinen - have you seen this thread? It relates to getting jupyter notebooks running Julia with plotting working on binder.)

@tkelman tkelman mentioned this issue Jul 11, 2017
Open
@jheinen
Copy link

jheinen commented Jul 11, 2017

GR can bee build without X11, but with limited output support. For IJulia, this should be no problem, as long as SVG is the default output format. SVG does not have any package requirements (same for PDF and PS).

@yuvipanda
Copy link
Collaborator

@sje30 @tkelman I finally got time to merge the work we did into master (along with a ton of other cleanups!). I've added integration tests now to this repo running on Travis, including a simple PyPlot based Julia one at https://github.com/jupyter/repo2docker/tree/master/tests/julia/pyplot.

Thank you for all your work!

@yuvipanda
Copy link
Collaborator

Note that this isn't deployed on binder yet. I'm going to try to make a full release of repo2docker first before deploying on Binder.

@sje30
Copy link

sje30 commented Jul 30, 2017 via email

@tkelman
Copy link

tkelman commented Jul 30, 2017

Excellent. Guess the PYTHONHOME issue with PyCall would still need to be addressed to make virtualenv work, but conda working without hacks is good enough for most people. Is there a test log somewhere to check that PyCall and the binder notebook are using the same conda install, and PyCall isn't downloading its own private one?

@yuvipanda
Copy link
Collaborator

@tkelman I just turned on full log capturing for all tests (and not just private ones), and you can see the logs for the Julia test here: https://travis-ci.org/jupyter/repo2docker/jobs/259053546 (it's just building).

I'd also love to have at least a couple other Julia integration tests here :D

@tkelman
Copy link

tkelman commented Jul 30, 2017

PyCall is using python (Python 3.6.1) at /srv/conda/bin/python is a very good sign. I think you can get that from the value of PyCall.python

@yuvipanda
Copy link
Collaborator

w00t. Should I add it to the verify script at https://github.com/jupyter/repo2docker/blob/master/tests/julia/pyplot/verify?

@yuvipanda
Copy link
Collaborator

@tkelman I've made a PR incorporating that - #50. I'll merge once tests go green!

@yuvipanda
Copy link
Collaborator

@tkelman so PyCall.python did not work - 92d0c5e did however!

@choldgraf @tkelman @sje30 Do you think we can now officially call Julia 'supported' in repo2docker? :)

@choldgraf
Copy link
Member Author

Awesome! Nicely done everybody!

I think we should call it "officially" supported but with a "this has not been totally vetted yet" caveat ;-) I guess if we're still in beta technically that's kinda assumed

@sje30
Copy link

sje30 commented Jul 31, 2017

Thanks everyone, this is great.

@yuvipanda
Copy link
Collaborator

Awesome! I'm going to close this one then! Tyvm everyone!

@choldgraf
Copy link
Member Author

FYI we've now got examples of this here: https://github.com/binder-examples/julia_python

@tkelman
Copy link

tkelman commented Oct 21, 2017

Cool, did the reason for the hack of needing to copy the conda folder get resolved somewhere?

@sje30
Copy link

sje30 commented Oct 23, 2017 via email

@yuvipanda
Copy link
Collaborator

@sje30 yeah, the 'julia' branch there seems to work for me. You can remove the 'conda' folder in there too

@choldgraf
Copy link
Member Author

as a neuroscientist, just wanna say that repo is super cool :-)

@sje30
Copy link

sje30 commented Nov 15, 2017 via email

@dpsanders
Copy link

dpsanders commented Nov 15, 2017 via email

@sje30
Copy link

sje30 commented Nov 15, 2017

Where is the julia 0.6 kernel? Just added a REQUIRE, but not finding julia 0.6 on http://beta.mybinder.org (which now redirects)

@sje30
Copy link

sje30 commented Nov 16, 2017

Thanks @dpsanders -- all working now! http://sje30.github.io/post/neuro-binder/

@yuvipanda yuvipanda mentioned this issue Jan 18, 2019
Merged
@ems0029
Copy link

ems0029 commented Aug 24, 2023 •
edited
Loading

Not sure where this issue should go, but the IJulia demo notebook is broken, specifically the Interactive plotting section:

cannot assign a value to variable Base.sleep from module Main

Stacktrace:
 [1] top-level scope
   @ In[16]:2
 [2] eval
   @ ./boot.jl:373 [inlined]
 [3] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1196

@rgaiacs
Copy link

rgaiacs commented Aug 24, 2023

Dear @ems0029,

Thanks for the message.

I can't reproduce the error when I follow the next steps:

  1. Visit https://mybinder.org/
  2. Fill the repository field with https://github.com/binder-examples/demo-julia
  3. Click launch
  4. After Jupyter Notebook is loaded, run the cells.

The Interactive plotting section work as expected.

Because your issue is directly related to https://github.com/binder-examples/demo-julia, please open an issue there.

Screenshot 2023-08-24 at 17-01-24 demo ipynb - JupyterLab

shota-matsumoto-lm pushed a commit to lifematics/CS-repo2docker that referenced this issue Feb 1, 2024
@jtpio
Merge pull request jupyterhub#23 from jtpio/lowercase-repo-name
658a8d2 
Lowercase the autogenerated name from the repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for Julia buildpack jupyterhub/repo2docker
8 participants
@yuvipanda @dpsanders @sje30 @rgaiacs @choldgraf @jheinen @tkelman @ems0029