Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jupyter Interactive Tool 1.0.1/24.07 #18861

Merged
merged 2 commits into from
Nov 12, 2024
Merged

Jupyter Interactive Tool 1.0.1/24.07 #18861

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1e0f928
Jupyter Interactive Tool 1.0.1/24.07
natefoo Sep 20, 2024
615bd19
Keep Jupyter 1.0.1 IT as a separate tool
natefoo Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions lib/galaxy/config/sample/tool_conf.xml.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@
<tool file="interactive/interactivetool_ethercalc.xml" />
<tool file="interactive/interactivetool_guacamole_desktop.xml" />
<tool file="interactive/interactivetool_hicbrowser.xml" />
<tool file="interactive/interactivetool_jupyter_notebook_1.0.1.xml" />
<tool file="interactive/interactivetool_jupyter_notebook_1.0.0.xml" />
<tool file="interactive/interactivetool_jupyter_notebook.xml" />
<tool file="interactive/interactivetool_neo4j.xml" />
Expand Down
3 changes: 3 additions & 0 deletions tools/interactive/interactivetool_jupyter_notebook_1.0.0.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
<requirements>
<container type="docker">quay.io/bgruening/docker-jupyter-notebook:2021-03-05</container>
</requirements>
<required_files>
<include path="default_notebook.ipynb"/>
</required_files>
<entry_points>
<entry_point name="JupyTool interactive tool" label="jupytool" requires_domain="False" requires_path_in_url="True">
<port>8888</port>
Expand Down
271 changes: 271 additions & 0 deletions tools/interactive/interactivetool_jupyter_notebook_1.0.1.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,271 @@
<tool id="interactive_tool_jupyter_notebook" tool_type="interactive" name="Interactive JupyterLab Notebook" version="1.0.1" profile="23.0">
<requirements>
<container type="docker">quay.io/bgruening/docker-jupyter-notebook:24.07</container>
</requirements>
<required_files>
<include path="default_notebook.ipynb"/>
</required_files>
<entry_points>
<entry_point name="JupyTool interactive tool" label="jupytool" requires_domain="False" requires_path_in_url="True">
<port>8888</port>
<url>ipython/lab</url>
</entry_point>
</entry_points>
<environment_variables>
<environment_variable name="HISTORY_ID">$__history_id__</environment_variable>
<environment_variable name="REMOTE_HOST">$__galaxy_url__</environment_variable>
<environment_variable name="GALAXY_WEB_PORT">8080</environment_variable>
<environment_variable name="GALAXY_URL">$__galaxy_url__</environment_variable>
<environment_variable name="API_KEY" inject="api_key"/>
<environment_variable name="EP_PATH" inject="entry_point_path_for_label">jupytool</environment_variable>
</environment_variables>
<configfiles>
<inputs name="inputs" filename="galaxy_inputs.json" data_style="staging_path_and_source_path"/>
<configfile name="staging_script" filename="staging_script.py"><![CDATA[
import json, os, shutil
import os.path
galaxy_inputs = "galaxy_inputs.json"
galaxy_inputs_raw = "galaxy_inputs_raw.json"
base_staging_path = "jupyter"
base_data_staging_path = "galaxy_inputs"
data_staging_path = os.path.join(base_staging_path, base_data_staging_path)
os.makedirs(data_staging_path, exist_ok=True)
shutil.copyfile(galaxy_inputs, os.path.join(data_staging_path, galaxy_inputs_raw))
galaxy_input_dict = {}
for user_input in json.load(open(galaxy_inputs, "r")).get("user_inputs", []):
input_values = user_input['input_type']['input_value']
input_name = user_input["input_name"]
assert input_name not in galaxy_input_dict, ValueError("Each name can only be used once, but a duplicate was found: %s" % (input_name))
if 'dataset' in user_input['input_type']['input_type_selector']:
input_values2 = []
if not isinstance(input_values, list):
input_values = [input_values]
for input_value_dict in input_values:
path, filename = os.path.split(input_value_dict['staging_path'])
input_value_dict2 = {"path": os.path.join(base_data_staging_path, input_name, path, filename), "metadata_files": []}
## TODO: enable passing in all metadata values; ideally just handed off from original json creation
path = os.path.join(data_staging_path, input_name, path)
os.makedirs(path, exist_ok=True)
try:
os.symlink(input_value_dict['source_path'], os.path.join(path, filename))
except FileExistsError:
pass
for metadata_value_dict in input_value_dict['metadata_files']:
path, filename = os.path.split(metadata_value_dict['staging_path'])
input_value_dict2["metadata_files"].append(os.path.join(base_data_staging_path, input_name, path, filename))
path = os.path.join(data_staging_path, input_name, path)
os.makedirs(path, exist_ok=True)
try:
os.symlink(metadata_value_dict['source_path'], os.path.join(path, filename))
except FileExistsError:
pass
input_values2.append(input_value_dict2)
input_values = input_values2
galaxy_input_dict[input_name] = input_values
with open(os.path.join(data_staging_path, galaxy_inputs), "w") as fh:
json.dump(galaxy_input_dict, fh)
]]>
</configfile>
<configfile name="galaxy_input_startup_script"><![CDATA[
import json, os
from galaxy_ie_helpers import get
from galaxy_ie_helpers import put
from galaxy_ie_helpers import get_galaxy_connection
HISTORY_ID = os.environ.get('HISTORY_ID', None)
try:
GALAXY_INPUTS = json.load(open(os.path.join(os.environ.get("GALAXY_WORKING_DIR", ""), "jupyter", "galaxy_inputs", "galaxy_inputs.json")))
except FileNotFoundError:
GALAXY_INPUTS = {}
]]>
</configfile>
</configfiles>
<command><![CDATA[
python staging_script.py &&
export GALAXY_WORKING_DIR=`pwd` &&
mkdir -p ./jupyter/outputs/collection &&
mkdir -p ./jupyter/galaxy_inputs &&
export PROXY_PREFIX=\${EP_PATH%/ipython*} &&

## change into the directory where the notebooks are located
cd ./jupyter/ &&
export HOME=/home/jovyan/ &&
export PATH=/home/jovyan/.local/bin:\$PATH &&
cp '${galaxy_input_startup_script}' /home/jovyan/.ipython/profile_default/startup/00-load.py &&

#set $output_notebook_name = 'ipython_galaxy_notebook.ipynb'
#if $mode.mode_select == 'scratch':
## copy default notebook
cp '$__tool_directory__/default_notebook.ipynb' ${output_notebook_name} &&
jupyter trust ${output_notebook_name} &&
jupyter lab --allow-root --no-browser --NotebookApp.shutdown_button=True &&
#else:
#import re
#set $cleaned_name = re.sub('[^\w\-\.]', '_', str($mode.ipynb.element_identifier))
#if not $cleaned_name.endswith(".ipynb"):
#set $cleaned_name = "%s.ipynb" % $cleaned_name
#end if
cp '$mode.ipynb' '${cleaned_name}' &&
jupyter trust '${cleaned_name}' &&

#if $mode.run_it:
jupyter nbconvert --to notebook --execute --output '${output_notebook_name}' --allow-errors '${cleaned_name}' &&
#else:
#set $output_notebook_name = $cleaned_name
jupyter lab --allow-root --no-browser --NotebookApp.shutdown_button=True &&
#end if
#end if
cp '${output_notebook_name}' '$jupyter_notebook'
]]>
</command>
<inputs>
<conditional name="mode">
<param name="mode_select" type="select" label="Do you already have a notebook?" help="If not, no problem we will provide you with a default one.">
<option value="scratch">Start with a fresh notebook</option>
<option value="previous">Load a previous notebook</option>
</param>
<when value="scratch"/>
<when value="previous">
<param name="ipynb" type="data" format="ipynb" label="IPython Notebook"/>
<param name="run_it" type="boolean" truevalue="true" falsevalue="false" label="Execute notebook and return a new one." help="This option is useful in workflows when you just want to execute a notebook and not dive into the webfrontend."/>
</when>
</conditional>
<repeat name="user_inputs" title="User inputs">
<param name="input_name" type="text" value="" label="Name for parameter" optional="False" help="Required. ASCII letters and numbers only.">
<validator type="empty_field" message="Name is required"/>
<sanitizer>
<valid initial="string.ascii_letters,string.digits"/>
</sanitizer>
</param>
<param name="description" type="text" label="Additional optional description" optional="true"/>
<conditional name="input_type">
<param name="input_type_selector" type="select" label="Choose the input type">
<option value="dataset" selected="true">Dataset</option>
<option value="dataset_multiple">Multiple datasets</option>
<option value="dataset_collection">Dataset collection</option>
<option value="dataset_collection_list">Dataset collection (list)</option>
<option value="dataset_collection_paired">Dataset collection (paired)</option>
<option value="dataset_collection_list_paired">Dataset collection (list:paired)</option>
<option value="text">Text</option>
<option value="integer">Integer</option>
<option value="float">Floating point</option>
<option value="boolean">Boolean</option>
<option value="color">Color selector</option>
<option value="dataset_optional" selected="true">Optional Dataset</option>
<option value="dataset_multiple_optional">Optional Multiple datasets</option>
<option value="dataset_collection_optional">Optional Dataset collection</option>
<option value="dataset_collection_list_optional">Optional Dataset collection (list)</option>
<option value="dataset_collection_paired_optional">Optional Dataset collection (paired)</option>
<option value="dataset_collection_list_paired_optional">Optional Dataset collection (list:paired)</option>
<option value="text_optional">Optional Text</option>
<option value="integer_optional">Optional Integer</option>
<option value="float_optional">Optional Floating point</option>
<option value="boolean_optional">Optional Boolean</option>
<option value="color_optional">Optional Color selector</option>
</param>
<when value="dataset">
<param name="input_value" type="data" format="data" label="Select value" multiple="false" optional="false"/>
</when>
<when value="dataset_multiple">
<param name="input_value" type="data" format="data" label="Select value" multiple="true" optional="false"/>
</when>
<when value="dataset_collection">
<param name="input_value" type="data_collection" format="data" label="Select value" optional="false"/>
</when>
<when value="dataset_collection_list">
<param name="input_value" type="data_collection" format="data" label="Select value" collection_type="list" optional="false"/>
</when>
<when value="dataset_collection_paired">
<param name="input_value" type="data_collection" format="data" label="Select value" collection_type="paired" optional="false"/>
</when>
<when value="dataset_collection_list_paired">
<param name="input_value" type="data_collection" format="data" label="Select value" collection_type="list:paired" optional="false"/>
</when>
<when value="text">
<param name="input_value" type="text" value="" label="Select value" optional="false"/>
</when>
<when value="integer">
<param name="input_value" type="integer" value="" label="Select value" optional="false"/>
</when>
<when value="float">
<param name="input_value" type="float" value="" label="Select value" optional="false"/>
</when>
<when value="boolean">
<param name="input_value" type="boolean" truevalue="true" falsevalue="false" label="Select value" optional="false"/>
</when>
<when value="color">
<param name="input_value" type="color" label="Select value" optional="false"/>
</when>
<when value="dataset_optional">
<param name="input_value" type="data" format="data" label="Select value" multiple="false" optional="true"/>
</when>
<when value="dataset_multiple_optional">
<param name="input_value" type="data" format="data" label="Select value" multiple="true" optional="true"/>
</when>
<when value="dataset_collection_optional">
<param name="input_value" type="data_collection" format="data" label="Select value" optional="true"/>
</when>
<when value="dataset_collection_list_optional">
<param name="input_value" type="data_collection" format="data" label="Select value" collection_type="list" optional="true"/>
</when>
<when value="dataset_collection_paired_optional">
<param name="input_value" type="data_collection" format="data" label="Select value" collection_type="paired" optional="true"/>
</when>
<when value="dataset_collection_list_paired_optional">
<param name="input_value" type="data_collection" format="data" label="Select value" collection_type="list:paired" optional="true"/>
</when>
<when value="text_optional">
<param name="input_value" type="text" value="" label="Select value" optional="true"/>
</when>
<when value="integer_optional">
<param name="input_value" type="integer" value="" label="Select value" optional="true"/>
</when>
<when value="float_optional">
<param name="input_value" type="float" value="" label="Select value" optional="true"/>
</when>
<when value="boolean_optional">
<param name="input_value" type="boolean" truevalue="true" falsevalue="false" label="Select value" optional="true"/>
</when>
<when value="color_optional">
<param name="input_value" type="color" label="Select value" optional="true"/>
</when>
</conditional>
</repeat>
</inputs>
<outputs>
<data name="jupyter_notebook" format="ipynb" label="Executed JupyTool Notebook"/>
<data name="output_dataset" format="data">
<discover_datasets pattern="__designation_and_ext__" directory="jupyter/outputs" visible="true" assign_primary_output="true"/>
</data>
<collection name="output_collection" type="list" label="JupyTool output collection">
<discover_datasets pattern="__designation_and_ext__" directory="jupyter/outputs/collection" visible="false"/>
</collection>
</outputs>
<tests>
<test expect_num_outputs="1">
<param name="mode" value="previous"/>
<param name="ipynb" value="test.ipynb"/>
<param name="run_it" value="true"/>
<output name="jupyter_notebook" file="test.ipynb" ftype="ipynb"/>
</test>
</tests>
<help>
Welcome to the **JupyTool**! Here you can create, run, and share custom Galaxy tools based upon Jupyter Notebooks.

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations,
visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization,
machine learning, and much more.

Galaxy offers you to use Jupyter Notebooks directly in Galaxy accessing and interacting with Galaxy datasets as you like. A very common use-case is to
do the heavy lifting and data reduction steps in Galaxy and the plotting and more `interactive` part on smaller datasets in Jupyter.

You can start with a new Jupyter notebook from scratch or load an already existing one, e.g. from your colleague and execute it on your dataset.
You can specify any number of user-defined inputs using the repeat input, providing `name` value, selecting the type of input, and then providing values.

You can make the JupyTool reusable in a workflow, by allowing the user to specify input values for the defined input blocks.
Inputs can be accessed by `name` from the automatically provided `GALAXY_INPUTS` dictionary.
Outputs can be written automatically to the user's history by writing to the `outputs` directory for one individual file or to the `outputs/collection` directory for multiple files.
Using collection tools, you can parse out the individual elements from the collection, as needed.

For backwards compatibility, you can import data into the notebook via a predefined `get()` function and write results back to Galaxy with a `put()` function.
</help>
</tool>
Loading