Skip to content

Commit

Permalink
Bugfix: Data explorer local runner usage (#307)
Browse files Browse the repository at this point in the history
I have tested `fondant explore` by using local manifest on a M1.
Therefore, I ran into some issues.

- I updated the data explorer Dockerfile according to this solution
proposal
(streamlit/streamlit#4842 (comment))
which allows the execution of the streamlit container on a M1

- If a user has not named the local data directory as `artifacts`, the
local path can not be resolved, and the data explorer can not load the
parquet files. Therefore, I propose keeping the original name of the
data directory when mounting it to the Docker container.

- Some smaller typo fixes of the data explorer documentation

---------

Co-authored-by: NielsRogge <[email protected]>
Co-authored-by: Robbe Sneyders <[email protected]>
  • Loading branch information
3 people authored Jul 20, 2023
1 parent 147ce7c commit d4a594f
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 9 deletions.
3 changes: 2 additions & 1 deletion data_explorer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,5 @@ COPY ./app /app

WORKDIR /app

ENTRYPOINT ["streamlit", "run", "/app/main.py", "--server.port=8501", "--server.address=0.0.0.0"]
ENTRYPOINT [ "streamlit", "run" ]
CMD [ "/app/main.py", "--server.headless", "true", "--server.fileWatcherType", "none", "--browser.gatherUsageStats", "false", "--server.port=8501", "--server.address=0.0.0.0"]
6 changes: 3 additions & 3 deletions data_explorer/app/widgets.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,12 @@ def build_sidebar() -> Tuple[Optional[str], Optional[str], Optional[Dict]]:
"""
# text field for manifest path
st.sidebar.title("Subset loader")
# find all the files with filename `manifest.txt` in the `/artifacts` folder
# find all the files with filename `manifest.json` in the `/artifacts` folder
manifest_path = st.sidebar.text_input("Manifest path",
help="""Path to the manifest file.
If a data directory is mounted, the files are under
`/artifacts`. Remote files can be accessed under
their URL (e.g. gs://bucket/folder/manifest.txt, if the
`/<data-directory>`. Remote files can be accessed under
their URL (e.g. gs://bucket/folder/manifest.json, if the
correct credentials are set up.""")

# load manifest
Expand Down
4 changes: 2 additions & 2 deletions docs/data_explorer.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Data explorer

## How to use?
You can setup the data explorer container with the `fondant-explorer` CLI command, which is installed together with the Fondant python package.
You can setup the data explorer container with the `fondant explore` CLI command, which is installed together with the Fondant python package.

```
fondant-explorer --data-directory LOCAL_FOLDER_TO_MOUNT [--port PORT --container CONTAINER --tag TAG]
fondant explore --data-directory LOCAL_FOLDER_TO_MOUNT [--port PORT --container CONTAINER --tag TAG]
```

## Data explorer UI
Expand Down
4 changes: 3 additions & 1 deletion src/fondant/explorer.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ def run_explorer_app(
# mount the local data directory to the container
if data_directory:
print(data_directory)
cmd.extend(["-v", f"{shlex.quote(data_directory)}:/artifacts"])
cmd.extend(
["-v", f"{shlex.quote(data_directory)}:{shlex.quote(data_directory)}"],
)

# add the image name
cmd.extend(
Expand Down
4 changes: 2 additions & 2 deletions tests/test_explorer.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ def test_run_data_explorer_with_data_dir():
"-p",
f"{DEFAULT_PORT}:8501",
"-v",
f"{data_directory}:/artifacts",
f"{data_directory}:{data_directory}",
f"{DEFAULT_CONTAINER}:{DEFAULT_TAG}",
],
stdout=subprocess.PIPE,
Expand Down Expand Up @@ -111,7 +111,7 @@ def test_run_data_explorer_full_option():
"-v",
"/path/to/credentials:ro",
"-v",
"/path/to/source:/artifacts",
"/path/to/source:/path/to/source",
"ghcr.io/ml6team/data_explorer_test:earliest",
],
stdout=-1,
Expand Down

0 comments on commit d4a594f

Please sign in to comment.