Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update README & documentation with user journey in mind #572

Closed
9 of 10 tasks
RobbeSneyders opened this issue Oct 30, 2023 · 5 comments
Closed
9 of 10 tasks

Update README & documentation with user journey in mind #572

RobbeSneyders opened this issue Oct 30, 2023 · 5 comments
Assignees
@RobbeSneyders
Labels
documentation Improvements or additions to documentation

Comments

@RobbeSneyders
Copy link
Member

RobbeSneyders commented Oct 30, 2023 •
edited
Loading

Tasks
Preview Give feedback
  1. Restructure documentation #605
    PhilippeMoussalli
  2. Update the docs to have a reference to both python sdk/cli #601
    documentation
    PhilippeMoussalli
  3. Add high level explanation for caching #610
    PhilippeMoussalli
  4. Update component hub #602
    documentation
    PhilippeMoussalli
  5. Add section on publishing components #600
    documentation
    PhilippeMoussalli
  6. Revisit component section and add clarity #603
    documentation
    RobbeSneyders
  7. Update FAQ #604
    documentation
    PhilippeMoussalli
  8. Add Fondant architecture diagram #598
    documentation
    PhilippeMoussalli
  9. Insert extensive guide on Docker installation #599
    documentation
    PhilippeMoussalli
  10. Write a documentation page on our technology choices and trade-offs #509
    documentation
    RobbeSneyders
@RobbeSneyders RobbeSneyders moved this from Backlog to Breakdown in Fondant development Oct 30, 2023
@PhilippeMoussalli
Copy link
Contributor

PhilippeMoussalli commented Oct 30, 2023 •
edited
Loading

I would start from a landing page that has a nice overview of the logical ordering of all the concepts that the users need to know. The ordering should:

  • Enable users to quickly start with a pipeline
  • Start from simple/essential concepts and buildup towards more complex ones

Proposal on how this landing page could look like:

Getting started with Fondant

The goal here is to get the user started with Fondant quickly by showcasing how to get started and showcasing some implemented example pipelines. Going through this section should be as frictionless as possible and complex concepts should be delayed.

  • Introduction (links to page explaining what fondant is, problems it's trying to solve, roadmap, getting involved, link to blogpost)
  • Installing Fondant (docker + pip)
  • Running your first pipeline (Links to notebook
  • Example pipelines (showcase current existing example pipelines, ...)
  • Get support (github issues and discord)

Fondant fundamentals

At this stage, the user has run their first pipeline but they still have a shallow knowledge on how Fondant works, this section should start introducing different concepts. I think the concept of a pipeline is simpler to understand then the component, so it's best to start by explaining how to build a pipeline

  • Building a pipeline
  • Components (can be a similar to existing one on landing page, add a section on publishing components)
  • Exploring the dataset

Supported Runners

The user now understands how to build a pipeline and is ready to start deploying on different runners if they would want to scale their pipeline, the goal here is to showcase the different runners and how to set them up and user them + tradeoffs and when to choose which

  • Overview of supported runners + explaining which runner to chose
  • Separate page per runner showcasing how to deploy on a certain runners + runner configuration

Internals

Now that the users understand know how to get started, understand the fundamentals and runners, they can take a deep dive into how Fondant is built and advanced complex related to integration with other frameworks (Dask, partition, ...). Other concepts and internals can be explained such as caching, manifest, how streaming work, dashboard, performance, index

Reference

  • API reference
  • FAQ (clears doubts about Fondant, can repeat some concepts above to further retain some concepts, frequently encountered issues and how to solve them)
    • How to test a pipeline
    • How to run a GPU component
    • Subset/field clarification
    • How to scale
    • How to publish
      ....

@RobbeSneyders
Copy link
Member Author

We should document both the CLI & SDK for usage from a notebok.

@RobbeSneyders RobbeSneyders moved this from Breakdown to In Progress in Fondant development Nov 6, 2023
@RobbeSneyders
Copy link
Member Author

Thanks @PhilippeMoussalli.

I wouldn't work with a single landing page though, but structure our whole documentation with these ideas in mind. Iterating on your proposal, I would propose the following structure:

  • Introduction (= readme)
  • Getting started
    • Installation
    • Running an existing pipeline with local runner
    • Exploring the results
  • Components
    • Using reusable components
    • Using generic components
    • Creating custom components
  • Runners
    • Local runner
    • Vertex Runner
    • KfP runner
  • Advanced
    • Architecture

Instead of having a separate API reference, I would integrate the relevant snippets as part of the documentation pages (see the new connexion documentation as an example).

@PhilippeMoussalli
Copy link
Contributor

Thanks @PhilippeMoussalli.

I wouldn't work with a single landing page though, but structure our whole documentation with these ideas in mind. Iterating on your proposal, I would propose the following structure:

  • Introduction (= readme)

  • Getting started

    • Installation
    • Running an existing pipeline with local runner
    • Exploring the results

  • Components

    • Using reusable components
    • Using generic components
    • Creating custom components

  • Runners

    • Local runner
    • Vertex Runner
    • KfP runner

  • Advanced

    • Architecture

Instead of having a separate API reference, I would integrate the relevant snippets as part of the documentation pages (see the new connexion documentation as an example).

Yes that seems logical. We could maybe have an additional page as a guide to the documentation similar to Beam https://beam.apache.org/documentation/. Can be after the introduction

@PhilippeMoussalli PhilippeMoussalli self-assigned this Nov 7, 2023
@PhilippeMoussalli PhilippeMoussalli added the documentation Improvements or additions to documentation label Nov 7, 2023
@PhilippeMoussalli PhilippeMoussalli mentioned this issue Nov 8, 2023
Merged
@PhilippeMoussalli
Copy link
Contributor

PhilippeMoussalli commented Nov 8, 2023 •
edited
Loading

a

RobbeSneyders pushed a commit that referenced this issue Nov 9, 2023
@PhilippeMoussalli
Restructure documentation (#597)
92bf041 
PR that restructures the documentation to take in mind the user's
journey #572

Includes other changes:
* Updating main README documentation (components, some text tweaks,
runner visuals, ...)
* Renaming some components to match the format
<component_function>_<modality> (e.g. text_normalization ->
normalize_text)

There are many other changes that need to be made but best to track them
in separate PRs to make reviewing them easier. Created issues can be
found here #572
@RobbeSneyders RobbeSneyders moved this from In Progress to On hold in Fondant development Nov 22, 2023
@github-project-automation github-project-automation bot moved this from On hold to Done in Fondant development Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RobbeSneyders RobbeSneyders

Labels
documentation Improvements or additions to documentation
Projects
Fondant development
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@RobbeSneyders @PhilippeMoussalli