Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Studio docs #2455

Merged
merged 33 commits into from
May 12, 2021
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
76acd79
Initialized docs for Studio
tapadipti May 5, 2021
d8dc64b
Completed the structure of studio docs and added content to some files
May 7, 2021
a4e1813
Update studio docs and add static images
tapadipti May 10, 2021
54a6b21
Prettified cml-experiments.md
tapadipti May 10, 2021
a7665b4
Added docs for team creation and incorporated some feedback from Ivan
tapadipti May 10, 2021
aff29ea
Completed draft of docs to be shared with the team
tapadipti May 10, 2021
0cb54e8
Removed the paragraph on CML from experiments.md
tapadipti May 10, 2021
9ffd9af
Apply suggestions from code review
jorgeorpinel May 10, 2021
4462653
Apply suggestions from code review
tapadipti May 11, 2021
6ec82c3
Incorporated PR review comments
tapadipti May 11, 2021
49e66fe
Moved Studio before DVCLive + other changes from PR review
tapadipti May 11, 2021
82550d1
Added grayscale icon for Studio
tapadipti May 11, 2021
15f1b71
Changed the image for trends chart
tapadipti May 11, 2021
9865910
Minor changes after proof-reading the docs contents
tapadipti May 11, 2021
b241ae8
Update content/docs/studio/create-view.md
jorgeorpinel May 11, 2021
5d7b96b
Update content/docs/studio/get-started.md
jorgeorpinel May 11, 2021
782a384
Update content/docs/studio/create-view.md
jorgeorpinel May 12, 2021
7774313
studio: correct setting names
jorgeorpinel May 12, 2021
30af001
Update content/docs/studio/create-view.md
jorgeorpinel May 12, 2021
5284553
Restyled by prettier
restyled-commits May 12, 2021
83e223a
Merge pull request #2466 from iterative/restyled/studio-docs
jorgeorpinel May 12, 2021
23369fa
studio: run-exps page text simplification
jorgeorpinel May 12, 2021
5c39c9d
Restyled by prettier
restyled-commits May 12, 2021
756347c
Merge pull request #2467 from iterative/restyled/studio-docs
jorgeorpinel May 12, 2021
ce22026
Accepted PR suggestion (add link to CML)
tapadipti May 12, 2021
7eb44e3
Accepted PR suggestion (minor change to teams text)
tapadipti May 12, 2021
1b14980
Prettified + typo correction
tapadipti May 12, 2021
4ba41b1
Incorporated PR suggestions
tapadipti May 12, 2021
997f022
Clarified about advanced settings and experiments
tapadipti May 12, 2021
0640323
Used Studio images from the static repo and deleted them from the sta…
tapadipti May 12, 2021
6a96e64
Minor edits for typos, "click on" -> click, etc.
iesahin May 12, 2021
8d14bc3
Merge branch 'studio-docs' of github.com:iterative/dvc.org into studi…
iesahin May 12, 2021
2fb4a5d
Merge branch 'master' into studio-docs
iesahin May 12, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions content/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -514,6 +514,22 @@
}
]
},
{
"label": "Studio",
"slug": "studio",
"source": "studio/index.md",
"icon": "studio",
"children": [
"get-started",
{
"label": "Create a View",
"slug": "create-view"
},
"visualize-experiments",
"run-experiments",
"teams"
]
},
{
"label": "Dvclive",
"slug": "dvclive",
Expand Down
54 changes: 54 additions & 0 deletions content/docs/studio/create-view.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Create a View

To create a new view for your repository, follow these steps.

1. Click on `Add a View`. All the organizations that you have access to will be
listed. ![](/img/studio/create_view.png)
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

2. Open the organization whose repository you want to connect to. You can also
use the search bar to search directly for the repo which you want to connect
to. ![](/img/studio/select_repo.png)

3. Select the Git repository for which you want to create a view. For this,
hover over the required repository and click on `Connect`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe minor: Looks like 2 and 3 are the same step. Could be combined I think.

Copy link
Contributor

@jorgeorpinel jorgeorpinel May 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it looks like you can click anywhere, doesn't have to be in the Connect button.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like 2 and 3 are the same step. Could be combined I think.

They are slightly different - 2 is about finding the repo and 3 is about connecting to it. Not much difference, but I think we can let it be.

doesn't have to be in the Connect button.

Hmm, ok. But maybe in the docs, we will say click on Connect, just for clarity.

Copy link
Contributor

@jorgeorpinel jorgeorpinel May 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UX-wise they seem like the same step to me (since you can search for the repo from the beginning), but maybe we just need to rephrase a little.
Saying "Click on Connect" doesn't clarify IMO, since you can click anywhere.

Lets please leave unresolved discussions unresolved πŸ‘ (they're hard to find otherwise)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure πŸ‘

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to #2468


4. Once you click on `Connect`, you will be able to specify advanced settings
for the connection. ![](/img/studio/view_settings.png)

> In most scenarios, you can skip the advanced settings. Refer to the
> [Advanced Settings](#advanced-settings) section below for more details.

You should now see that a view has been added in your dashboard.

![](/img/studio/view_added.png)

## Advanced Settings

You will need to speciy advanced settings in the following scenarios.

> These settings can be skipped when creating the view -- they can be edited
> later.

- **Monorepo:** If you have connected to a
[monorepo](https://en.wikipedia.org/wiki/Monorepo), then specify the full path
to the root directory for the project that you want to connect to.

- **Data remotes:** The metrics and parameters that you want to include in the
view may be present in a data remote (cloud storage or another location
outside of the Git repo). If you want to include such data in your views, then
you will have to grant DVC Studio access to the data remote.

- **Custom metrics and parameters:** DVC Studio automatically detects metrics,
plots, and parameters files specified in the project's `dvc.yaml`. If you want
to connect custom files, you can add them by clicking on the `Add file`
button. Enter the full file path, and specify whether the file is for
`Metrics` or `Parameters`.

## Share a view

You can share your views on the web. Simply click on the button labelled
`Private` next to the name of the view. In the menu that pops up, turn on
`Share to Web`. ![](/img/studio/view_share.png)

You can change a shared (public) view and make it private by turning off
`Share to web`.
63 changes: 63 additions & 0 deletions content/docs/studio/get-started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Get Started with DVC Studio

Here, we will walk you through a tutorial to use DVC Studio for collaboration on
your ML projects. You will need access to a Github, Gitlab or Bitbucket account
which has access to the Git repositories you want to connect to from DVC Studio.
DVC Studio creates views from repositories when you connect to them.

## What is a view?

A _view_ is an interactive representation of the experiments run in your ML
project. DVC Studio automatically identifies datasets, metrics and
hyperparameters in your ML experiments. You can also add custom files with the
required data (more on this later). Using this data, DVC Studio creates a
summary view. This includes tables of all the metrics across all your
experiments. You can also generate plots and compare experiments here.

## Access your first view

In your browser, open <https://viewer.iterative.ai>. Sign in with your Github,
Gitlab, or Bitbucket account.

![](/img/studio/login_home.png) _When you first login, an example view is
already created for you to explore, and you can add more views._

When you first login, you will find that there already exists a view connecting
to an example DVC project. Use this view to explore the features that DVC Studio
has to offer.
Copy link
Contributor

@jorgeorpinel jorgeorpinel May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you first login, you will find that there already exists a view connecting to an example DVC project.

This did not happen for me. I just see:

image

Copy link
Contributor

@jorgeorpinel jorgeorpinel May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the paragraph is repetitive with the img caption. I'd change the caption to just DVC Studio Views page .

Copy link
Contributor Author

@tapadipti tapadipti May 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This did not happen for me.

A separate task is ongoing to add the example view by default.

change the caption to just DVC Studio Views page

sounds good. will change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A separate task is ongoing to add the example view by default.

Meaning a docs issue? Didn't find this in #2468 or #2469

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's a demo view that loads by default for new users now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks for the follow-up!


DVC Studio automatically identifies datasets, metrics and hyperparameters in
your ML experiments. Each view on the dashboard displays the metrics. In the
figure above, you can see that `avg_prec` and `roc_auc` metrics are displayed.

## Components of a view

You can dive deep into all the experiments committed to the repo. For this, open
the view by clicking on the view name (in this case, `example-get-started`).

A table will be generated as shown below. This includes metrics, hyperparameters
and information about the datasets. All the data is flattened and neatly
presented for you to evaluate and compare the experiments.

![](/img/studio/main.png)
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

This tabular display has the following components:

- The branches in your Git repository.
- All commits in each branch. Each commit, corresponding to a single row in the
table, represents an experiment.
- Values of all the metrics, files and parameters in the given commits;
corresponding to the table columns.
- Various buttons for performing actions:
- **Filters:** Filter commits
- **Columns:** Select columns to display
- **Show plots:** Show plots for the selected commits
- **Compare:** Compare different experiements
- **Run:** Run experiments by selecting any one commit (Refer
[here](/doc/studio/run-experiments) for how to run experiments)
- **Trends:** Generate trend charts to show metric evolution over time
- **Delta mode:** Toggle between absolute values and difference from the first
row

You can connect to additional repositories and add more views as needed. You can
find out how to do this in the next section.
26 changes: 26 additions & 0 deletions content/docs/studio/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# DVC Studio

[`DVC Studio`](https://viewer.iterative.ai/) is a comprehensive and interactive
collboration tool for your Machine Learning projects.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention it's visual? Maybe something like

DVC Studio is a comprehensive and interactive web application to manage and collaborate on Machine Learning projects visually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the overall document explains about the tool being visual. So maybe we can skip it in this sentence. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the opening sentence so I consider it pretty important. We'll address later πŸ‘


Using the power of leading open-source tools DVC, CML and Git, it enables you to
seamlessly manage data and models, run and track experiments, and visualize and
share results.

Project website: https://viewer.iterative.ai
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

![](/img/studio/main.png) _DVC Studio lets you not just visualize and share
results, but also run and track experiments in a single intuitive dashboard.
Above, a dashboard for an example project._
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

Use DVC, CML and Studio to:

- **Visualize, collaborate, and do everything that a regular ML tracking tool
does.**

- **Keep your code, data and model connected at all times.**

- **Use the power of Git to track and preserve all your experiments.**

- **Run CI/CD for your ML projects on cloud resources of your choice without any
new tools.**
22 changes: 22 additions & 0 deletions content/docs/studio/run-experiments.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Run Experiments

You can also run your ML experiments directly from DVC Studio. Your regular
CI/CD set-up is used to run the experiments. For instance, Github Actions can be
used to run ML experiments on each new commit.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

To run experiments from DVC Studio, select the commit that you want to use and
click the `Run` button. A form will let you to specify all the changes that you
want to make to your experiment input files/dirs and parameters.

<img src="/img/studio/cml.png" alt="drawing" width="300"/>

The default values in this form are extracted from your selected commit. For
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For another iteration:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to new ticket #2468

instance, in the given example, you can increase `max_features` to 4000 or
decrease `ngrams` to 1. Enter your commit message and description, select the
branch to commit to (either the base branch or a new branch), and click on
`Commit changes`.

Now, the CI/CD pipeline that you have set up (eg, Github Actions) will be
invokved to run the experiment. If your refresh the view (reload the page), the
Copy link
Contributor

@jorgeorpinel jorgeorpinel May 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The CI/CD part may need more explaining/linking (otherwise it's a bit misleading that DVC Studio can "run" experiments).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to new ticket #2468

experiment (i.e. the commit that you just pushed) along with its results
(metrics) will be included in the list of all commits.
73 changes: 73 additions & 0 deletions content/docs/studio/teams.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Teams

So far, you have seen how you can create personal and public views. In this
section, we will illustrate how you can create teams. In DVC Studio, you can
define teams with one or more team members. The team members are also called
collaborators, and you can assign different roles to the collaborators. The
views that you create in your team's page will be accessible to all members of
the team.

## Roles

Team members can have different roles.

- **View.** Users with the View access cannot edit team settings or create new
Comment on lines +9 to +13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be moved down after or even merged with Invite collaborators.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to new ticket #2468

views within the team's page. They have read-only access to the views created
by other team members.
- **Edit.** In addition to accessing all the team's views, users with the Edit
role can also create new views for the team, edit the views' settings, and run
experiments.
- **Admin.** Admin users have full access to the team's views and settings. They
can do everything that viewers and editors can do. Additionally, they can add
(invite) and remove collaborators as well as change team settings such as
cloud credentials (data remotes).

## Create a team

To create a team, click on the drop down next to `Personal`. All the teams that
you have created so far will be listed within "Teams" in the drop down menu. If
you have not created any team so far, this list will be empty. Now, click on
`Create a team`. ![](/img/studio/team_create.png)

You will be asked to enter the URL namespace for your team. Enter a unique name.
The URL for your team will be formed using this name.
![](/img/studio/team_enter_name.png)

Then, click on the `Create team` button on the top right corner.

## Invite collaborators

The next step is to invite the team members (or collaborators) for your team.
You can also choose to skip adding collaborators at this point. For this, click
on `Skip and Close` on the top right corner. You will be able to add
collaborators by accessing team settings later.

If you wish to add collaborators now, enter their email addresses. An email
invite will be sent to each invitee, and they will have to join using their
Github, Gitlab or Bitbucket account.

You can add multiple collaborators. Each collaborator can be assigned the Admin,
Edit, or View role. ![](/img/studio/team_roles.png)

Once you have added the people that you wish to add to your team, click on
`Send Invites and Close` on the top right corner.

## Perform actions within the team's page
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

Once you have created the team, the team's page opens up.

![](/img/studio/team_page.png)

On this page, you can perform three types of tasks:

- **Add a view.** Click on the `View` menu item to add views to the team's page.
The process for adding a view is the same as that for adding personal views
([instructions](/doc/studio/create-view)). However, the views that you create
within the team will be accessible to all members (collaborators) of the team.

- **Edit collaborators.** You can click on the `Team` menu item to edit the
collaborators in the team.

- **Change settings.** Finally, you can click on the `Settings` menu item to
change the team name, add credentials for the data remotes, and delete the
team. ![](/img/studio/team_settings.png)
40 changes: 40 additions & 0 deletions content/docs/studio/visualize-experiments.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Visualize and Compare Experiments

You can visualize and compare experiements using a variety of tools.

## Generate metric plots

You can visualize certain metrics of machine learning experiments as plots.
Usual plot examples are AUC curves, loss functions, and confusion matrices,
among others. This type of metrics files are created by users, or generated by
user data processing code, and can be defined in dvc.yaml (plots field) for
tracking (optional). Refer to the
[DVC plots documentation](/doc/command-reference/plots) for details on how plots
are generated.

To generate metric plots, select one or more experiements (represented by the
commits), and click on the 'Show plots' button.

<img src="/img/studio/select_commits.png" alt="drawing" width="300"/>

The plots will appear below the tabular display. If you have selected more than
one experiment, you can use the plots to compare them.

<img src="/img/studio/plots.png" alt="drawing" width="300"/>

## Generate trend charts

Click on the Trends button to generate a plot of how the metrics changed over
the course of the different experiments. For each metric, the trend charts show
Comment on lines +27 to +28
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make the first sentence into a p and insert an image of the Trends button? It's not easy to find by just looking at the view (I had to Ctrl+F the first time). Something like this:

image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also for Compare for consistency. That button is much more obvious/visible though (so maybe this is a deisgn issue in fact).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to new ticket #2468

how the metric changed from one commit to another. You can include one or more
branches in the trend chart.

<img src="/img/studio/trends.png" alt="drawing" width="300"/>

## Compare experiements

To compare different experiements, select two experiements (represented by the
commits), and click on the `Compare` button. The metrics, parametes and files in
the selected experiments will be displayed side by side for easy comparison.

<img src="/img/studio/compare.png" alt="drawing" width="300"/>
4 changes: 3 additions & 1 deletion src/components/Documentation/Layout/SidebarMenu/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import Link from '../../../Link'
import { ReactComponent as ExternalLinkIcon } from './external-link-icon.svg'
import { ReactComponent as HouseIcon } from './house.svg'
import { ReactComponent as CMLIcon } from './cml_bw_logo.svg'
import { ReactComponent as StudioIcon } from './studio_gray_icon.svg'

import {
structure,
Expand All @@ -25,7 +26,8 @@ import styles from './styles.module.css'
// Use the key string here as the "icon" field in sidebar.json
const ICONS: { [key: string]: React.FC<{ className?: string }> } = {
house: HouseIcon,
cml: CMLIcon
cml: CMLIcon,
studio: StudioIcon
}

interface ISidebarMenuItemProps {
Expand Down
Loading