Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

studio: further enhancements #2510

Merged
merged 18 commits into from
Jun 2, 2021
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion content/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -525,7 +525,11 @@
"label": "Create a View",
"slug": "create-view"
},
"visualize-experiments",
"view-settings",
{
"label": "Visualize and Compare",
"slug": "visualize-experiments"
},
"run-experiments",
"teams"
]
Expand Down
86 changes: 45 additions & 41 deletions content/docs/studio/create-view.md
Original file line number Diff line number Diff line change
@@ -1,67 +1,71 @@
# Create a View

To create a new view for your repository, follow these steps.
You can create views from your Git repositories, just like the `Demo` view you
saw in the last section. To create a new view, follow these steps.

1. Sign into your [DVC Studio](https://studio.iterative.ai/) dashboard.
![](https://static.iterative.ai/img/studio/login_home.png)
1. Sign into your [DVC Studio](https://studio.iterative.ai/) dashboard using
tapadipti marked this conversation as resolved.
Show resolved Hide resolved
GitHub, GitLab or Bitbucket.

2. Click on `Add a View`. All the organizations that you have access to will be
listed. ![](https://static.iterative.ai/img/studio/create_view.png)
listed.

> To create views from your GitHub repositories, you must install the DVC Studio
> GitHub app. Refer to the section on
> [GitHub app installation](#dvc-studio-github-app-installation) for more
> details.

3. Open the organization whose repository you want to connect to. You can also
use the search bar to search directly for the repo which you want to connect
to. ![](https://static.iterative.ai/img/studio/select_repo.png)
use the search bar to search directly for the repository. Click on the
required repository to connect to it.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

4. Select the Git repository for which you want to create a view. For this,
hover over the required repository and click `Connect`.
![](https://static.iterative.ai/img/studio/select_repo_v2.png)
shcheklein marked this conversation as resolved.
Show resolved Hide resolved

5. Once you click on `Connect`, you will be able to specify advanced settings
for the connection.
![](https://static.iterative.ai/img/studio/view_settings.png)
4. You will be able to specify additional settings for the connection.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

> In most scenarios, you can skip the advanced settings. Refer to the
> [Advanced Settings](#advanced-settings) section below for more details.
![](https://static.iterative.ai/img/studio/view_settings_v2.png)

You should now see that a view has been added in your dashboard.
> In most scenarios, you can skip these additional View settings. Refer to the
> [View Settings](/doc/studio/view-settings) section for more details.

![](https://static.iterative.ai/img/studio/view_added.png)
You should now see that a view has been added in your dashboard.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

If your project requires any of the advanced settings, then remember to
configure them by opening the View settings. Otherwise, your view may not work
as expected. To go to View settings, click on the
![](https://static.iterative.ai/img/studio/view_open_settings_icon.png) icon in
the view. In the menu that opens up (highlighted in yellow in the screenshot
below), click on `Settings`.
![](https://static.iterative.ai/img/studio/view_open_settings_icon_v2.png) icon
in the view. In the menu that opens up, click on `Settings`.

![](https://static.iterative.ai/img/studio/view_open_settings.png)
![](https://static.iterative.ai/img/studio/view_open_settings_v2.png)

## Advanced Settings
## Share a view

You will need to specify advanced settings in the following scenarios.
You can share your views on the web. Simply click on the button labelled
`Private` next to the name of the view. In the menu that pops up, turn on
`Share to Web`.

> These settings can be skipped when creating the view -- they can be edited
> later.
![](https://static.iterative.ai/img/studio/view_share_v2.png)

- **Monorepo:** If you have connected to a
[monorepo](https://en.wikipedia.org/wiki/Monorepo), then specify the full path
to the root directory of the project that you want to connect.
You can change a shared (public) view and make it private by turning off
`Share to web`.
shcheklein marked this conversation as resolved.
Show resolved Hide resolved

- **Data remotes:** The metrics and parameters that you want to include in the
view may be present in a <abbr>data remote</abbr> (cloud storage or another
location outside of the Git repo). If you want to include such data in your
views, then you will have to grant DVC Studio access to the data remote.
## DVC Studio GitHub app installation

- **Custom metrics and parameters:** DVC Studio automatically detects metrics,
plots, and parameters files specified in the project's `dvc.yaml`. If you want
to connect custom files, you can add them by clicking the `Add file` button.
Enter the full file path, and specify whether the file is for `Metrics` or
`Parameters`.
If you are using a GitHub account, you will have to install the DVC Studio
GitHub app on your GitHub repositories / organizations that you want to use with
tapadipti marked this conversation as resolved.
Show resolved Hide resolved
DVC Studio. When you try to create a view, if the app is not already installed,
DVC Studio will not be able to fetch the list of your GitHub repositories. In
this case, you will be prompted to configure Git integrations as shown below.

## Share a view
![](https://static.iterative.ai/img/studio/configure_git_integrations.png)

You can share your views on the web. Simply click on the button labelled
`Private` next to the name of the view. In the menu that pops up, turn on
`Share to Web`. ![](https://static.iterative.ai/img/studio/view_share.png)
Click on the link to `Configure Git integrations settings`. This will open the
`Git integrations` section of your profile page.

You can change a shared (public) view and make it private by turning off
`Share to web`.
![](https://static.iterative.ai/img/studio/configure_github.png)

Click on the `Configure` button, and you will be redirected to the GitHub page
to install and authorize the DVC Studio GitHub app.
shcheklein marked this conversation as resolved.
Show resolved Hide resolved

> Note that you do not have to authorize DVC Studio on all the repositories in
> your GitHub organization. You can limit access to only those repositories
> which you want to use with DVC Studio.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved
21 changes: 11 additions & 10 deletions content/docs/studio/get-started.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Get Started with DVC Studio

Here, we will walk you through a tutorial to use DVC Studio for collaboration on
your ML projects. You will need access to a Github, Gitlab or Bitbucket account
your ML projects. You will need access to a GitHub, GitLab or Bitbucket account
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the "DVC Studio creates views" view is not defined yet. Better to use some human readable terms - e.g. table with experiments? etc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add a link from 'views' to the 'What is a view?' section on that page. Works?

which has access to the Git repositories you want to connect. DVC Studio creates
views from repositories when you connect to them.

Expand All @@ -10,22 +10,23 @@ views from repositories when you connect to them.
A _view_ is an interactive representation of the experiments run in your ML
project. DVC Studio automatically identifies datasets, metrics and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think automatically is not the best term here. It doesn't do anything automatically. It parses Git history and specific metafiles (or files you show to it). Automatically sounds like some magic is happening.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"intelligently?"

hyperparameters in your ML experiments. You can also add custom files with the
required data (more on this later). Using this data, DVC Studio creates a
summary view. This includes tables of all the metrics across all your
experiments. You can also generate plots and compare experiments here.
required data (more on this [here](/doc/studio/view-settings)). Using this data,
DVC Studio creates a summary view. This includes tables of all the metrics
across all your experiments. You can also generate plots and compare experiments
here.

## DVC Studio Views page

In your browser, open <https://studio.iterative.ai>. Sign in with your Github,
Gitlab, or Bitbucket account.
GitLab, or Bitbucket account.

![](https://static.iterative.ai/img/studio/login_home.png) _When you first
![](https://static.iterative.ai/img/studio/login_home_v2.png) _When you first
login, an example view is already created for you to explore, and you can add
more views._

When you first login, you will find that there already exists a view connecting
to an example DVC project. Use this view to explore the features that DVC Studio
has to offer.
When you first login, you will find that there already exists a `Demo` view
connecting to an example DVC project. Use this view to explore the features that
DVC Studio has to offer.

DVC Studio automatically identifies datasets, metrics and hyperparameters in
your ML experiments. Each view on the dashboard displays the metrics. In the
Expand All @@ -40,7 +41,7 @@ A table will be generated as shown below. This includes metrics, hyperparameters
and information about the datasets. All the data is flattened and neatly
presented for you to evaluate and compare the experiments.

![](https://static.iterative.ai/img/studio/view_components.png)
![](https://static.iterative.ai/img/studio/view_components_v2.png)

This tabular display has the following components:

Expand Down
17 changes: 9 additions & 8 deletions content/docs/studio/index.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,26 @@
# DVC Studio

[`DVC Studio`](https://studio.iterative.ai/) is a comprehensive and interactive
collaboration tool for your Machine Learning projects.
web application to manage and collaborate on your Machine Learning projects
shcheklein marked this conversation as resolved.
Show resolved Hide resolved
visually.

Using the power of leading open-source tools DVC, CML and Git, it enables you to
seamlessly manage data and models, run and track experiments, and visualize and
share results.

Project website: https://studio.iterative.ai

![](https://static.iterative.ai/img/studio/main.png) _DVC Studio experiments
![](https://static.iterative.ai/img/studio/main_v2.png) _DVC Studio experiments
dashboard_

Use DVC, CML and Studio to:

- **Visualize, collaborate, and do everything that a regular ML tracking tool
does.**
- Visualize, collaborate, and do everything that a regular ML tracking tool
does.

- **Keep your code, data and model connected at all times.**
- Keep your code, data and model connected at all times.

- **Use the power of Git to track and preserve all your experiments.**
- Use the power of Git to track and preserve all your experiments.

- **Run [CI/CD](https://en.wikipedia.org/wiki/CI/CD) for your ML projects on
cloud resources of your choice without any new tools.**
- Run [CI/CD](https://en.wikipedia.org/wiki/CI/CD) for your ML projects on cloud
shcheklein marked this conversation as resolved.
Show resolved Hide resolved
resources of your choice without any new tools.
63 changes: 47 additions & 16 deletions content/docs/studio/run-experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,55 @@ CI/CD set-up is used to run the experiments. For instance,
[CML](https://dvc.org/doc/cml) Github Actions can be used to run ML experiments
on each new commit.

> Note that you cannot run experiments on the demo view (`example-get-started`)
> that is provided to you by default. Once you create views for your ML project
> repositories, you can follow the instructions given below and run experiments
> directly from DVC Studio.
> Note that due to access restrictions, you cannot run experiments on the demo
> view (`example-get-started`) that is provided to you by default. Once you
> create views for your ML project repositories, you can follow the instructions
> given below and run experiments directly from DVC Studio.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

To run experiments from DVC Studio, select the commit that you want to use and
click the `Run` button. A form will let you to specify all the changes that you
want to make to your experiment input files/dirs and parameters.
click the `Run` button. A form will let you specify all the changes that you
want to make to your experiment. On this form, there are 2 types of inputs that
you can change:

<img src="https://static.iterative.ai/img/studio/cml.png" alt="drawing" width="300"/>
1. **Input data files**: You can change datasets as well as metrics and model
files. The list of files that you can change will depend on your ML project.
For instance, in the `example-get-started` project, you can change the
`data.xml` file. If you select the
`Show all input parameters (including hidden)` option, then you can also
change the hidden files such as the `model.pkl` model file and the
`scores.json` metrics file. You can also choose not to change any input data
files if you only wish to change the values of one or more hyperparameters.
2. **Hyperparameters**: You can change the values of the hyperparameters that
are defined in your ML project. For instance, in the `example-get-started`
project, you can change `max_features` (the maximum number of features that
the model uses), `ngrams`, etc. You can also choose not to change any
hyperparameters if you only wish to change one or more input data files.

The default values in this form are extracted from your selected commit. For
instance, in the given example, `max_features` can be increased to 4000,
`ngrams` can be changed to 2, and so on. Enter your commit message and
description, select the branch to commit to (either the base branch or a new
branch), and click on `Commit changes`.
The default values of the input data files and hyperparameters in this form are
extracted from your selected commit.

Now, the CI/CD pipeline that you have set up (eg, Github Actions) will be
invoked to run the experiment. If you refresh the view (reload the page), the
experiment (i.e. the commit that you just pushed) along with its results
(metrics) will be included in the list of all commits.
![](https://static.iterative.ai/img/studio/cml_changes_v2.png)

Once you have made all the required changes, enter your Git commit message and
description. Then, select the branch to commit to. You can commit to either the
base branch or a new branch. If you commit to a new branch, a Git pull request
will automatically be created from the new branch to the base branch. Now, click
on `Commit changes`.

![](https://static.iterative.ai/img/studio/cml_commit_v2.png)

At this point, the new experiment appears in the view table. If you check in
your Git repository, you will find the Git commit that you just made. If you had
committed to a new branch, a new pull request will also have been created from
the new branch to the base branch.
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

If your project is integrated with a CI/CD setup (eg, GitHub Actions), the CI/CD
tapadipti marked this conversation as resolved.
Show resolved Hide resolved
setup will get invoked. If this setup includes a model training process, it will
be triggered, which means that your ML experiment will run automatically. The
model training can happen on any cloud or Kubernetes. For more details on how to
set up CI/CD pipelines for your ML project, refer to
[CML](https://dvc.org/doc/cml) .
tapadipti marked this conversation as resolved.
Show resolved Hide resolved

Once the experiment completes, its metrics will be available in the view table.
You can then generate plots and trend charts for it, or compare it with the
other experiments.
45 changes: 24 additions & 21 deletions content/docs/studio/teams.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,31 +6,16 @@ define teams with one or more team members. The team members are also called
collaborators, and you can assign different roles to them. The views that you
create in your team's page will be accessible to all members of the team.

## Roles

Team members can have different roles.

- **View.** Users with the View access cannot edit team settings or create new
views within the team's page. They have read-only access to the views created
by other team members.
- **Edit.** In addition to accessing all the team's views, users with the Edit
role can also create new views for the team, edit the views' settings, and run
experiments.
- **Admin.** Admin users have full access to the team's views and settings. They
can do everything that viewers and editors can do. Additionally, they can add
(invite) and remove collaborators as well as change team settings such as
cloud credentials (data remotes).

## Create a team

To create a team, click on the drop down next to `Personal`. All the teams that
you have created so far will be listed within "Teams" in the drop down menu. If
you have not created any team so far, this list will be empty. Now, click on
`Create a team`. ![](https://static.iterative.ai/img/studio/team_create.png)
`Create a team`. ![](https://static.iterative.ai/img/studio/team_create_v2.png)

You will be asked to enter the URL namespace for your team. Enter a unique name.
The URL for your team will be formed using this name.
![](https://static.iterative.ai/img/studio/team_enter_name.png)
![](https://static.iterative.ai/img/studio/team_enter_name_v2.png)

Then, click the `Create team` button on the top right corner.

Expand All @@ -43,19 +28,35 @@ collaborators by accessing team settings later.

If you wish to add collaborators now, enter their email addresses. An email
invite will be sent to each invitee, and they will have to join using their
Github, Gitlab or Bitbucket account.
GitHub, GitLab or Bitbucket account.

You can add multiple collaborators. Each collaborator can be assigned the Admin,
Edit, or View role. ![](https://static.iterative.ai/img/studio/team_roles.png)
Edit, or View role. Refer to the [Roles](#roles) section below for more details
about the roles. ![](https://static.iterative.ai/img/studio/team_roles_v2.png)

Once you have added the people that you wish to add to your team, click on
`Send Invites and Close` on the top right corner.

## Roles

Team members can have different roles.
shcheklein marked this conversation as resolved.
Show resolved Hide resolved

- **View.** Users with the View access cannot edit team settings or create new
views within the team's page. They have read-only access to the views created
by other team members.
- **Edit.** In addition to accessing all the team's views, users with the Edit
role can also create new views for the team, edit the views' settings, and run
experiments.
- **Admin.** Admin users have full access to the team's views and settings. They
can do everything that viewers and editors can do. Additionally, they can add
(invite) and remove collaborators as well as change team settings such as
cloud credentials (data remotes).

## Manage your team and its views

Once you have created the team, the team's page opens up.

![](https://static.iterative.ai/img/studio/team_page.png)
![](https://static.iterative.ai/img/studio/team_page_v2.png)

On this page, you can perform three types of tasks:

Expand All @@ -69,4 +70,6 @@ On this page, you can perform three types of tasks:

- **Change settings.** Finally, you can click on the `Settings` menu item to
change the team name, add credentials for the data remotes, and delete the
team. ![](https://static.iterative.ai/img/studio/team_settings.png)
team.

![](https://static.iterative.ai/img/studio/team_settings_v2.png)
Loading