Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Usage -> User Guide (initial) #382

Closed
wants to merge 19 commits into from
Closed

Usage -> User Guide (initial) #382

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ae23976
guides: rename Usage->Config and
jorgeorpinel Nov 18, 2022
4b08551
redirect /usage -> /config
jorgeorpinel Nov 18, 2022
f70bf8e
Merge branch 'master' into config--ref and
jorgeorpinel Nov 24, 2022
d7dbc4e
guide: fix image links again (in captions)
jorgeorpinel Nov 24, 2022
42cd4ff
guide: put example repos back in main tabs (Config)
jorgeorpinel Nov 24, 2022
82b74d5
ref: bring comment-create details & examples
jorgeorpinel Nov 24, 2022
7c886ac
Update content/docs/ref/index.md
jorgeorpinel Nov 25, 2022
ff6bef4
guide: group lose guides into new section and
jorgeorpinel Nov 25, 2022
9662dd8
Merge branch 'config--ref' of github.com:iterative/cml.dev into confi…
jorgeorpinel Nov 25, 2022
9a0e162
Update content/docs/ref/index.md
jorgeorpinel Dec 7, 2022
4c222e9
Merge branch 'master' into config--ref
jorgeorpinel Dec 7, 2022
4b33274
Merge branch 'config--ref' of github.com:iterative/cml.dev into confi…
jorgeorpinel Dec 7, 2022
608f010
Merge branch 'master' into config--ref
jorgeorpinel Dec 15, 2022
3fc4554
ref: roll back cleanup to comment cmd
jorgeorpinel Dec 16, 2022
9a2d12b
guide: fix link
jorgeorpinel Dec 16, 2022
f862dfe
Merge branch 'master' into config--ref and
jorgeorpinel Dec 27, 2022
986736e
Merge branch 'master' into config--ref
jorgeorpinel Feb 24, 2023
2a4e24e
roll back unknown changes
jorgeorpinel Feb 24, 2023
84db09c
Roll back unrelated change
jorgeorpinel Mar 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 32 additions & 12 deletions content/docs/ref/comment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Post a Markdown report as a comment on a commit, pull/merge or issue request.
cml comment create [options] <markdown report file>
```

By default PR comments are created, with a fallback to creating commit comments.
By default, PR comments are created (see `--target` for more details).
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

## update

Expand All @@ -32,43 +32,63 @@ comment to `update`.
Any [generic option](/doc/ref) in addition to:

- `--target`: Specify comment type and target (`pr`, `commit`, `issue/12`,
`pr/17` or `commit/abcdef`).
`pr/17` or `commit/abcdef`). Defaults to 1) using the PR associated with the
workflow context, 2) finding a PR containing the `HEAD` commit, or 3)
attaching the comment to the `HEAD` commit itself.

- `--watch`: Watch for changes and automatically update the comment (doesn't
exit, consider
[appending `&` to run in the background](<https://en.wikipedia.org/wiki/Job_control_(Unix)#Implementation>)).

- `--publish=<true|false>`: Upload any local images found in the Markdown report
[default: `true`].

- `--publish-native`: Use `--driver`'s native capabilities to `--publish` assets
instead of `--publish-url` (not available on `--driver=github`).

- `--publish-url=<...>`: Self-hosted image server URL [default:
`https://asset.cml.dev`], see
[minroud-s3](https://github.com/iterative/minroud-s3).

jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
- `--watermark-title=<...>`: Hidden comment marker (useful to
[specify which comment to update](#managing-multiple-comments) in subsequent
`cml comment update` calls); `"{workflow}"` and `"{run}"` are auto-replaced.

## Examples

### Creating commit or issue comments
### Post a text report

Write to your report using whatever method you prefer. For example, copy the
contents of a text file containing the results of ML model training:

```cli
$ cat results.txt >> report.md
$ cml comment create report.md
```

### Post a graphic report

Display images using Markdown or HTML. Note that if an image is an output of
your ML workflow (i.e. it is produced by your workflow), you can use Markdown to
embed it in a CML report. For example, if `plot.png` is output by
`python train.py`, run:

```cli
$ echo '![](./plot.png)' >> report.md
$ cml comment create report.md
```

`cml comment create` and `cml comment update` will create/update PR comments by
default, determining the PR from the workflow context or searching for PRs
containing the `HEAD` commit. If that fails, the fallback is to attach the
comment to the `HEAD` commit.
### Commenting on commits or issues
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

The `--target` flag allows more fine-grained control of this feature.
Use the `--target` option for fine-grained control on where to post the comment.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

```cli
# Create an issue comment
$ cml comment create --target=issue/12 report.md
```

jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
```cli
# Create a pull/merge request comment for a specific PR
$ cml comment create --target=pr/12 report.md
```

jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
```cli
# Create a commit comment attached to a specific commit
$ cml comment create --target=commit/abcdef report.md
```
Expand Down
13 changes: 13 additions & 0 deletions content/docs/ref/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
# Command Reference

CML provides a number of commands to help package the outputs of ML workflows
into a CML report, which may include numeric data or model performance
visualizations. Let's look at the typical sequence (after
[configuration](/doc/user-guide)):
Comment on lines 1 to +6
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved here from https://cml.dev/doc/usage#cml-commands

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we move it back?

@casperdcl don't we need some content in the cmd ref index though? Currently it's empty except for a Generic Options section. Seems a little bare. Imagine a user lands in this page as their entry point to the docs site (per analytics this does happen sometimes) or as one of the first pages visited. It should give some general info on cml commands as a whole, link to GS or Usage, etc., I think.

Granted, this could be addressed separately and/or we can repeat the text above in you prefer to also keep this general guidance in the Usage (now UG index) page. Lmk

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please revert


∞ `cml runner` launches a runner hosted by a cloud compute provider or
[on-premise](/doc/self-hosted-runners).

∞ `cml pr` commits a set of files to a new branch and create a pull request.

∞ `cml comment` posts a Markdown report as a comment on a commit or pull/merge
request.

## Generic Options

All CML commands support the following options:
Expand Down
19 changes: 12 additions & 7 deletions content/docs/sidebar.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,19 @@
}
]
},
"usage",
{
"label": "CML with DVC",
"slug": "cml-with-dvc"
},
{
"label": "Self-hosted Runners",
"slug": "self-hosted-runners"
"slug": "user-guide",
"source": "user-guide/index.md",
"children": [
{
"label": "CML with DVC",
"slug": "cml-with-dvc"
},
{
"label": "Self-hosted Runners",
"slug": "self-hosted-runners"
Comment on lines +32 to +37
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved these 2 existing docs into the (new) UG section

}
]
},
{
"label": "Command Reference",
Expand Down
0 content/docs/cml-with-dvc.md → content/docs/user-guide/cml-with-dvc.md
View file
Open in desktop
File renamed without changes.
111 changes: 41 additions & 70 deletions content/docs/usage.md → content/docs/user-guide/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Using CML
# User Guide: Introduction

A GitLab, GitHub, or Bitbucket account is required. Familiarity with
Comment on lines -1 to 3
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repurposed (simplified) much of this doc as the new UG index page

[GitHub Actions](https://help.github.com/en/actions),
Expand All @@ -9,7 +9,9 @@ also be beneficial.
<toggle>
<tab title="GitHub">

The key file in any CML project is `.github/workflows/cml.yaml`:
Integrate CML to your
[workflow configuration](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
using a `.github/workflows/cml.yaml` file, for example:

```yaml
name: CML
Expand Down Expand Up @@ -46,8 +48,10 @@ jobs:
cml comment create report.md
```

The example above generates visual reports in pull requests:
[![](/img/cml_first_report.png)](https://github.com/iterative/cml_base_case/pull/2)
This generates visual reports in pull requests (see `cml comment create`):

![First CML report in GitHub](/img/cml_first_report.png) _From sample
[increase forest depth](https://github.com/iterative/cml_base_case/pull/2) PR_

We helpfully provide CML and other useful libraries pre-installed on our
[custom Docker images](/doc/self-hosted-runners#docker-images). In the above
Expand All @@ -67,7 +71,9 @@ and CML set up on an Ubuntu LTS base for convenience.
</tab>
<tab title="GitLab">

The key file in any CML project is `.gitlab-ci.yml`:
Integrate CML to your
[pipeline configuration](https://docs.gitlab.com/ee/ci/pipelines/#configure-a-pipeline)
using the `.gitlab-ci.yml` file, for example:

```yaml
train-model:
Expand All @@ -86,12 +92,19 @@ create-CML-report:
- cml comment create report.md
```

⚠️ You _must_ provide a
<admon type="warn">

You _must_ provide a
[personal or project access token (PAT)](/doc/self-hosted-runners#personal-access-token)
via a `REPO_TOKEN` variable.

The example above generates visual reports in merge requests:
[![](/img/GitLab_CML_report.png '=400')](https://gitlab.com/iterative.ai/cml-base-case/-/merge_requests/1)
</admon>

This generates visual reports in pull requests (see `cml comment create`):

![First CML report in GitLab](/img/GitLab_CML_report.png '=400') _From sample
[Experiment](https://gitlab.com/iterative.ai/cml-base-case/-/merge_requests/1)
MR_

We helpfully provide CML and other useful libraries pre-installed on our
[custom Docker images](/doc/self-hosted-runners#docker-images). In the above
Expand All @@ -110,7 +123,9 @@ set up on an Ubuntu LTS base for convenience.
</tab>
<tab title="Bitbucket">

The key file in any CML project is `bitbucket-pipelines.yml`:
Integrate CML to your
[pipeline configuration](https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/)
using the `bitbucket-pipelines.yml` file, for example:

```yaml
image: iterativeai/cml:0-dvc2-base1
Expand All @@ -130,19 +145,31 @@ pipelines:
- cml comment create report.md
```

⚠️ You _must_ provide
<admon type="warn">

You _must_ provide
[access credentials](/doc/self-hosted-runners#personal-access-token) via a
`REPO_TOKEN` variable.

The example above generates visual reports in pull requests:
[![](/img/bitbucket_cloud_pr.png '=600')](https://bitbucket.org/iterative-ai/cml-base-case/pull-requests/1)
</admon>

This generates visual reports in pull requests (see `cml comment create`):

![First CML report in BitBucket](/img/bitbucket_cloud_pr.png '=600') _From
sample
[Experiment](https://bitbucket.org/iterative-ai/cml-base-case/pull-requests/1)
PR_

⚠️ CML works with Bitbucket Cloud, where you can use the
<admon type="warn">

CML works with Bitbucket Cloud, where you can use the
[Bitbucket Pipelines](https://bitbucket.org/product/features/pipelines) CI/CD
system to run workflows automatically on triggering events. Bitbucket Server is
not yet supported.

### Example projects
</admon>

### Example project

- [Basic CML project](https://bitbucket.org/iterative-ai/cml-base-case)
- [CML with DVC to pull data](https://bitbucket.org/iterative-ai/cml-dvc-case)
Expand All @@ -151,59 +178,3 @@ not yet supported.

</tab>
</toggle>

## CML Commands

CML provides a number of commands to help package the outputs of ML workflows
(including numeric data and visualizations about model performance) into a CML
report.

Below is a list of CML commands for starting cloud compute runners, writing and
publishing Markdown reports to your CI/CD system.

∞ **[`runner`](/doc/ref/runner)**\
Launch a runner hosted by a cloud compute provider or locally on-premise (see [self-hosted runners](/doc/self-hosted-runners))\
e.g. `cml runner launch --cloud={aws,azure,gcp,kubernetes} ...`

∞ **[`pr`](/doc/ref/pr)**\
Commit specified files to a new branch and create a pull request\
e.g. `cml pr create "**/*.json" "**/*.py" --md >> report.md`

∞ **[`comment`](/doc/ref/comment)**\
Post a Markdown report as a commit comment\
e.g. `cml comment create report.md`

∞ **[`check`](/doc/ref/check)**\
Post a Markdown report as a GitHub check\
e.g. `cml check create report.md`

∞ **[`tensorboard`](/doc/ref/tensorboard)**\
Return a link to a <https://tensorboard.dev> page\
e.g. `cml tensorboard connect --logdir=./logs --md >> report.md`

### CML Reports

The `cml comment create` command can be used to post reports. CML reports are
written in Markdown ([GitHub](https://github.github.com/gfm),
[GitLab](https://docs.gitlab.com/ee/user/markdown.html), or
[Bitbucket](https://confluence.atlassian.com/bitbucketserver/markdown-syntax-guide-776639995.html)
flavors). That means they can contain images, tables, formatted text, HTML
blocks, code snippets and more — really, what you put in a CML report is up to
you. Some examples:

📝 **Text** Write to your report using whatever method you prefer. For example,
copy the contents of a text file containing the results of ML model training:

```cli
$ cat results.txt >> report.md
```

🖼️ **Images** Display images using the markdown or HTML. Note that if an image
is an output of your ML workflow (i.e. it is produced by your workflow), you can
use markdown to embed it in a CML report. For example, if `plot.png` is output
by `python train.py`, run:

```cli
$ echo '![](./plot.png)' >> report.md
$ cml comment create report.md
```
0 content/docs/self-hosted-runners.md → ...nt/docs/user-guide/self-hosted-runners.md
View file
Open in desktop
File renamed without changes.
12 changes: 8 additions & 4 deletions redirects-list.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,15 @@
"^/doc/start/start-github(/.*)?$ /doc/start/github",
"^/doc/start/start-gitlab(/.*)?$ /doc/start/gitlab",

"^/doc/ref/send-github-check(/.*)?$ /doc/ref/check$1 301",
"^/doc/ref/send-comment(/.*)?$ /doc/ref/comment$1 301",
"^/doc/ref/tensorboard-dev(/.*)?$ /doc/ref/tensorboard$1 301",
"^/doc/usage$ /doc/user-guide",
"^/doc/cml-with-dvc$ /doc/user-guide/cml-with-dvc 302",
"^/doc/self-hosted-runners$ /doc/user-guide/self-hosted-runners 302",
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

"^/doc/cml-with-npm(/.*)?$ /doc/install$1 301",
"^/doc/ref/send-github-check(/.*)?$ /doc/ref/check$1",
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
"^/doc/ref/send-comment(/.*)?$ /doc/ref/comment$1",
"^/doc/ref/tensorboard-dev(/.*)?$ /doc/ref/tensorboard$1",

"^/doc/cml-with-npm(/.*)?$ /doc/install$1",

"^/(.+)/$ /$1"
]