Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc (spec): add error message guideline #1163

Merged
merged 52 commits into from
Feb 23, 2024
Merged
Changes from 47 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
aa34c23
doc: add ORAS CLI proposal doc
FeynmanZhou Jan 10, 2023
5e05838
doc: add ORAS CLI proposal doc
FeynmanZhou Jan 10, 2023
cce1f66
doc: update the compatibility mode proposal
FeynmanZhou Jan 10, 2023
0066516
update compatibility doc
FeynmanZhou Jan 28, 2023
4013c48
Merge branch 'main' into main
FeynmanZhou Jan 28, 2023
49a20b0
update compatibility doc to resolve Billy's comments
FeynmanZhou Jan 29, 2023
cb5e7bc
Merge branch 'main' of https://github.com/FeynmanZhou/oras
FeynmanZhou Jan 29, 2023
9addaa3
update the compatibility doc to resolve Shiwei's comments
FeynmanZhou Jan 30, 2023
e4d1922
update the compatibility doc to resolve Shiwei's comments
FeynmanZhou Jan 30, 2023
506ec41
Merge branch 'main' into main
FeynmanZhou Jan 30, 2023
0c5d128
Merge branch 'main' of https://github.com/FeynmanZhou/oras; branch 'm…
FeynmanZhou Mar 6, 2023
c5bc0bf
fix broken logo link
FeynmanZhou Apr 27, 2023
eb2a1ae
Merge branch 'main' of https://github.com/oras-project/oras
FeynmanZhou Aug 21, 2023
e65a543
Merge branch 'main' of https://github.com/oras-project/oras
FeynmanZhou Nov 2, 2023
0ec0703
add error handling guide
FeynmanZhou Nov 2, 2023
4b63f4a
update error message examples
FeynmanZhou Nov 10, 2023
76db991
update error message examples
FeynmanZhou Nov 10, 2023
20b2084
Merge branch 'main' of https://github.com/oras-project/oras into erro…
FeynmanZhou Nov 10, 2023
39ff6ae
update error message examples
FeynmanZhou Nov 10, 2023
95ec9cb
resolve comments
FeynmanZhou Nov 15, 2023
16e86cf
resolve comments
FeynmanZhou Nov 15, 2023
764b9bb
resolve comments and update examples
FeynmanZhou Nov 17, 2023
68c8907
update cli error guideline
FeynmanZhou Nov 29, 2023
c74a6d3
refine CLI error guideline
FeynmanZhou Nov 29, 2023
3510594
refine examples
FeynmanZhou Dec 5, 2023
1792ab0
update cli guideline
FeynmanZhou Dec 11, 2023
b4f8c26
resolve comments
FeynmanZhou Dec 11, 2023
7adf1aa
update CLI error guideline
FeynmanZhou Dec 13, 2023
5ac34af
resolve comments
FeynmanZhou Dec 18, 2023
6669a4e
resolve comments
FeynmanZhou Dec 18, 2023
5287047
add more examples
FeynmanZhou Dec 20, 2023
5e31ce2
resolve comments and add more examples
FeynmanZhou Dec 22, 2023
d960843
Merge branch 'main' of https://github.com/oras-project/oras into erro…
FeynmanZhou Dec 22, 2023
7dccd63
resolve comments and add more examples
FeynmanZhou Dec 22, 2023
9d25f26
Merge branch 'main' into error-guide
qweeah Dec 25, 2023
1526f87
resolve comments and refine error guideline
FeynmanZhou Dec 25, 2023
4b36290
Merge branch 'main' of https://github.com/oras-project/oras into erro…
FeynmanZhou Dec 25, 2023
2c95904
Merge branch 'error-guide' of https://github.com/FeynmanZhou/oras int…
FeynmanZhou Dec 25, 2023
60f889e
resolve comments
FeynmanZhou Dec 25, 2023
7915e64
resolve comments
FeynmanZhou Dec 26, 2023
193ac35
resolve comments and refine spec
FeynmanZhou Jan 2, 2024
15eb3cc
add two examples and one suggestion
FeynmanZhou Jan 2, 2024
c423335
add two examples and one suggestion
FeynmanZhou Jan 2, 2024
889bb38
Merge branch 'main' of https://github.com/oras-project/oras into erro…
FeynmanZhou Jan 2, 2024
0f75e08
resolve comments and refine spec
FeynmanZhou Jan 2, 2024
4ce6b05
Merge branch 'main' of https://github.com/oras-project/oras into erro…
FeynmanZhou Jan 4, 2024
0de8d59
resolve comments
FeynmanZhou Jan 4, 2024
aaeb6bb
resolve comments
FeynmanZhou Jan 9, 2024
1a575b8
Merge branch 'main' of https://github.com/oras-project/oras into erro…
FeynmanZhou Jan 9, 2024
d8f921b
resolve comments
FeynmanZhou Jan 9, 2024
5568f2c
Update docs/proposals/error-handling-guideline.md
FeynmanZhou Jan 16, 2024
d338fce
Merge branch 'main' into error-guide
Feb 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
295 changes: 295 additions & 0 deletions docs/proposals/error-handling-guideline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,295 @@
# ORAS CLI Error Handling and Message Guideline

This document aims to provide the guidelines for ORAS contributors to improve existing error messages and error handling method as well as the new error output format. It will also provide recommendations and examples for ORAS CLI contributors for how to write friendly and standard error messages, avoid generating inconsistent and ambiguous error messages.

## General guiding principles

A clear and actionable error message is very important when raising an error, so make sure your error message describes clearly what the error is and tells users what they need to do if possible.
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved

First and foremost, make the error messages descriptive and informative. Error messages are expected to be helpful to troubleshoot where the user has done something wrong and the program is guiding them in the right direction. A great error message is recommended to contain the following elements:

FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
- HTTP status code: optional, when the logs are generated from the server side, it's recommended to print the HTTP status code in the error description
- Error description: describe what the error is
- Suggestion: for those errors that have potential solution, print out the recommended solution. Versioned troubleshooting document link is nice to have

Second, when necessary, it is highly suggested for ORAS CLI contributors to provide recommendations for users how to resolve the problems based on the error messages they encountered. Showing descriptive words and straightforward prompt with executable commands as a potential solution is a good practice for error messages.

Third, for unhandled errors you didn't expect the user to run into. For that, have a way to view full traceback information as well as full debug or verbose logs output, and instructions on how to submit a bug.

Fourth, signal-to-noise ratio is crucial. The more irrelevant output you produce, the longer it's going to take the user to figure out what they did wrong. If your program produces multiple errors of the same type, consider grouping them under a single explanatory header instead of printing many similar-looking lines.

Fifth, CLI program termination should follow the standard [Exit Status conventions](https://www.gnu.org/software/libc/manual/html_node/Exit-Status.html) to report execution status information about success or failure. ORAS returns `EXIT_FAILURE` if and only if ORAS reports one or more errors.

Last, error logs can also be useful for post-mortem debugging and can also be written to a file, truncate them occasionally and make sure they don't contain ansi color codes.

## Error output recommendation

### Dos

- Provide full description if the user input does not match what ORAS CLI expected. A full description should include the actual input received from the user and expected input
- Use the capital letter ahead of each line of any error message
- Print human readable error message. If the error message is mainly from the server and varies by different servers, tell users that the error response is from server. This implies that users may need to contact server side for troubleshooting
- Provide specific and actionable prompt message with argument suggestion or show the example usage for reference. (e.g, Instead of showing flag or argument options is missing, please provide available argument options and guide users to "--help" to view more examples)
- If the actionable prompt message is too long to show in the CLI output, consider guide users to ORAS user manual or troubleshooting guide with the versioned permanent link
- If the error message is not enough for troubleshooting, guide users to use "--verbose" to print much more detailed logs
- If the error message from the server side is too generic or has no value, especially in a few edge cases like example 12 and 13 below, consider providing customized and trimmed error logs to make it clearer. The original server logs can be displayed in debug mode
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved

### Don'Ts

- Do not use a formula-like or a programming expression in the error message. (e.g, `json: cannot unmarshal string into Go value of type map[string]map[string]string.`, or `Parameter 'xyz' must conform to the following pattern: '^[-\\w\\._\\(\\)]+$'`)
- Do not use ambiguous expressions which mean nothing to users. (e.g, `Something unexpected happens`, or `Error: accepts 2 arg(s), received 0`)
- Do not print irrelevant error message to make the output noisy. The more irrelevant output you produce, the longer it's going to take the user to figure out what they did wrong.

## How to write friendly error message

### Recommended error message structure

Here is a sample structure of an error message:
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved

```text
{Error|Error response from registry}: {Error description (HTTP status code can be printed out if any)}
[Usage: {Command usage}]
[{Recommended solution}]
```

- HTTP status code is an optional information. Printed out the HTTP status code if the error message is generated from the server side.
- Command usage is also an optional information but it's recommended to be printed out when user input doesn't follow the standard usage or examples.
- Recommended solution (if any) should follow the general guiding principles described above.

### Examples

Here are some examples of writing error message with helpful prompt actionable information:

#### Example 1: When no reference provided in `oras copy`

Current behavior and output:

```console
$ oras cp
Error: accepts 2 arg(s), received 0
```

Suggested error message:

```console
$ oras cp
Error: "oras copy" requires exactly 2 arguments but received 0.
Usage: oras copy [flags] <from>{:<tag>|@<digest>} <to>[:<tag>[,<tag>][...]]
Please specify 2 arguments as source and destination respectively. Run "oras copy -h" for more options and examples
```

#### Example 2: When user mistakenly uses `tag list` command

Current behavior and output:

```console
$ oras tag list ghcr.io/oras-project/oras
Error: unable to add tag for 'list': invalid reference: missing repository
```

Suggested error message:

```console
$ oras tag list ghcr.io/oras-project/oras
Error: There is no "list" sub-command for "oras tag" command.
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
Usage: oras tag [flags] <name>{:<tag>|@<digest>} <new_tag> [...]
If you want to list available tags in a repository, use "oras repo tags"
```

#### Example 3: No tag or digest provided when fetching a manifest

Current behavior and output:

```console
$ oras manifest fetch --oci-layout /tmp/ginkgo1163328512
Error: "/tmp/ginkgo1163328512": no tag or digest when expecting <name:tag|name@digest>
```

Suggested error message:

```console
$ oras manifest fetch --oci-layout /tmp/ginkgo1163328512
Error: "/tmp/ginkgo1163328512": no tag or digest specified
Usage: oras manifest fetch [flags] <name>{:<tag>|@<digest>}
You need to specify an artifact reference in the form of "<name>:<tag>" or "<name>@<digest>". Run "oras manifest fetch -h" for more options and examples
```

#### Example 4: Push a manifest if no media type flag provided
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved

Current behavior and output:

```console
$ oras manifest push --oci-layout /sample/images:foobar:mediatype
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
Error: media type is not recognized.
```

Suggested error message:

```console
$ oras manifest push --oci-layout /sample/images:foobar:mediatype
Error: media type is not recognized.
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
Usage: oras manifest push [flags] <name>[:<tag>[,<tag>][...]|@<digest>] <file>
You need to specify a valid media type in the manifest JSON or via the "--media-type" flag
```

#### Example 5: Attach an artifact if the given option is unknown

Current behavior and output:

```console
$ oras attach --artifact-type oras/test localhost:5000/command/images:foobar --distribution-spec v1.0
Error: unknown distribution specification flag: v1.0
```

Suggested error message:

```console
$ oras attach --artifact-type oras/test localhost:5000/sample/images:foobar --distribution-spec ???
Error: unknown distribution specification flag: "v1.0".
Available options: v1.1-referrers-api, v1.1-referrers-tag
```

#### Example 6: When attaching, if neither file reference nor annotation is provided

Current behavior and output:

```console
$ oras attach --artifact-type sbom/example localhost:5000/sample/images:foobar
Error: no blob is provided
```

Suggested error message:

```console
$ oras attach --artifact-type sbom/example localhost:5000/sample/images:foobar
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
Error: neither file nor annotation provided in the command
Usage: oras attach [flags] --artifact-type=<type> <name>{:<tag>|@<digest>} <file>[:<type>] [...]
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
To attach to an existing artifact, please provide files via argument or annotations via flag "--annotation". Run "oras attach -h" for more options and examples
```

#### Example 7: When pushing files, if the annotation file doesn't match the required format

Current behavior and output:

```console
$ oras push --annotation-file sbom.json ghcr.io/library/alpine:3.9
Error: failed to load annotations from sbom.json: json: cannot unmarshal string into Go value of type map[string]map[string]string. Please refer to the document at https://oras.land/docs/how_to_guides/manifest_annotations
```

Suggested error message:

```console
$ oras push --annotation-file annotation.json ghcr.io/library/alpine:3.9
Error: invalid annotation json file: failed to load annotations from annotation.json.
Annotation file doesn't match the required format. Please refer to the document at https://oras.land/docs/how_to_guides/manifest_annotations
```

#### Example 8: When pushing files, if the annotation value doesn't match the required syntax

Current behavior and output:

```console
$ oras push --annotation "key:value" ghcr.io/library/alpine:3.9
Error: missing key in `--annotation` flag: key:value
```

Suggested error message:

```console
$ oras push --annotation "key:value" ghcr.io/library/alpine:3.9
Error: annotation value doesn't match the required format.
Please use the correct format in the flag: --annotation "key=value"
```

#### Example 9: When failed to pull files from a public registry

```console
$ oras pull docker.io/nginx:latest
Error: failed to resolve latest: GET "https://registry-1.docker.io/v2/nginx/manifests/latest": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:nginx Type:repository]]
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
```

Suggested error message:

```console
$ oras pull docker.io/nginx:latest
Error response from registry: pull access denied for docker.io/nginx:latest : response status code 401: unauthorized: requested access to the resource is denied
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
Namespace is missing, do you mean `oras pull docker.io/library/nginx:latest`?
```

FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
#### Example 10: Neither registry nor OCI image layout provided when pushing a folder

Current behavior and output:

```console
$ oras push /oras --format json
Error: Head "https:///v2/oras/manifests/sha256:ffa50b27cd0096150c0338779c5090db41ba50d01179d851d68afa50b393c3a3": http: no Host in request URL
```

Suggested error message:

```console
$ oras push /oras --format json
Error: "/oras" is an invalid reference
Usage: oras push [flags] <name>[:<tag>[,<tag>][...]] <file>[:<type>] [...]
Please specify a valid reference in the form of <registry>/<repo>[:tag|@digest]
```

#### Example 11: Push a file or folder that doesn't exist

Current behavior and output:

```console
$ oras push localhost:5000/oras:v1 hello.txt
Error: failed to stat /home/user/hello.txt: stat /home/user/hello.txt: no such file or directory
```

Suggested error message:

```console
$ oras push localhost:5000/oras:v1 hello.txt
Error: /home/user/hello.txt: no such file or directory
```

#### Example 12: failed to authenticate with registry using an error credential from credential store

Current behavior and output:

```console
$ oras pull localhost:7000/repo:tag --registry-config auth.config
Error: failed to resolve tag: GET "http://localhost:7000/v2/repo/manifests/tag": credential required for basic auth
```

Suggested error message:

```console
$ oras pull localhost:7000/repo:tag --registry-config auth.config
Error: pulling content from localhost:7000/repo:tag failed with status: 401 Unauthorized
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
Please check whether the registry credential stored in the authentication file is correct
```

#### Example 13: failed to resolve the digest with incorrect username or password
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved

Current behavior and output:

```console
oras resolve localhost:7000/command/artifacts:foobar -u t -p 2
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from registry: <nil>
```

Suggested error message:

```console
oras resolve localhost:7000/command/artifacts:foobar -u t -p 2
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error: resolving the digest of artifact localhost:7000/command/artifacts:foobar failed with status: 401 Unauthorized
Authentication failed. Please verify your login credentials and try again.
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved
```

## Reference
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved

Parts of the content are borrowed from these guidelines.

- [Azure CLI Error Handling Guidelines](https://github.com/Azure/azure-cli/blob/dev/doc/error_handling_guidelines.md)
- [Command Line Interface Guidelines](https://clig.dev/#errors)
- [12 Factor CLI Apps](https://medium.com/@jdxcode/12-factor-cli-apps-dd3c227a0e46)
FeynmanZhou marked this conversation as resolved.
Show resolved Hide resolved