Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs with Transform FPE advanced I/O handling features #12744

Merged
merged 4 commits into from
Oct 15, 2021
Merged

Update docs with Transform FPE advanced I/O handling features #12744

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c00d898
Update docs with Trasnform FPE advanced I/O handling features.
victorr Oct 5, 2021
5cb8ee1
FIXUP: PR review fixes:
victorr Oct 6, 2021
24e80a0
FIXUP: PR fixes #2: examples for encode/decode formats, etc.
victorr Oct 8, 2021
14efaaa
FIXUP: PR review fixes #3: final documentation tweaks.
victorr Oct 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 88 additions & 37 deletions website/content/api-docs/secret/transform.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ configuration endpoints, and will be removed in a future release.
```json
{
"type": "fpe",
"template": "builtin/creditcardnumber",
"template": "example-template",
"tweak_source": "internal",
"allowed_roles": ["example-role"]
}
Expand Down Expand Up @@ -237,7 +237,7 @@ transformation exists, it will be updated with the new attributes.

```json
{
"template": "builtin/creditcardnumber",
"template": "example-template",
"tweak_source": "internal",
"allowed_roles": ["example-role"]
}
Expand Down Expand Up @@ -381,7 +381,7 @@ $ curl \
{
"data": {
"allowed_roles": ["example-role"],
"templates": ["builtin/creditcardnumber"],
"templates": ["example-template"],
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious— why remove the reference to the builtin template?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That change does look unwarranted. I'll undo it along with the other two other cases where I did the same.

"tweak_source": "internal",
"type": "fpe"
}
Expand Down Expand Up @@ -468,13 +468,26 @@ template exists, it will be updated with the new attributes.
Specifies the name of the alphabet to use when this template is used for FPE
encoding and decoding operations.

- `encode_format` `(string)` -
The regular expression template to use to format encoded values. This is only
used during FPE transformations.

- `decode_formats` `(key-value-map: {})` -
An optional map of regular expression templates that can be used to customize
decoded output. This is only used during FPE transformations.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a quick use-case-based example here, such as "customize decoded output. For example, this can be used to decode only the last four digits of a credit card number. This is only used during FPE transformations."

Copy link
Contributor Author

@victorr victorr Oct 7, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent idea, I'll add it.
Update: added.


### Sample Payload

```json
{
"type": "regex",
"pattern": "(\\d{9})",
"alphabet": "builtin/numeric"
"alphabet": "builtin/numeric",
"pattern": "(\\d{3})[-/](\\d{2})[-/](\\d{4})",
"encode_format": "$1-$2-$3",
"decode_formats": {
"first-three": "$1",
"last-four": "$3"
}
}
```

Expand Down Expand Up @@ -513,7 +526,12 @@ $ curl \
{
"data": {
"alphabet": "builtin/numeric",
"pattern": "(\\d{9})",
"decode_formats": {
"first-three": "$1",
"last-four": "$3"
},
"encode_format": "$1-$2-$3",
"pattern": "(\\d{3})[-/](\\d{2})[-/](\\d{4})",
"type": "regex"
}
}
Expand Down Expand Up @@ -708,9 +726,6 @@ The database user configured here should only have permission to `SELECT`,
Specifies the database driver to use, and thus which SQL database type.
Currently the supported options are `postgres` or `mysql`

- `supported_transformations: `(list: ["tokenization"])` The types of transformations this store can host. Currently only`tokenization`
is supported.

- `connection_string` `(string: <required>)` -
A database connection string with template slots for username and password that
Vault will use for locating and connecting to a database. Each
Expand All @@ -724,6 +739,9 @@ The database user configured here should only have permission to `SELECT`,
- `password`: `(string: <required>)` -
The password value to use when connecting to the database.

- `supported_transformations: `(list: ["tokenization"])` The types of transformations this store can host. Currently only`tokenization`
victorr marked this conversation as resolved.
Show resolved Hide resolved
is supported.

- `schema`: `(string: "public")` -
The schema within the database to expect tokenization state tables.

Expand All @@ -734,7 +752,7 @@ The database user configured here should only have permission to `SELECT`,
The maximum number of idle connections to the database at any given time.

- `max_connection_lifetime` `(duration: 0)` -
The maximum amount of time a connection can be open before closing it.
The maximum amount of time a connection can be open before closing it.
0 means no limit.

### Sample Payloads
Expand Down Expand Up @@ -787,15 +805,15 @@ operation.
Specifies the name of the store to create or update. This is part of
the request URL.

- `transformation_type`: `(string: "tokenization")` -
The transformation type. Currently only tokenization is supported.

- `username`: `(string: <required>)` -
The username value to use when connecting to the database.

- `password`: `(string: <required>)` -
The password value to use when connecting to the database.

- `transformation_type`: `(string: "tokenization")` -
The transformation type. Currently only tokenization is supported.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick, but you use code formatting elsewhere, so continue it here? tokenization

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


### Sample Payload

```json
Expand Down Expand Up @@ -934,16 +952,16 @@ This endpoint encodes the provided value using a named role.
transformations with `supplied` as the tweak source. The tweak must be a
7-byte value that is then base64 encoded.

- `reference` `(string: "")` -
- `reference` `(string: "")` -
Only valid on batch requests using 'batch_input' below. A user supplied
string that will be present on the corresponding item in the batch
string that will be present on the corresponding item in the batch
response, to assist in correlating inputs with outputs.

- `batch_input` `(array<object>: nil)` -
Specifies a list of items to be encoded in a single batch. When this
parameter is set, the 'value', 'transformation', 'ttl', 'tweak' and
'reference' parameters are ignored. Instead, the aforementioned parameters
should be provided within each object in the list.
'reference' parameters are ignored. Instead, the aforementioned parameters
should be provided within each object in the list.

```json
[
Expand Down Expand Up @@ -1048,9 +1066,9 @@ $ curl \

This endpoint decodes the provided value using a named role.

| Method | Path |
| :----- | :----------------------------- |
| `POST` | `/transform/decode/:role_name` |
| Method | Path |
| :----- | :---------------------------------------------- |
| `POST` | `/transform/decode/:role_name(/:decode_format)` |

### Parameters

Expand All @@ -1061,6 +1079,10 @@ This endpoint decodes the provided value using a named role.
- `value` `(string: <required>)` –
Specifies the value to be decoded.

- `decode_format` `(string)` -
victorr marked this conversation as resolved.
Show resolved Hide resolved
The name of the decode format to use for decoding. If one is not specified,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add some extra information here if you want? For example:

"The name of the decode format to use for decoding. These is defined in decode_formats when creating the transformation, and can be used to selectively decode or format the output. If one is not defined or specified, the template's pattern will be used. Only applicable for FPE transformations."

Just a suggestion, though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great suggestion. I will use the text with one modification: "... when creating the transformation's template".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

the template's pattern will be used. Only applicable for FPE transformations.

- `transformation` `(string)` –
Specifies the transformation within the role that should be used for this
decode operation. If a single transformation exists for role, this parameter
Expand All @@ -1071,16 +1093,16 @@ This endpoint decodes the provided value using a named role.
applicable for FPE transformations with `supplied` or `generated` as the tweak
source. The tweak must be a 7-byte value that is then base64 encoded.

- `reference` `(string: "")` -
- `reference` `(string: "")` -
Only valid on batch requests using 'batch_input' below. A user supplied
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: user supplied should be user-supplied

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

string that will be present on the corresponding item in the batch
string that will be present on the corresponding item in the batch
response, to assist in correlating inputs with outputs.

- `batch_input` `(array<object>: nil)` -
Specifies a list of items to be encoded in a single batch. When this
Specifies a list of items to be decoded in a single batch. When this
parameter is set, the 'value', 'transformation', 'tweak' and
'reference' parameters are ignored. Instead, the aforementioned parameters
should be provided within each object in the list.
'reference' parameters are ignored. Instead, the aforementioned parameters
should be provided within each object in the list.

```json
[
Expand All @@ -1093,10 +1115,39 @@ This endpoint decodes the provided value using a named role.

### Sample Payload

```json
{
"value": "418-56-4374",
"transformation": "example-transformation"
}
```

### Sample Request

```shell-session
$ curl \
--header "X-Vault-Token: ..." \
--request POST \
--data @payload.json \
http://127.0.0.1:8200/v1/transform/decode/example-role
```

### Sample Response

```json
{
"value": "5682-4613-6822-8064",
"transformation": "ccn-fpe"
"data": {
"decoded_value": "111-22-3333"
}
}
```

### Sample Payload

```json
{
"value": "418-56-4374",
"transformation": "example-transformation"
}
```

Expand All @@ -1107,15 +1158,15 @@ $ curl \
--header "X-Vault-Token: ..." \
--request POST \
--data @payload.json \
http://127.0.0.1:8200/v1/transform/encode/example-role
http://127.0.0.1:8200/v1/transform/decode/example-role/last-four
```

### Sample Response

```json
{
"data": {
"decoded_value": "1111-2222-3333-4444"
"decoded_value": "4444"
}
}
```
Expand Down Expand Up @@ -1190,16 +1241,16 @@ Only valid for tokenization transformations.
may be skipped and will be inferred. If multiple transformations exist, one
must be specified.

- `reference` `(string: "")` -
- `reference` `(string: "")` -
Only valid on batch requests using 'batch_input' below. A user supplied
string that will be present on the corresponding item in the batch
string that will be present on the corresponding item in the batch
response, to assist in correlating inputs with outputs.

- `batch_input` `(array<object>: nil)` -
Specifies a list of items to be validated in a single batch. When this
parameter is set, the 'value', 'transformation' and
'reference' parameters are ignored. Instead, the aforementioned parameters
should be provided within each object in the list.
'reference' parameters are ignored. Instead, the aforementioned parameters
should be provided within each object in the list.

```json
[
Expand Down Expand Up @@ -1265,9 +1316,9 @@ transformations.
may be skipped and will be inferred. If multiple transformations exist, one
must be specified.

- `reference` `(string: "")` -
- `reference` `(string: "")` -
Only valid on batch requests using 'batch_input' below. A user supplied
string that will be present on the corresponding item in the batch
string that will be present on the corresponding item in the batch
response, to assist in correlating inputs with outputs.

- `batch_input` `(array<object>: nil)` -
Expand Down Expand Up @@ -1339,9 +1390,9 @@ Only valid for tokenization transformations.
may be skipped and will be inferred. If multiple transformations exist, one
must be specified.

- `reference` `(string: "")` -
- `reference` `(string: "")` -
Only valid on batch requests using 'batch_input' below. A user supplied
string that will be present on the corresponding item in the batch
string that will be present on the corresponding item in the batch
response, to assist in correlating inputs with outputs.

- `batch_input` `(array<object>: nil)` -
Expand Down
20 changes: 18 additions & 2 deletions website/content/docs/secrets/transform/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ management tool.
```text
$ vault write transform/template/ccn \
type=regex \
pattern='(\d{4})-(\d{4})-(\d{4})-(\d{4})' \
pattern='(\d{4})[- ](\d{4})[- ](\d{4})[- ](\d{4})' \
encode_format='$1-$2-$3-$4' \
decode_formats=last-four='$4' \
alphabet=numerics
Success! Data written to: transform/template/ccn
```
Expand Down Expand Up @@ -106,6 +108,20 @@ values.
transformation. A tweak must be provided if the tweak source for the
transformation is "supplied" or "generated".

1. Decode some input value using the `/decode` endpoint with a named role and decode format:

```text
$ vault write transform/decode/payments/last-four value=9300-3376-4943-8903
Key Value
--- -----
decoded_value 4444
```

A transformation must be provided if the role contains more than one
transformation. A tweak must be provided if the tweak source for the
transformation is "supplied" or "generated". A decode format can optionally
be provided.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth adding here as well what happens if no decode format is provided.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a sentence explaining it.


## Roles, Transformations, Templates, and Alphabets

The Transform secrets engine contains several types of resources that
Expand Down Expand Up @@ -250,7 +266,7 @@ for external storage.
#### Mapping Modes

[Tokenization](transform/tokenization) stores the results of an encode operation
in storage using a cryptographic construct that enhances the safety of its values.
in storage using a cryptographic construct that enhances the safety of its values.
In the `default` mapping mode, the token itself is transformed via a one way
function involving the transform key and elements of the token. As Vault does
not store the token, the values in Vault storage themselves cannot be used to
Expand Down