Skip to content

Commit

Permalink
Build: Enable testing without magic comments (elastic#46180)
Browse files Browse the repository at this point in the history
Previously we only turned on tests if we saw either `// CONSOLE` or
`// TEST`. These magic comments are difficult for the docs build to deal
with so it has moved away from using them where possible. We should
catch up. This adds another trigger to enable testing: marking a snippet
with the `console` language. It looks like this:

```
[source,console]
----
GET /
----
```

This saves a line which is nice, I guess. But it is more important to me
that this is consistent with the way the docs build works now.

Similarly this enables response testing when you mark a snippet with the
language `console-result`. That looks like:
```
[source,console-result]
----
{
  "result": "0.1"
}
----
```

`// TESTRESPONSE` is still available for situations like `// TEST`: when
the response isn't *in* the console-result language (like `_cat`) or
when you want to perform substitutions on the generated test.

Should unblock elastic#46159.
  • Loading branch information
nik9000 committed Sep 4, 2019
1 parent 1a29711 commit c504142
Show file tree
Hide file tree
Showing 4 changed files with 34 additions and 27 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -202,11 +202,12 @@ public class RestTestsFromSnippetsTask extends SnippetsTask {
previousTest = snippet
return
}
if (snippet.testResponse) {
if (snippet.testResponse || snippet.language == 'console-result') {
response(snippet)
return
}
if (snippet.test || snippet.console) {
if (snippet.test || snippet.console ||
snippet.language == 'console') {
test(snippet)
previousTest = snippet
return
Expand Down
35 changes: 24 additions & 11 deletions docs/README.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,14 @@ Elasticsearch documentation build process.

See: https://github.com/elastic/docs

Snippets marked with `// CONSOLE` are automatically annotated with "VIEW IN
CONSOLE" and "COPY AS CURL" in the documentation and are automatically tested
by the command `gradle :docs:check`. To test just the docs from a single page,
use e.g. `./gradlew :docs:integTestRunner --tests "*rollover*"`.
Snippets marked with `[source,console]` are automatically annotated with
"VIEW IN CONSOLE" and "COPY AS CURL" in the documentation and are automatically
tested by the command `./gradlew -pdocs check`. To test just the docs from a
single page, use e.g. `./gradlew -ddocs integTestRunner --tests "*rollover*"`.

NOTE: Previously we use `// CONSOLE` instead of `[source,console]`. This worked
well for a long time so you'll see it all over early branches but we're phasing
it out because it requires some unpleasant hackery on the docs build side.

NOTE: If you have an elasticsearch-extra folder alongside your elasticsearch
folder, you must temporarily rename it when you are testing 6.3 or later branches.
Expand Down Expand Up @@ -45,10 +49,21 @@ for its modifiers:
header. If the response doesn't include a `Warning` header with the exact
text then the test fails. If the response includes `Warning` headers that
aren't expected then the test fails.
* `// TESTRESPONSE`: Matches this snippet against the body of the response of
the last test. If the response is JSON then order is ignored. If you add
`// TEST[continued]` to the snippet after `// TESTRESPONSE` it will continue
in the same test, allowing you to interleave requests with responses to check.
* `[source,console-result]`: Matches this snippet against the body of the
response of the last test. If the response is JSON then order is ignored. If
you add `// TEST[continued]` to the snippet after `[source,console-result]`
it will continue in the same test, allowing you to interleave requests with
responses to check.
* `// TESTRESPONSE`: Explicitly marks a snippet as a test response even without
`[source,console-result]`. Similarly to `// TEST` this is mostly used for
its modifiers.
* You can't use `[source,console-result]` immediately after `// TESTSETUP`.
Instead, consider using `// TEST[continued]` or rearrange your snippets.
NOTE: Previously we only used `// TESTRESPONSE` instead of
`[source,console-result]` so you'll see that a lot in older branches but we
prefer `[source,console-result]` now.
* `// TESTRESPONSE[s/foo/bar/]`: Substitutions. See `// TEST[s/foo/bar]` for
how it works. These are much more common than `// TEST[s/foo/bar]` because
they are useful for eliding portions of the response that are not pertinent
Expand All @@ -62,8 +77,6 @@ for its modifiers:
"figures out" the path. This is especially useful for making sweeping
assertions like "I made up all the numbers in this example, don't compare
them" which looks like `// TESTRESPONSE[s/\d+/$body.$_path/]`.
* You can't use `// TESTRESPONSE` immediately after `// TESTSETUP`. Instead,
consider using `// TEST[continued]` or rearrange your snippets.
* `// TESTRESPONSE[non_json]`: Add substitutions for testing responses in a
format other than JSON. Use this after all other substitutions so it doesn't
make other substitutions difficult.
Expand Down Expand Up @@ -98,7 +111,7 @@ endyaml
```

This allows slightly more expressive testing of the snippets. Since that syntax
is not supported by CONSOLE the usual way to incorporate it is with a
is not supported by `[source,console]` the usual way to incorporate it is with a
`// TEST[s//]` marker like this:

```
Expand Down
18 changes: 6 additions & 12 deletions docs/painless/painless-guide/painless-execute-script.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ If no context is specified then this context is used by default.

Request:

[source,js]
[source,console]
----------------------------------------------------------------
POST /_scripts/painless/_execute
{
Expand All @@ -43,17 +43,15 @@ POST /_scripts/painless/_execute
}
}
----------------------------------------------------------------
// CONSOLE

Response:

[source,js]
[source,console-result]
--------------------------------------------------
{
"result": "0.1"
}
--------------------------------------------------
// TESTRESPONSE

===== Filter context

Expand All @@ -69,7 +67,7 @@ index:: The name of an index containing a mapping that is compatible with the do

*Example*

[source,js]
[source,console]
----------------------------------------------------------------
PUT /my-index
{
Expand Down Expand Up @@ -99,17 +97,15 @@ POST /_scripts/painless/_execute
}
}
----------------------------------------------------------------
// CONSOLE

Response:

[source,js]
[source,console-result]
--------------------------------------------------
{
"result": true
}
--------------------------------------------------
// TESTRESPONSE


===== Score context
Expand All @@ -125,7 +121,7 @@ query:: If `_score` is used in the script then a query can specified that will b

*Example*

[source,js]
[source,console]
----------------------------------------------------------------
PUT /my-index
{
Expand Down Expand Up @@ -159,14 +155,12 @@ POST /_scripts/painless/_execute
}
}
----------------------------------------------------------------
// CONSOLE

Response:

[source,js]
[source,console-result]
--------------------------------------------------
{
"result": 0.8
}
--------------------------------------------------
// TESTRESPONSE
3 changes: 1 addition & 2 deletions docs/reference/docs/index_.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ comma-separated list of patterns you want to allow, or prefix each pattern with
`+` or `-` to indicate whether it should be allowed or blocked. When a list is
specified, the default behaviour is to disallow.

[source,js]
[source,console]
--------------------------------------------------
PUT _cluster/settings
{
Expand All @@ -164,7 +164,6 @@ PUT _cluster/settings
}
}
--------------------------------------------------
// CONSOLE

<1> Allow auto-creation of indices called `twitter` or `index10`, block the
creation of indices that match the pattern `index1*`, and allow creation of
Expand Down

0 comments on commit c504142

Please sign in to comment.