This documentation build process is provided to the public purely for the purpose of testing documentation changes before submitting pull requests to the appropriate Elasticsearch repository.
The documents produced by this build process may be published only on http://www.elastic.co. They may not be published in any other form or on any other website without explicit permission.
First, you need to make sure that you have the following installed:
-
perl— systemperlis fine -
python— systempythonis fine -
xsltproc— if not already installed, see Installing a Processor
|
Tip
|
Windows users are recommended to use Cygwin (install asciidoc package and libxslt).
|
You may need to install extra perl modules that do not come with your system’s perl by default. For example, on Fedora 25, you will need to run:
sudo dnf install perl-Storable perl-Digest-MD5Clone the docs repository with:
git clone [email protected]:elastic/docs.gitYou can test that everything is working correctly by building this README as follows:
cd docs/
./build_docs.pl --doc README.asciidoc --openThis should convert README.asciidoc into HTML and open it
in your browser.
When you are making changes to documentation in a locally checked
out repository, and you want to check that they are building
correctly, use build_docs.pl with the --doc parameter to
generate the HTML version of the docs:
cd path/to/your/repo
build_docs.pl --doc path/to/index.asciidocBy default, the HTML docs are generated in ./html_docs. You can
change the output directory with the --out parameter:
build_docs.pl --doc path/to/index.asciidoc --out output/dir/|
Warning
|
The output/dir/ will be deleted and recreated, so don’t point it at a directory with contents that you want to keep.
|
To view the generated docs in your web browser immediately after
the build has finished, use the --open parameter:
build_docs.pl --doc path/to/index.asciidoc --openBy default, the build process generates an HTML file per
part/chapter/section. To generate all of the docs in a single
file instead, use the --single parameter:
build_docs.pl --doc path/to/index.asciidoc --singleAnd if you want a table of contents added at the beginning
of the single page output, add the --toc parameter:
build_docs.pl --doc path/to/index.asciidoc --single --toc|
Note
|
The multi-page output always contains tables-of-content where appropriate. |
Asciidoc combines all of the source files into one big file, which it converts to a single XML file for DocBook to process. Chunking is the process of breaking that XML file up into multiple HTML pages.
By default, each part (= Part Title) and chapter (== Chapter Title) is
`chunked'' into a separate HTML file. However, for bigger books, such as the
Elasticsearch reference docs, this results in enormous pages. These bigger
books are also chunked at the first section level (=== Section One Title`).
This behaviour is specified in the conf.yaml file, but can also be specified
on the command line when building the docs locally:
build_docs.pl --doc path/to/index.asciidoc --chunk 1Asciidoc is converted to DocBook, which is then converted to HTML. DocBook is strict, which means that you have to follow the rules (see Asciidoc Guide) otherwise the build process will throw an error.
The most common errors are:
-
Asciidoc syntax error, eg
=Titleinstead of= Titleor unclosed block delimiters, ie======or------ -
incorrect order of elements, eg
= Titlefollowed by a=== Level 3header -
Linking to an non-existent ID
Errors may be thrown either by Asciidoc or by DocBook. Asciidoc errors refer
to the actual .asciidoc page where the error occurred, but DocBook errors
only refer to a line in the generated XML file, and look something like this:
path/to/index.xml:1518: element xref: validity error : XXXXX
The easiest way to debug these is to open the XML file (index.xml in the
example above) and navigate to the specified line number (1518 in the
example above). This marks the end of the section containing the error.
Building a big book is time consuming. Unfortunately, in order to test that
all links exist, the entire book needs to be built in one go. However, while
you’re writing docs it is useful to be able to build just a single .asciidoc
page.
You can do this by telling build_docs.pl to ignore errors with the --lenient
parameter:
build_docs.pl --doc path/to/some/page.asciidoc --lenient --single --openIf the page you are building contains links to content in other pages, the above command will output warnings like:
ERROR: xref linking to relevance-intro has no generated link text.
|
Note
|
You should still build the whole book without the --lenient parameter
before commiting, to be sure that you haven’t broken any links.
|
Usually you don’t need to build all the docs that will be uploaded to the website, but if you are linking between documents (e.g. between the Java API docs and the main reference documentation), then building all of the docs will report any missing links.
You can build all the docs with:
build_docs.pl --allThe first time you run this will be slow as it needs to:
-
clone each repository
-
build the docs for each branch
Subsequent runs will pull any changes to the repos and only build the branches that have changed.
Because the docs are built from the remote repositories, you will
need to push your changes to the main repo before running
build_docs.pl --all.
Assuming you have already checked that your docs build correctly using the local build process, the only other errors that might occur at this stage are bad cross-document links.
Once the docs build correctly, you don’t need to do anything more. The changes that you have pushed to your repository will be picked up automatically by the docs build service.
The documentation that appears on the http://www.elastic.co/guide
website is controlled by the
conf.yaml file in the docs repo.
You can add a new repository under the repos section, if it
doesn’t already exist, and you can add a new `book'' under the
`contents section.
The repos.$NAME.branches[] key lists all of the branches which
should be built — all of these branches will be available on the
website — while repos.$NAME.current lists the branch which
should be used as the default version on the site.
|
Note
|
The branches and current settings can be overridden in
the config for each book. For instance, the Community Clients
docs are built only from the master branch.
|
When you release a new version of your code, you need to add
a new branch to the config and to update the current branch
for your project. Commit the change to conf.yaml and push
to the remote docs repo.
Asciidocs can be built as a book, article, manpage etc.
All our docs are built as a book, and thus follow the
layout for books. The most basic structure is as follows:
= Book title # level 0
== Chapter title # level 1
=== Section title # level 2
==== Section title # level 3
===== Section title # level 4Usually this structure will be sufficient for most of your documentation needs. More complicated ``books'', such as the Elasticsearch reference docs, however, require a few additional elements, described on the following pages.
By default, each chapter will generate a new chunk or HTML file. You can control the name of the file by giving the header an ID, as follows:
[[intro-to-xyz]]
== Intro to XYZThis chapter would then be written to a file called
intro-to-xyz.html. If no ID is provided, then a
filename will be auto-generated. See Controlling chunking
for more.
These IDs are also used to link to sections within each book. See Linking.
Books may also be divided into multiple parts, which are indicated
with level 0 headers:
= Book title # level 0
= Part title # level 0
== Chapter title # level 1
=== Section title # level 2
... etc ...Each part also creates a new chunk or HTML file.
A part may include text before the first chapter, but
it must be marked with [partintro] in order to be valid:
= Book title # level 0
= Part one # level 0
[partintro]
A paragraph introducing this Part
== Chapter title # level 1
... etc ...Longer partintro blocks should be wrapped in an
open block
which starts and ends with two dashes: --:
= Part two # level 0
[partintro]
.A partintro title
-- (1)
This section may contain multiple paragraphs.
[float]
== A header should use `[float]`
Everything up to the closing -- marker
will be considered part of the partintro.
-- (1)
== Chapter title # level 2
... etc ...-
The open block delimiters
Books may include other sections such as a preamble, a preface, a glossary or appendices.
[preface]
= Preface title # level 0
=== Preface header # level 2 (1)
= Part one # level 0and
[appendix]
= Appendix title # level 0
=== Appendix header # level 2 (1)-
Any headers in the appendix or in the preface start out-of-sequence at
level 2, not atlevel 1.
[glossary]
= Glossary title # level 0
[glossary]
Term one::
Defn for term one
Term two::
Defn for term two|
Note
|
The two
|
If you need to use some of these more advanced structural
elements, have a look at the example of a multi-part book
included in this repo in resources/asciidoc-8.6.8/doc/book-multi.txt.
A paragraph consists of multiple lines of text which start in the left hand column:
This is a paragraph
even though it contains
line breaks.
This is a second paragraph.Like most elements, a paragraph can have a title:
.Paragraph title
Text of my paragraphText of my paragraph
A paragraph which starts with TIP:, NOTE:, IMPORTANT:,
WARNING: or CAUTION: is rendered as an admonition paragraph,
eg:
NOTE: Compare admonition paragraphs with <<admon-blocks>>.This renders as:
|
Note
|
Compare admonition paragraphs with Admonition blocks. |
Literal paragraphs, which are rendered as <pre>
blocks without any source highlighting, must be
indented:
.Optional title
This para must
be indentedThis para must be indented
See also Code blocks for blocks with syntax highlighting.
Inline text can be formatted as follows:
emphasis
|
emphasis |
bold
|
bold |
| \`mono` |
|
| \`\`double quoted'' |
``double quoted'' |
| \`single quoted' |
`single quoted' |
superscript
|
superscript |
subscript
|
subscript |
These formatting characters expect to adjoin whitespace or
common punctuation characters. To combine bold with emphasis,
double up the quotes (ie use __ and **):
This example co__mb**in**es__ bold and emphasisThis example combines bold and emphasis.
Unwanted quotes can be escaped with a \ character.
You can link to any block in the document that has an ID — an
identifier before the block which is wrapped in double
square brackets:
[[para-id]]
This paragraph can be linked to using the ID `para-id`.When you need to combine an ID with a style, you can either specify each on a separate line:
[[note-id]]
[NOTE]
===============================
This note can be linked to using the ID `note-id`.
===============================or in one line:
["NOTE",id="note-id"] (1)
===============================
This note can be linked to using the ID `note-id`.
===============================-
In the one line format, the
NOTEmust be enclosed in double quotes.
Both of the above render as:
|
Note
|
This note can be linked to using the ID |
The ID is added to the HTML document as an <a> anchor
and, as explained in Controlling chunking, the ID is used as the
filename for sections which are chunked — written to
separate HTML files.
You can link to any ID within a document using double angle brackets:
* <<setup>>
* <<structure>>
* <<note-id>>It will use the title associated with each ID as the
link text. In the example above, note-id is not associated
with any title, which is why the text is rendered as "Note".
Alternative link text can be provided as a second parameter inside the angle brackets:
See the <<note-id,note about IDs>>.See the note about IDs.
Links to external websites can just be added as normal inline text, optionally with custom link text in square brackets:
See http://github.com/elasticsearch
or http://github.com/elasticsearch/docs[this repository]The existence of external links is not confirmed by the build process.
Links to other Elasticsearch docs are essentially the same as external links. However, for conciseness and maintainability, you should use an attribute to represent the absolute URL of the docs.
Attributes can be added to the beginning of the docs, under the book title:
= My Book Title
:ref: http://www.elastic.co/guide/elasticsearch/reference/current
Here is a link to the {ref}/search.html[search page]Here is a link to the search page
The main benefit of using attributes for cross document links is
that, when the docs for an old version contain links that
no longer exist in the current branch, you can update
all the links in the document to point to the older version,
by just updating a single attribute.
Cross document links are checked when build_docs.pl is
run with the --all parameter. See For the website.
Bullet point lists are written using asterisks:
.Optional title
* Point
* Point
** Sub-point
*** Sub-sub-point
* A point can have multiple paragraphs
+
But use a `+` instead of an empty line between paras.
An empty line signifies the end of the list.-
Point
-
Point
-
Sub-point
-
Sub-point
-
-
A point can have multiple paragraphs
But use a
+instead of an empty line between paras
An empty line signifies the end of the list.
Ordered lists use . instead of *, and will alternate
between numbers and letters automatically:
.Optional title
. foo
.. bar
... baz
.... balloo-
foo
-
bar
-
baz
-
balloo
-
-
-
Alternatively, you can control whether it uses a number or a letter as follows:
a. Start with a letter
b. Another letter
1. Now numbers
2. And more numbers-
Start with a letter
-
Another letter
-
Now numbers
-
And more numbers
-
Definition lists are used to define terms. The term must be
followed by a double colon :: eg:
term one:: Definition for term one
term two::
Can start on the next line
term three:: A definition can have multiple
+
paragraphs, but use `+` to separate them
term four::: Definitions can be nested
by adding more colons
term five:: A definition can even include
lists:
* point one
* point two- term one
-
Definition for term one
- term two
-
Can start on the next line
- term three
-
A definition can have multiple
paragraphs, but use
+to separate them- term four
-
Definitions can be nested by adding more colons
- term five
-
A definition can even include lists:
-
point one
-
point two
-
Often definition lists are better rendered horizontally, eg:
[horizontal]
term one:: Definition for term one
term two::
Can start on the next line
term three:: A definition can have multiple
+
paragraphs, but use `+` to separate them
term four::: Definitions can be nested
by adding more colons
term five:: A definition can even include
lists:
* point one
* point two| term one |
Definition for term one |
| term two |
Can start on the next line |
| term three |
A definition can have multiple paragraphs, but use
|
| term five |
A definition can even include lists:
|
Blocks are used for special blocks of content, such as Code blocks, Example blocks, Sidebars and Admonition blocks.
Blocks are delimited with a start and end line which uses
the same characters, like =====.
Code blocks are rendered as <pre> blocks, and use
syntax highlighting, eg:
.Optional title
[source,js]
----------------------------------
{
"query": "foo bar"
}
----------------------------------{
"query": "foo bar"
}|
Important
|
The language to use for source highlighting — eg js above — must be specified, otherwise Asciidoc
emits invalid DocBook.
|
Code blocks can use callouts to add an explanatory footnote to a particular line of code:
[source,js]
----------------------------------
{
"query": "foo bar" <1>
}
----------------------------------
\<1> Here's the explanation{
"query": "foo bar" (1)
}-
Here’s the explanation
Code blocks can be followed by a `View in Sense'' link which, when clicked,
will open the code snippet in Sense. The snippet can either be taken directly
from the code block (`AUTOSENSE), or be a link to a custom snippet.
[source,js]
----------------------------------
GET /_search
{
"query": "foo bar" <1>
}
----------------------------------
// AUTOSENSE (1)
\<1> Here's the explanation-
The
// AUTOSENSEline must follow immediately after the code block, before any callouts.
GET /_search
{
"query": "foo bar" (1)
}-
Here’s the explanation
|
Note
|
The ``View in Sense'' links will only work if the docs are viewed in web-browser mode, as follows: build_docs.pl -d my_doc.asciidoc --open <1> --web (2)
The local web browser can be stopped with |
Sometimes you will want to show a small amount of code in the code block, but to provide a full recreation in the Sense snippet. In this case, you need to:
-
Save the snippet file in the
./snippets/directory in the root docs directory. -
Under the code block, specify the name of the snippet file with
// SENSE: path/to/snippet.json
For instance, to add a custom snippet to the file ./one/two/three.asciidoc, save the snippet
to ./snippets/one/two/three/example_1.json, then add the SENSE link below the code block:
[source,js]
----------------------------------
GET /_search
{
"query": "foo bar" <1>
}
----------------------------------
// SENSE:one/two/three/example_1.json (1)
\<1> Here's the explanation-
The path should not contain the initial
snippetsdirectory
Admonition blocks are much the same as Admonition paragraphs, except that they can be longer and contain more than just a paragraph. For instance:
[NOTE]
=========================
This note contains a list:
* foo
* bar
* baz
and some code
[source,js]
----------------------------------
{ "query": "foo bar"}
----------------------------------
=========================This renders as:
|
Note
|
This note contains a list:
and some code { "query": "foo bar"} |
Sidebars are used to highlight a block of content that is outside the usual flow of text:
.Optional title
**********************************
So why does the `bulk` API have such a
funny format? Sit down and I'll tell you
all about it!
**********************************So why does the bulk API have such a
funny format? Sit down and I’ll tell you
all about it!
Example blocks contain normal text which is used as an example. The title, if any, is labelled as an example and numbered:
.My first example
========================================
Text explaining the first example.
========================================
.My second example
========================================
Text explaining the second example.
========================================This renders as:
Text explaining the first example.
Text explaining the second example.
|
Caution
|
The === and --- delimiters can
sometimes be confused with a header, resulting
in an error. To resolve this, add newlines
between the delimiter and the content
before and after it.
|
For long documentation, you probably want to break up the Asciidoc files into smaller units, and just include them where appropriate:
include::myfolder/mydoc.asciidoc[]Paths are relative to the file which
contains the include statement.
Documentation is built for various branches, eg 0.90,
1.00, master. However, we release versions
0.90.0, 0.90.1, etc, which are all based on the
0.90 branch.
When adding new functionality to a branch, or deprecating
existing functionality, you can mark the change as
added, coming or deprecated. Use coming when the addition is
in an as yet unreleased version of the current branch, and added when
the functionality is already released.
The update_versions.pl script can be used to change coming notices
to added notices when doing a new release, and can also be used
to remove added, coming and deprecated notices completely.
Use inline notifications for small changes, such as the addition or deprecation of individual parameters.
[horizontal]
`foo.bar`:: Does XYZ. added[0.90.4]
`foo.bar`:: Does XYZ. coming[0.90.4]
`foo.baz`:: Does XYZ. deprecated[0.90.4]
foo.bar
|
Does XYZ. added[0.90.4] |
foo.bar
|
Does XYZ. coming[0.90.4] |
foo.baz
|
Does XYZ. deprecated[0.90.4] |
You can also include details about additional notes in the notifications which show up when the user hovers over it:
[horizontal]
`foo.bar`:: Does XYZ. added[0.90.4,Replaces `foo.baz`]
`foo.bar`:: Does XYZ. coming[0.90.4,Replaces `foo.baz`]
`foo.baz`:: Does XYZ. deprecated[0.90.4,Replaced by `foo.bar`]
foo.bar
|
Does XYZ. added[0.90.4,Replaces |
foo.bar
|
Does XYZ. coming[0.90.4,Replaces |
foo.baz
|
Does XYZ. deprecated[0.90.4,Replaced by |
Use section notifications to mark an entire chapter or section as added/deleted. Notifications can just refer to the version in which the change was made:
==== New section
added[0.90.4]
Text about new functionality...
==== New section not yet released
coming[0.90.9]
Text about new functionality...
==== Old section
deprecated[0.90.4]
Text about old functionality...Or they can include extra text, including more Asciidoc markup:
[[new-section]]
==== New section
added[0.90.4,Replaces `foo.bar`. See <<old-section>>]
Text about new functionality...
[[coming-section]]
==== New section not yet released
coming[0.90.9,Replaces `foo.bar`. See <<old-section>>]
Text about new functionality...
[[old-section]]
==== Old section
deprecated[0.90.4,Replace by `foo.baz`. See <<new-section>>]
Text about old functionality...APIs or parameters that are experimental or in beta can be marked as such, using markup similar to that used in Additions and deprecations. For instance:
[[new-feature]]
=== New experimental feature
experimental[]
experimental[Custom text goes here]
Text about new feature...
[[old-feature]]
=== Established feature
This feature has been around for a while, but we're adding
a new experimental parameter:
[horizontal]
`established_param`:: This param has been around for ages and won't change.
`experimental_param`:: experimental[] This param is experimental and may change in the future.
`experimental_param`:: experimental[Custom text goes here] This param is experimental and may change in the future.experimental[]
experimental[Custom text goes here]
Text about new feature…
This feature has been around for a while, but we’re adding a new experimental parameter:
established_param
|
This param has been around for ages and won’t change. |
experimental_param
|
experimental[] This param is experimental and may change in the future. |
experimental_param
|
experimental[Custom text goes here] This param is experimental and may change in the future. |
[[new-beta-feature]]
=== New beta feature
beta[]
beta[Custom text goes here]
Text about new feature...
[[old-beta-feature]]
=== Established feature
This feature has been around for a while, but we're adding
a new beta parameter:
[horizontal]
`established_param`:: This param has been around for ages and won't change.
`beta_param`:: beta[] This param is beta and may change in the future.
`beta_param`:: beta[Custom text goes here] This param is beta and may change in the future.This feature has been around for a while, but we’re adding a new beta parameter:
established_param
|
This param has been around for ages and won’t change. |
beta_param
|
beta[] This param is experimental and may change in the future. |
beta_param
|
beta[Custom text goes here] This param is beta and may change in the future. |
Any images you want to include should be saved in a folder
in your repo, and included using a path relative
to the document where the image:: statement appears.
[[cat]]
.A scaredy cat
image::resources/cat.jpg[Alt text]
A link to <<cat>>
A link to A scaredy cat
The width and/or height of the image can be
specified in pixels or as a percentage:
image::resources/cat.jpg["Alt text",width=50]
image::resources/cat.jpg["Alt text",width="20%"]
In general, tables are frowned upon in DocBook as they don’t display well in formats other than HTML, eg PDF, ePub, etc.
It’s almost always better to use Horizontal definition lists instead, but if you really want to use tables, you can read about them here.
In Basic book structure, we said that each part or chapter generates
a new chunk or HTML file. For more complex documentation,
you may want the first level of sections to also generate
new chunks.
For instance, in the ES reference docs, we have:
= Search APIs # part
== Request body search # chapter
=== Query # section level 1
=== From/Size # section level 1
... etc ...There are too many parameters for ``Request body search'' to list them all on one page. In this case, it is worth turning on chunking for top level sections.
To enable section chunking when building docs in a local repository,
pass the --chunk parameter:
build_docs.pl --doc path/to/index.asciidoc --chunk 1To enable section chunking when building docs for the website,
add chunk: 1 to the
conf.yaml file in the docs repo.
contents:
-
title: Elasticsearch reference
prefix: elasticsearch/reference
repo: elasticsearch
index: docs/reference/index.asciidoc
chunk: 1 (1)-
Chunking is enabled for this book
If you enable session chunking, you will probably find that you have a few short sections which you want to keep on the same page.
To do this, you can use the [float] marker before a
section header, to tell Asciidoc that what follows isn’t
a ``real'' header:
[[chapter-one]]
== chapter # new chunk
[[section-one]]
=== Section one # new chunk
[[section-two]]
[float]
=== Section two # same chunk
[[section-three]]
=== Section three # new chunkThe above would produce three HTML files, named for their IDs:
-
chapter-one.html -
section-one.htmlwhich would also contain ``Section two'' -
section-three.html
To link to `Section two'' from an external
document, you would use the URL: `section-one.html#section-two