diff --git a/docs/developer/core-development.asciidoc b/docs/developer/core-development.asciidoc index 447a4c6d2601b..8f356abd095f2 100644 --- a/docs/developer/core-development.asciidoc +++ b/docs/developer/core-development.asciidoc @@ -7,6 +7,7 @@ * <> * <> * <> +* <> include::core/development-basepath.asciidoc[] @@ -19,3 +20,5 @@ include::core/development-elasticsearch.asciidoc[] include::core/development-unit-tests.asciidoc[] include::core/development-functional-tests.asciidoc[] + +include::core/development-es-snapshots.asciidoc[] diff --git a/docs/developer/core/development-es-snapshots.asciidoc b/docs/developer/core/development-es-snapshots.asciidoc new file mode 100644 index 0000000000000..4cd4f31e582db --- /dev/null +++ b/docs/developer/core/development-es-snapshots.asciidoc @@ -0,0 +1,113 @@ +[[development-es-snapshots]] +=== Daily Elasticsearch Snapshots + +For local development and CI, Kibana, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests). + +A snapshot is just a group of tarballs, one for each supported distribution/architecture/os of Elasticsearch, and a JSON-based manifest file containing metadata about the distributions. + +https://ci.kibana.dev/es-snapshots[A dashboard] is available that shows the current status and compatibility of the latest Elasticsearch snapshots. + +==== Process Overview + +1. Elasticsearch snapshots are built for each current tracked branch of Kibana. +2. Each snapshot is uploaded to a public Google Cloud Storage bucket, `kibana-ci-es-snapshots-daily`. +** At this point, the snapshot is not automatically used in CI or local development. It needs to be tested/verified first. +3. Each snapshot is tested with the latest commit of the corresponding Kibana branch, using the full CI suite. +4. After CI +** If the snapshot passes, it is promoted and automatically used in CI and local development. +** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and Kibana. + +==== Using the latest snapshot + +When developing locally, you may wish to use the most recent Elasticsearch snapshot, even if it's failing CI. To do so, prefix your commands with the follow environment variable: + +["source","bash"] +----------- +KBN_ES_SNAPSHOT_USE_UNVERIFIED=true +----------- + +You can use this flag with any command that downloads and runs Elasticsearch snapshots, such as `scripts/es` or the FTR. + +For example, to run functional tests with the latest snapshot: + +["source","bash"] +----------- +KBN_ES_SNAPSHOT_USE_UNVERIFIED=true node scripts/functional_tests_server +----------- + +===== For Pull Requests + +Currently, there is not a way to run your pull request with the latest unverified snapshot without a code change. You can, however, do it with a small code change. + +1. Edit `Jenkinsfile` in the root of the Kibana repo +2. Add `env.KBN_ES_SNAPSHOT_USE_UNVERIFIED = 'true'` at the top of the file. +3. Commit the change + +Your pull request should then use the latest snapshot the next time that it runs. Just don't merge the change to `Jenkinsfile`! + +==== Google Cloud Storage buckets + +===== kibana-ci-es-snapshots-daily + +This bucket stores snapshots that are created on a daily basis, and is the primary location used by `kbn-es` to download snapshots. + +Snapshots are automatically deleted after 10 days. + +The file structure for this bucket looks like this: + +* `/manifest-latest.json` +* `/manifest-latest-verified.json` +* `/archives//*.tar.gz` +* `/archives//*.tar.gz.sha512` +* `/archives//manifest.json` + +===== kibana-ci-es-snapshots-permanent + +This bucket stores only the most recently promoted snapshot for each version. Old snapshots are only deleted when new ones are uploaded. + +This bucket serves as permanent snapshot storage for old branches/versions that are no longer being built. `kbn-es` checks the daily bucket first, followed by this one if no snapshots were found. + +The file structure for this bucket looks like this: + +* `/*.tar.gz` +* `/*.tar.gz.sha512` +* `/manifest.json` + +==== How snapshots are built, tested, and promoted + +Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of Kibana. + +===== Build + +https://kibana-ci.elastic.co/job/elasticsearch+snapshots+build/[This Jenkins job] builds the Elasticsearch snapshots and uploads them to GCS. + +The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the kibana repo]. + +1. Checkout Elasticsearch repo for the given branch/version. +2. Run `./gradlew -p distribution/archives assemble --parallel` to create all of the Elasticsearch distributions. +3. Create a tarball for each distribution. +4. Create a manifest JSON file containing info about the distribution, as well as its download URL. +5. Upload the tarballs and manifest to a unique location in the GCS bucket `kibana-ci-es-snapshots-daily`. +** e.g. `/archives/` +6. Replace `/manifest-latest.json` in GCS with this newest manifest. +** This allows the `KBN_ES_SNAPSHOT_USE_UNVERIFIED` flag to work. +7. Trigger the verification job, to run the full Kibana CI test suite with this snapshot. + +===== Verification and Promotion + +https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full Kibana CI pipeline, and promotes if it there are no test failures. + +The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the kibana repo]. + +1. Checkout Kibana and set up CI environment as normal. +2. Set the `ES_SNAPSHOT_MANIFEST` env var to point to the latest snapshot manifest. +3. Run CI (functional tests, integration tests, etc). +4. After CI +** If there was a test failure or other build error, send out an e-mail notification and stop. +** If there were no errors, promote the snapshot. + +Promotion is done as part of the same pipeline: + +1. Replace the manifest at `kibana-ci-es-snapshots-daily//manifest-latest-verified.json` with the manifest from the tested snapshot. +** At this point, the snapshot has been promoted and will automatically be used in CI and in local development. +2. Replace the snapshot at `kibana-ci-es-snapshots-permanent//` with the tested snapshot by copying all of the tarballs and the manifest file. \ No newline at end of file