Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Various versions (7.13.2, 7.11.1) crashing on Apple Silicon (Old versions like 7.10.2 working) #81383

Closed
dewey opened this issue Dec 6, 2021 · 11 comments
Closed

Various versions (7.13.2, 7.11.1) crashing on Apple Silicon (Old versions like 7.10.2 working) #81383

dewey opened this issue Dec 6, 2021 · 11 comments
Assignees
@pugnascotia
Labels
>bug :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts Team:Delivery Meta label for Delivery team

Comments

@dewey
Copy link

dewey commented Dec 6, 2021

Elasticsearch version (bin/elasticsearch --version):

  • 7.13.2 (Crash)
  • 7.11.1 (Crash)
  • 7.10.2 (Works)

Plugins installed: Using default docker image with no additions

JVM version (java -version): Using default docker image with no additions

OS version (uname -a if on a Unix-like system):

~|⇒ uname -a
Darwin computer.local 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:01 PDT 2021; root:xnu-8019.41.5~1/RELEASE_ARM64_T6000 arm64

Description of the problem including expected versus actual behavior:

Running Docker images on Apple Silicon isn't working as the container is crashing seconds after startup. Using Docker for Mac. (Docker Desktop 4.3.0 (71786) is currently the newest version available.).

  Model Identifier:	MacBookPro18,3
  Chip:	Apple M1 Pro
  Total Number of Cores:	10 (8 performance and 2 efficiency)
  Memory:	32 GB

Steps to reproduce:

  1. Run docker run with the appropriate image
  2. Run docker logs -f
  3. See how container is in a crash loop

Command used in the lows below.

Provide logs (if relevant):

Crash with 7.13.2: docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.13.2 {"type": "server", "timestamp": "2021-12-06T17:58:59,131Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "version[7.13.2], pid[9], build[default/docker/4d960a0733be83dd2543ca018aa4ddc42e956800/2021-06-10T21:01:55.251515791Z], OS[Linux/5.10.76-linuxkit/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]" } {"type": "server", "timestamp": "2021-12-06T17:58:59,132Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" } {"type": "server", "timestamp": "2021-12-06T17:58:59,132Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-7687026872334525382, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms3975m, -Xmx3975m, -XX:MaxDirectMemorySize=2084569088, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,830Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [aggs-matrix-stats]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,830Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [analysis-common]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,830Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [constant-keyword]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [frozen-indices]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [ingest-common]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [ingest-geoip]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [ingest-user-agent]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [kibana]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [lang-expression]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [lang-mustache]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [lang-painless]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [mapper-extras]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [mapper-version]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [parent-join]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [percolator]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [rank-eval]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [reindex]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [repositories-metering-api]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [repository-encrypted]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [repository-url]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [runtime-fields-common]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [search-business-rules]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [searchable-snapshots]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [snapshot-repo-test-kit]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [spatial]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [transform]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [transport-netty4]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [unsigned-long]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [vectors]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [wildcard]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-aggregate-metric]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-analytics]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-async]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-async-search]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-autoscaling]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-ccr]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-core]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-data-streams]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-deprecation]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-enrich]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-eql]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-fleet]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-graph]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-identity-provider]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-ilm]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-logstash]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-ml]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-monitoring]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-ql]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-rollup]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-security]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-shutdown]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-sql]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-stack]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,836Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-text-structure]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,836Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-voting-only-node]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,836Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "loaded module [x-pack-watcher]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,836Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "no plugins loaded" } {"type": "server", "timestamp": "2021-12-06T17:59:00,858Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "using [1] data paths, mounts [[/ (overlay)]], net usable_space [49gb], net total_space [58.4gb], types [overlay]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,858Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "heap size [3.8gb], compressed ordinary object pointers [true]" } {"type": "server", "timestamp": "2021-12-06T17:59:00,872Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0f39460a537e", "message": "node name [0f39460a537e], node ID [uoNbrw4LRQevNmy37ah8cg], cluster name [docker-cluster], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
Crash with 7.11.1: docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.11.1 {"type": "server", "timestamp": "2021-12-06T18:01:34,944Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "version[7.11.1], pid[7], build[default/docker/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/5.10.76-linuxkit/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]" } {"type": "server", "timestamp": "2021-12-06T18:01:34,946Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" } {"type": "server", "timestamp": "2021-12-06T18:01:34,946Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-6557252020405306841, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms3975m, -Xmx3975m, -XX:MaxDirectMemorySize=2084569088, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,671Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [aggs-matrix-stats]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,672Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [analysis-common]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,672Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [constant-keyword]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,672Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [flattened]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,672Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [frozen-indices]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,673Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [ingest-common]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,673Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [ingest-geoip]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,673Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [ingest-user-agent]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,673Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [kibana]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,674Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [lang-expression]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,674Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [lang-mustache]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,674Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [lang-painless]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,674Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [mapper-extras]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,674Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [mapper-version]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,675Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [parent-join]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,675Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [percolator]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,675Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [rank-eval]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,676Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [reindex]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,676Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [repositories-metering-api]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,676Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [repository-url]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,677Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [search-business-rules]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,677Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [searchable-snapshots]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,677Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [spatial]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,677Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [transform]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,678Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [transport-netty4]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,678Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [unsigned-long]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,678Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [vectors]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,679Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [wildcard]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,679Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-aggregate-metric]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,679Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-analytics]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,679Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-async]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-async-search]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-autoscaling]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-ccr]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-core]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-data-streams]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,681Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-deprecation]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,681Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-enrich]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,681Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-eql]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,681Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-fleet]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,682Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-graph]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,682Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-identity-provider]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,683Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-ilm]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,683Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-ingest]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,683Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-logstash]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,683Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-ml]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,684Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-monitoring]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,684Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-ql]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,684Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-rollup]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,684Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-runtime-fields]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,685Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-security]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,685Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-sql]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,685Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-stack]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,685Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-voting-only-node]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,686Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "loaded module [x-pack-watcher]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,686Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "no plugins loaded" } {"type": "server", "timestamp": "2021-12-06T18:01:36,713Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "using [1] data paths, mounts [[/ (overlay)]], net usable_space [49gb], net total_space [58.4gb], types [overlay]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,714Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "heap size [3.8gb], compressed ordinary object pointers [true]" } {"type": "server", "timestamp": "2021-12-06T18:01:36,746Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "node name [21d73683bd56], node ID [2H170Z2oQRyCftyyMFOECA], cluster name [docker-cluster], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" } {"type": "server", "timestamp": "2021-12-06T18:01:39,271Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "21d73683bd56", "message": "[controller/204] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV" }
Working with 7.10.2: docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.2 {"type": "server", "timestamp": "2021-12-06T18:02:42,879Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "version[7.10.2], pid[9], build[default/docker/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T04:42:47.157277Z], OS[Linux/5.10.76-linuxkit/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]" } {"type": "server", "timestamp": "2021-12-06T18:02:42,880Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" } {"type": "server", "timestamp": "2021-12-06T18:02:42,881Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-16502364269080688024, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,503Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [aggs-matrix-stats]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,503Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [analysis-common]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,503Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [constant-keyword]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,504Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [flattened]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,504Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [frozen-indices]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,504Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [ingest-common]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,504Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [ingest-geoip]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,504Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [ingest-user-agent]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [kibana]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [lang-expression]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [lang-mustache]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [lang-painless]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [mapper-extras]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [mapper-version]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [parent-join]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,505Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [percolator]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [rank-eval]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [reindex]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [repositories-metering-api]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [repository-url]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [search-business-rules]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [searchable-snapshots]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [spatial]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [transform]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,506Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [transport-netty4]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [unsigned-long]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [vectors]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [wildcard]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-analytics]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-async]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-async-search]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-autoscaling]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-ccr]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-core]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-data-streams]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-deprecation]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,507Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-enrich]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-eql]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-graph]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-identity-provider]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-ilm]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-logstash]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-ml]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-monitoring]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-ql]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,508Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-rollup]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,509Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-security]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,509Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-sql]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,509Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-stack]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,509Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-voting-only-node]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,509Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "loaded module [x-pack-watcher]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,510Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "no plugins loaded" } {"type": "server", "timestamp": "2021-12-06T18:02:44,542Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "using [1] data paths, mounts [[/ (overlay)]], net usable_space [49gb], net total_space [58.4gb], types [overlay]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,542Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "heap size [1gb], compressed ordinary object pointers [true]" } {"type": "server", "timestamp": "2021-12-06T18:02:44,574Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "node name [4b5ef30c8b10], node ID [1M_JjTAoQPWi4fFbqbuB7g], cluster name [docker-cluster], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" } {"type": "server", "timestamp": "2021-12-06T18:02:47,735Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "[controller/202] [Main.cc@114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2020 Elasticsearch BV" } {"type": "server", "timestamp": "2021-12-06T18:02:48,326Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" } {"type": "server", "timestamp": "2021-12-06T18:02:48,974Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" } {"type": "server", "timestamp": "2021-12-06T18:02:49,027Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "using discovery type [single-node] and seed hosts providers [settings]" } {"type": "server", "timestamp": "2021-12-06T18:02:49,331Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" } {"type": "server", "timestamp": "2021-12-06T18:02:49,567Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "initialized" } {"type": "server", "timestamp": "2021-12-06T18:02:49,568Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "starting ..." } {"type": "server", "timestamp": "2021-12-06T18:02:49,636Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}" } {"type": "server", "timestamp": "2021-12-06T18:02:49,939Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "setting initial configuration to VotingConfiguration{1M_JjTAoQPWi4fFbqbuB7g}" } {"type": "server", "timestamp": "2021-12-06T18:02:50,055Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "elected-as-master ([1] nodes joined)[{4b5ef30c8b10}{1M_JjTAoQPWi4fFbqbuB7g}{Av7faOHTTOSgjNiChmuwJw}{172.17.0.2}{172.17.0.2:9300}{cdhilmrstw}{ml.machine_memory=8337530880, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{4b5ef30c8b10}{1M_JjTAoQPWi4fFbqbuB7g}{Av7faOHTTOSgjNiChmuwJw}{172.17.0.2}{172.17.0.2:9300}{cdhilmrstw}{ml.machine_memory=8337530880, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" } {"type": "server", "timestamp": "2021-12-06T18:02:50,077Z", "level": "INFO", "component": "o.e.c.c.CoordinationState", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "cluster UUID set to [opc7OGq-QK-B_nfvLbsItQ]" } {"type": "server", "timestamp": "2021-12-06T18:02:50,093Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "master node changed {previous [], current [{4b5ef30c8b10}{1M_JjTAoQPWi4fFbqbuB7g}{Av7faOHTTOSgjNiChmuwJw}{172.17.0.2}{172.17.0.2:9300}{cdhilmrstw}{ml.machine_memory=8337530880, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}" } {"type": "server", "timestamp": "2021-12-06T18:02:50,124Z", "level": "INFO", "component": "o.e.x.c.t.IndexTemplateRegistry", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding legacy template [.ml-anomalies-] for [ml], because it doesn't exist", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,128Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,129Z", "level": "INFO", "component": "o.e.x.c.t.IndexTemplateRegistry", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding legacy template [.ml-state] for [ml], because it doesn't exist", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,129Z", "level": "INFO", "component": "o.e.x.c.t.IndexTemplateRegistry", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding legacy template [.ml-config] for [ml], because it doesn't exist", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,131Z", "level": "INFO", "component": "o.e.x.c.t.IndexTemplateRegistry", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding legacy template [.ml-inference-000003] for [ml], because it doesn't exist", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,129Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "started", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,134Z", "level": "INFO", "component": "o.e.x.c.t.IndexTemplateRegistry", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding legacy template [.ml-meta] for [ml], because it doesn't exist", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,135Z", "level": "INFO", "component": "o.e.x.c.t.IndexTemplateRegistry", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding legacy template [.ml-notifications-000001] for [ml], because it doesn't exist", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,142Z", "level": "INFO", "component": "o.e.x.c.t.IndexTemplateRegistry", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding legacy template [.ml-stats] for [ml], because it doesn't exist", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,190Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "recovered [0] indices into cluster_state", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,373Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.ml-notifications-000001] for index patterns [.ml-notifications-000001]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,422Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.ml-config] for index patterns [.ml-config]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,457Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.ml-inference-000003] for index patterns [.ml-inference-000003]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,480Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.ml-state] for index patterns [.ml-state*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,507Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.ml-meta] for index patterns [.ml-meta]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,530Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.ml-stats] for index patterns [.ml-stats-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,569Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.ml-anomalies-] for index patterns [.ml-anomalies-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,592Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding component template [logs-mappings]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,613Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding component template [metrics-mappings]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,635Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding component template [logs-settings]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,655Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding component template [metrics-settings]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,677Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding component template [synthetics-mappings]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,698Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding component template [synthetics-settings]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,732Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [.triggered_watches] for index patterns [.triggered_watches*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,759Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [.watch-history-12] for index patterns [.watcher-history-12*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,784Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [.watches] for index patterns [.watches*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,808Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [ilm-history] for index patterns [ilm-history-3*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,829Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [.slm-history] for index patterns [.slm-history-3*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,855Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,894Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,927Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,952Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:50,987Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,018Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [logs] for index patterns [logs-*-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,044Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [metrics] for index patterns [metrics-*-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,070Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index template [synthetics] for index patterns [synthetics-*-*]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,094Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index lifecycle policy [ml-size-based-ilm-policy]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,123Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index lifecycle policy [logs]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,143Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index lifecycle policy [metrics]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,160Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index lifecycle policy [synthetics]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,181Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index lifecycle policy [watch-history-ilm-policy]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,201Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index lifecycle policy [ilm-history-ilm-policy]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,227Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "adding index lifecycle policy [slm-history-ilm-policy]", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,319Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "license [a22fe3e7-b316-45ff-9d3c-50adc2662c09] mode [basic] - valid", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" } {"type": "server", "timestamp": "2021-12-06T18:02:51,321Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "4b5ef30c8b10", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "opc7OGq-QK-B_nfvLbsItQ", "node.id": "1M_JjTAoQPWi4fFbqbuB7g" }
@dewey dewey added >bug needs:triage Requires assignment of a team area label labels Dec 6, 2021
@idegtiarenko idegtiarenko added the Team:Delivery Meta label for Delivery team label Dec 7, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-delivery (Team:Delivery)

@idegtiarenko idegtiarenko added the :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts label Dec 7, 2021
@pugnascotia
Copy link
Contributor

@dewey Is that all the log output? There's no stacktrace? What is the exit code of the containers? How much memory is your Docker VM configured to use?

These crashes don't replicate for me on Darwin 20.6.0 with Docker Desktop 3.6.0. If I can, I will try to replicate on the versions you report, but I would appreciate the answers to the above please.

@pugnascotia pugnascotia self-assigned this Dec 7, 2021
@pugnascotia pugnascotia removed the needs:triage Requires assignment of a team area label label Dec 7, 2021
@dewey
Copy link
Author

dewey commented Dec 7, 2021

Hey @pugnascotia, thanks for your reply!

That was the full output from docker logs -f, I think the container was in a crash-loop as after a while the logs repeated and in docker ps it showed it as being restarted. There was no stacktrace or OOM notices. I'm running Docker for Mac with 4 CPUs and 8GB of memory.

I just tried to replicate my examples again using the exact commands from the logs I posted and now I don't get any crashes. I haven't rebooted my computer or changed anything so that's a bit odd. I already restarted Docker for Mac yesterday a few times before filing the issues and that didn't change anything. I'll see if I can reproduce it again and get the exit code.

@pugnascotia
Copy link
Contributor

Hmm, it does sound like something in your environment, but let's see if it happens again and maybe we can dig into it.

@dewey
Copy link
Author

dewey commented Dec 7, 2021

I'll follow up if I can replicate it. It's a brand new machine with a clean install and almost nothing installed yet. This makes this a bit more mysterious. My best random guess right now is that maybe the resources got not applied correctly and it was actually just OOM'ing instead of crashing. So maybe more Docker for Mac related than ES. That would explain why the selection of images it works for is so random.

@pugnascotia
Copy link
Contributor

That's why I was asking about the exit code, 137 would suggest an OOM.

@droberts195
Copy link
Contributor

My best random guess right now is that maybe the resources got not applied correctly and it was actually just OOM'ing instead of crashing

If it turns out to be that then it will be related to #65905, which went into 7.11. That would explain why 7.10 works but 7.11 doesn't. And the problem would be that Java running inside Docker on macOS doesn't correctly determine how much memory is available to the Docker container (it does on Linux).

@pugnascotia
Copy link
Contributor

@droberts195 Docker Desktop deploys a Linux system to actually run the containers though, so the heap size calculations ought to work just fine.

@droberts195
Copy link
Contributor

ought to

Yes, but possibly there's some horrible combination of problems that means it doesn't.

A while ago we had #73381 which didn't mention Docker or macOS but just claimed Elasticsearch didn't run on Ubuntu on aarch64, which is not the case in general. But actually the way that user was experiencing the failure and the circumstantial evidence that they were running in Docker on Apple silicon means it could be the same thing as this, just OOMing at a slightly different place due to the exact amount of memory available.

Then I just did a bit of Googling and found this: https://bugs.openjdk.java.net/browse/JDK-8252359

That JDK bug is in versions that are consistent with the crashes reported here.

Anyway, the way to confirm is to try explicitly setting the ES JVM heap size to 1GB in 7.11.1 and 7.13.2. If that makes it work as well as 7.10.2 then it's pretty clear there's some combination of factors with macOS/Apple silicon/Docker/certain Java versions that confuses the auto-heap size code. Would you be able to try this @dewey?

@dewey
Copy link
Author

dewey commented Dec 7, 2021

I'd like to try that but unfortunately I can't reproduce the crash any more for now. Even setting the heap to 512MB like this seems to work now. If I experience it again I'll inspect the container for the exit code and try the heap size change but for now this looks like it "fixed" itself. When I opened the issue I could stop/rm/restart the container multiple times, adjusting memory settings in Docker for Mac and it would still crash reproducibly.

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" docker.elastic.co/elasticsearch/elasticsearch:7.11.1

@pugnascotia
Copy link
Contributor

I'm closing this issue since there's been no further reports of issues. We can reopen if more information becomes available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pugnascotia pugnascotia

Labels
>bug :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts Team:Delivery Meta label for Delivery team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dewey @idegtiarenko @droberts195 @pugnascotia @elasticmachine