Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrade to Elasticsearch v7.4 beta service #4505

Closed
23 tasks done
pkfec opened this issue Jul 27, 2020 · 6 comments · Fixed by #4657
Closed
23 tasks done

upgrade to Elasticsearch v7.4 beta service #4505

pkfec opened this issue Jul 27, 2020 · 6 comments · Fixed by #4657
Assignees
@pkfec @fec-jli
Milestone
Sprint 13.9

Comments

@pkfec
Copy link
Contributor

pkfec commented Jul 27, 2020 •
edited by fec-jli
Loading

The cloud.gov team is launching a beta of a new Elasticsearch service using AWS Elasticache. Upgrade API to the new beta service and test the legal services section of the website.

Note: Not backwards compatible to previous versions. Cannot load a 5.6 into a 7.4 version. This is a blocker for our migration.

Related issue #2921

Action items:

  • Create a new ES7.4 High Availability beta service on feature space.
  • Set up API app on feature space.
  • Setup circleci env variable and deploy api on feature space successfully.
  • Refactor all mappings to match the new ES7 service (docs, archived_mur indexes).
  • Add new ES client connection function using AWS4Auth and test the ES connection.
  • Upload all legal docs to feature space.(AO, ADR, AF, MUR, Archived MURs, Statues, Regulations)
  • Refactor existing index_management.py and move all the manage functions into the new file es_management.py.
  • Create new manage command (check_legal_doc) to view the legal data on the ES service.
  • Create a public S3 bucket service (fec-s3-api) to save legal pdf file.
  • Configure the public S3 bucket to list the object in a browser (cloud.gov put the listobject policy on this new bucket)
  • Create a new private s3 bucket service (fec-s3-snapshot) to save ES snapshot.(separated from public s3 bucket)
  • Config ES7 repository and test snapshots
  • Clean OCR text data on the arch-mur253(with 8 documents), which fixed the Message: Request size exceeded 10485760 bytes error while loading to ES7 service.
  • Refactor the legal/search and legal/docs api endpoints.
  • Integrate CMS and API and test the legal search.
  • Refactor all schedule task code.
  • Refactor and test downloads.
  • Refactor pytests.
  • Refactor and test snapshot and restore.
  • Refactor all management commands.
  • Conduct performance test (load test).

Completion criteria:

  • Cloud.gov is flagged about backwards compatibility problems
  • Identify the migration path from 5.6 to 7.4

Future work:

  • Our legal search is able to run on elasticsearch v7.4 without degraded performance
@pkfec pkfec added this to the Sprint 13.3 milestone Jul 27, 2020
@pkfec pkfec changed the title Upgrade API and test Elasticsearch v7.4 beta service Test Elasticsearch v7.4 beta service Jul 28, 2020
@pkfec
Copy link
Contributor Author

pkfec commented Jul 29, 2020

Here is the email we received from cloud.gov about the elasticsearch beta service plans:

The cloud.gov team is pleased to announce the launch of a beta of a new Elasticsearch service using AWS Elasticache. This work is an important step in moving cloud.gov towards a more stable and reliable platform by moving away from a bespoke Kubernetes instances to AWS-hosted services.

The intent of the Elasticsearch beta is to test out the service with a limited number of customers before making it available to all cloud.gov customers.

We ask that interested customers email [email protected] to request access to the beta and specify the org(s) in which they would like to enable the beta service.

We’ve outlined below a few important things to keep in mind before and while you are using the beta.

The beta should only be used in non-production environments.
For this beta, the cloud.gov operators will be limiting access to this new broker by specific organizations only. All spaces within an organization will have access to the beta.
While we took every measure to test the broker before releasing it to beta, testing for performance is one area that is difficult to assess pre-release. Therefore, we will be closely monitoring performance once in beta, but we expect some performance work may be required early in the beta.
The cloud.gov team will use the weekly office hours to solicit feedback from you about the beta.
The cloud.gov team reserves the right to update and change the broker and plan details based on testing and feedback results.
Elasticsearch instances created in beta testing will not be carried over into production once the service goes General Availability (GA). Once GA is announced, the beta plans will be disabled from creating new instances and deleted a short time later.
Please submit any feedback or support requests about the Redis broker beta to [email protected].
If you would like to participate in the beta, please contact [email protected] to request access to the beta and specify the org(s) in which you would like to enable the beta service.

Technical Notes for the beta release:

The service name will simply be called aws-elasticsearch
The elasticsearch offering will come in three plans:
BETA-es-dev
Single data only node
Non-prod use only
BETA-es-medium
Three master nodes
Two data nodes
BETA-es-medium-ha
Three master nodes
Four data nodes
All plans for Elasticsearch include the following:
Elasticsearch version 7.4
The system will automatically take snapshots hourly and will retain 336 of snapshots for up to 14 days. NOTE: these snapshots can only be used for a cluster restore only. Customers will need to perform manual snapshots if they want to backup their data for uses other than cluster restore.
Data nodes will have 10G of usable disk
This new offering will require customers to use AWS specific HTTP signing requests. Please see the following for more details: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html
Please note that the startup of Elasticsearch on AWS may be delayed. We’ve seen AWS take 5 mins per node and up to a couple hours for 2 nodes.
If customers have successful code changes in various languages that they would like to share examples of with other cloud.gov customers, we encourage PRs to our example repo: https://github.com/cloud-gov/aws-elasticsearch-example

--
Kara Reinsel
Product Manager - cloud.gov

@pkfec
Copy link
Contributor Author

pkfec commented Aug 18, 2020 •
edited
Loading

Test Elasticsearch 7.4 service on local env:

  1. Download Elasticsearch 7.4 from here:

  2. Tried to start the ES74 server and ran into OS related errors:
    Error stack trace on the terminal :

Caused by: org.elasticsearch.ElasticsearchException: Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory.

Error stack trace:
[2020-08-11T21:49:30,518][INFO ][o.e.x.s.a.s.FileRolesStore] [macadmins-mbp-5.lan] parsed [0] roles from file [/Users/pkasireddy/Projects/elasticsearch-7.4.0/config/roles.yml]
[2020-08-11T21:49:42,694][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [macadmins-mbp-5.lan] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: **false].]**
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]
Caused by: org.elasticsearch.ElasticsearchException: Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].
	at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:524) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$9(Node.java:450) ~[elasticsearch-7.4.0.jar:7.4.0]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_171]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_171]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_171]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_171]
	at org.elasticsearch.node.Node.<init>(Node.java:453) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]
	... 6 more
  1. Check my current Java version:
macadmins-mbp-5:~ pkasireddy$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
  1. The latest elasticsearch 7.* and JDK 1.8 are incompatible. To start elasticsearch v7.* JDK 11 or higher is required.
    Ran brew update and brew upgrade commands which installed the JDK14.
==> Upgrading openjdk 13.0.2+8_2 -> 14.0.1
==> Downloading https://homebrew.bintray.com/bottles/openjdk-14.0.1.catalina.bot
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/d44db8c5b212a36d73f11
######################################################################## 100.0%
==> Pouring openjdk-14.0.1.catalina.bottle.tar.gz
==> Caveats
For the system Java wrappers to find this JDK, symlink it with
  sudo ln -sfn /usr/local/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

openjdk is keg-only, which means it was not symlinked into /usr/local,
because it shadows the macOS `java` wrapper.

If you need to have openjdk first in your PATH run:
  echo 'export PATH="/usr/local/opt/openjdk/bin:$PATH"' >> /Users/pkasireddy/.bash_profile

For compilers to find openjdk you may need to set:
  export CPPFLAGS="-I/usr/local/opt/openjdk/include"
  1. Add these two variables to your bash_profile:
If you need to have openjdk first in your PATH run:
  echo 'export PATH="/usr/local/opt/openjdk/bin:$PATH"' >> /Users/pkasireddy/.bash_profile

For compilers to find openjdk you may need to set:
  export CPPFLAGS="-I/usr/local/opt/openjdk/include"

  1. Noticed OS incompatible errors in elasticsearch server 7.4 logs when the server starts.
    Add below line to elasticsearch.yml :
    xpack.ml.enabled: false
    See documentation - https://www.elastic.co/guide/en/elasticsearch/reference/master/ml-settings.html#ml-settings

  2. Create a new python virtualenv, activate it and install requirements:

  3. Updated elasticsearch pkg and installed the requirements again:
    elasticsearch==7.6.0
    elasticsearch-dsl==7.2.1

  4. Repeat step (6) for elasticsearch service 7.6

  5. start the elastic search76 service on local env:
    macadmins-mbp-5:bin pkasireddy$ ./elasticsearchmacadmins-mbp-5:bin pkasireddy$ ./elasticsearch

[2020-08-17T11:32:12,749][INFO ][o.e.e.NodeEnvironment    ] [macadmins-mbp-5.lan] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s1)]], net usable_space [357.1gb], net total_space [465.6gb], types [apfs]
[2020-08-17T11:32:12,758][INFO ][o.e.e.NodeEnvironment    ] [macadmins-mbp-5.lan] heap size [1gb], compressed ordinary object pointers [true]
[2020-08-17T11:32:12,971][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] node name [macadmins-mbp-5.lan], node ID [3RoktXyNTFKukIk0uOfDVw], cluster name [elasticsearch]
[2020-08-17T11:32:12,972][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] version[7.6.0], pid[51884], build[default/tar/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-06T00:09:00.449973Z], OS[Mac OS X/10.15.6/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/14.0.2/14.0.2+12-46]
[2020-08-17T11:32:12,972][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] JVM home [/Library/Java/JavaVirtualMachines/openjdk-14.0.2.jdk/Contents/Home]
[2020-08-17T11:32:12,973][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/folders/74/d360dq8x0b7c83df2nfhtpbs980dz9/T/elasticsearch-8627524160781265597, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/pkasireddy/Projects/elasticsearch-7.6.0, -Des.path.conf=/Users/pkasireddy/Projects/elasticsearch-7.6.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-08-17T11:32:15,445][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [aggs-matrix-stats]
[2020-08-17T11:32:15,447][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [analysis-common]
[2020-08-17T11:32:15,448][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [flattened]
[2020-08-17T11:32:15,448][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [frozen-indices]
[2020-08-17T11:32:15,449][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-common]
[2020-08-17T11:32:15,450][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-geoip]
[2020-08-17T11:32:15,451][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-user-agent]
[2020-08-17T11:32:15,451][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-expression]
[2020-08-17T11:32:15,452][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-mustache]
[2020-08-17T11:32:15,453][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-painless]
[2020-08-17T11:32:15,453][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [mapper-extras]
[2020-08-17T11:32:15,454][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [parent-join]
[2020-08-17T11:32:15,454][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [percolator]
[2020-08-17T11:32:15,455][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [rank-eval]
[2020-08-17T11:32:15,455][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [reindex]
[2020-08-17T11:32:15,456][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [repository-url]
[2020-08-17T11:32:15,456][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [search-business-rules]
[2020-08-17T11:32:15,456][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [spatial]
[2020-08-17T11:32:15,457][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [transform]
[2020-08-17T11:32:15,457][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [transport-netty4]
[2020-08-17T11:32:15,458][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [vectors]
[2020-08-17T11:32:15,458][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-analytics]
[2020-08-17T11:32:15,462][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ccr]
[2020-08-17T11:32:15,467][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-core]
[2020-08-17T11:32:15,468][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-deprecation]
[2020-08-17T11:32:15,468][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-enrich]
[2020-08-17T11:32:15,469][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-graph]
[2020-08-17T11:32:15,469][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ilm]
[2020-08-17T11:32:15,470][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-logstash]
[2020-08-17T11:32:15,470][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ml]
[2020-08-17T11:32:15,470][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-monitoring]
[2020-08-17T11:32:15,474][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-rollup]
[2020-08-17T11:32:15,474][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-security]
[2020-08-17T11:32:15,475][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-sql]
[2020-08-17T11:32:15,475][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-voting-only-node]
[2020-08-17T11:32:15,476][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-watcher]
[2020-08-17T11:32:15,478][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] no plugins loaded
[2020-08-17T11:32:20,644][INFO ][o.e.x.s.a.s.FileRolesStore] [macadmins-mbp-5.lan] parsed [0] roles from file [/Users/pkasireddy/Projects/elasticsearch-7.6.0/config/roles.yml]
[2020-08-17T11:32:21,814][DEBUG][o.e.a.ActionModule       ] [macadmins-mbp-5.lan] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-08-17T11:32:21,972][INFO ][o.e.d.DiscoveryModule    ] [macadmins-mbp-5.lan] using discovery type [zen] and seed hosts providers [settings]
[2020-08-17T11:32:22,944][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] initialized
[2020-08-17T11:32:22,945][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] starting ...
[2020-08-17T11:32:23,155][INFO ][o.e.t.TransportService   ] [macadmins-mbp-5.lan] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-08-17T11:32:23,653][WARN ][o.e.b.BootstrapChecks    ] [macadmins-mbp-5.lan] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-08-17T11:32:23,655][INFO ][o.e.c.c.Coordinator      ] [macadmins-mbp-5.lan] cluster UUID [xZcDd9BURe6rrELNXbtoOw]
[2020-08-17T11:32:23,669][INFO ][o.e.c.c.ClusterBootstrapService] [macadmins-mbp-5.lan] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-08-17T11:32:23,826][INFO ][o.e.c.s.MasterService    ] [macadmins-mbp-5.lan] elected-as-master ([1] nodes joined)[{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{Cbb3iZkKTimiFuA_Aui50g}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 29, delta: master node changed {previous [], current [{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{Cbb3iZkKTimiFuA_Aui50g}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}
[2020-08-17T11:32:23,925][INFO ][o.e.c.s.ClusterApplierService] [macadmins-mbp-5.lan] master node changed {previous [], current [{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{Cbb3iZkKTimiFuA_Aui50g}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}, term: 4, version: 29, reason: Publication{term=4, version=29}
[2020-08-17T11:32:23,970][INFO ][o.e.h.AbstractHttpServerTransport] [macadmins-mbp-5.lan] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-08-17T11:32:23,971][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] started
[2020-08-17T11:32:24,247][INFO ][o.e.l.LicenseService     ] [macadmins-mbp-5.lan] license [4cc6db56-a089-4d47-909a-075c08860a84] mode [basic] - valid
[2020-08-17T11:32:24,248][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [macadmins-mbp-5.lan] Active license is now [BASIC]; Security is disabled
[2020-08-17T11:32:24,259][INFO ][o.e.g.GatewayService     ] [macadmins-mbp-5.lan] recovered [0] indices into cluster_state
^C[2020-08-18T09:42:37,292][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] stopping ...
[2020-08-18T09:42:37,429][INFO ][o.e.x.w.WatcherService   ] [macadmins-mbp-5.lan] stopping watch service, reason [shutdown initiated]
[2020-08-18T09:42:37,440][INFO ][o.e.x.w.WatcherLifeCycleService] [macadmins-mbp-5.lan] watcher has stopped and shutdown
[2020-08-18T09:42:37,929][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] stopped
[2020-08-18T09:42:37,931][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] closing ...
[2020-08-18T09:42:37,972][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] closed
macadmins-mbp-5:bin pkasireddy$ git branch
fatal: not a git repository (or any of the parent directories): .git
macadmins-mbp-5:bin pkasireddy$ ./elasticsearch
[2020-08-18T10:32:41,801][INFO ][o.e.e.NodeEnvironment    ] [macadmins-mbp-5.lan] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s1)]], net usable_space [355.5gb], net total_space [465.6gb], types [apfs]
[2020-08-18T10:32:41,809][INFO ][o.e.e.NodeEnvironment    ] [macadmins-mbp-5.lan] heap size [1gb], compressed ordinary object pointers [true]
[2020-08-18T10:32:42,017][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] node name [macadmins-mbp-5.lan], node ID [3RoktXyNTFKukIk0uOfDVw], cluster name [elasticsearch]
[2020-08-18T10:32:42,018][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] version[7.6.0], pid[64532], build[default/tar/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-06T00:09:00.449973Z], OS[Mac OS X/10.15.6/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/14.0.2/14.0.2+12-46]
[2020-08-18T10:32:42,020][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] JVM home [/Library/Java/JavaVirtualMachines/openjdk-14.0.2.jdk/Contents/Home]
[2020-08-18T10:32:42,021][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/folders/74/d360dq8x0b7c83df2nfhtpbs980dz9/T/elasticsearch-3346049010720047220, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/pkasireddy/Projects/elasticsearch-7.6.0, -Des.path.conf=/Users/pkasireddy/Projects/elasticsearch-7.6.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-08-18T10:32:44,573][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [aggs-matrix-stats]
[2020-08-18T10:32:44,574][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [analysis-common]
[2020-08-18T10:32:44,574][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [flattened]
[2020-08-18T10:32:44,574][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [frozen-indices]
[2020-08-18T10:32:44,575][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-common]
[2020-08-18T10:32:44,575][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-geoip]
[2020-08-18T10:32:44,576][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-user-agent]
[2020-08-18T10:32:44,576][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-expression]
[2020-08-18T10:32:44,576][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-mustache]
[2020-08-18T10:32:44,577][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-painless]
[2020-08-18T10:32:44,579][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [mapper-extras]
[2020-08-18T10:32:44,580][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [parent-join]
[2020-08-18T10:32:44,580][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [percolator]
[2020-08-18T10:32:44,580][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [rank-eval]
[2020-08-18T10:32:44,581][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [reindex]
[2020-08-18T10:32:44,581][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [repository-url]
[2020-08-18T10:32:44,581][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [search-business-rules]
[2020-08-18T10:32:44,582][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [spatial]
[2020-08-18T10:32:44,582][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [transform]
[2020-08-18T10:32:44,582][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [transport-netty4]
[2020-08-18T10:32:44,583][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [vectors]
[2020-08-18T10:32:44,583][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-analytics]
[2020-08-18T10:32:44,583][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ccr]
[2020-08-18T10:32:44,584][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-core]
[2020-08-18T10:32:44,584][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-deprecation]
[2020-08-18T10:32:44,584][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-enrich]
[2020-08-18T10:32:44,584][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-graph]
[2020-08-18T10:32:44,585][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ilm]
[2020-08-18T10:32:44,585][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-logstash]
[2020-08-18T10:32:44,585][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ml]
[2020-08-18T10:32:44,586][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-monitoring]
[2020-08-18T10:32:44,586][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-rollup]
[2020-08-18T10:32:44,587][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-security]
[2020-08-18T10:32:44,587][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-sql]
[2020-08-18T10:32:44,587][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-voting-only-node]
[2020-08-18T10:32:44,587][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-watcher]
[2020-08-18T10:32:44,588][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] no plugins loaded
[2020-08-18T10:32:51,056][INFO ][o.e.x.s.a.s.FileRolesStore] [macadmins-mbp-5.lan] parsed [0] roles from file [/Users/pkasireddy/Projects/elasticsearch-7.6.0/config/roles.yml]
[2020-08-18T10:32:52,406][DEBUG][o.e.a.ActionModule       ] [macadmins-mbp-5.lan] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-08-18T10:32:52,575][INFO ][o.e.d.DiscoveryModule    ] [macadmins-mbp-5.lan] using discovery type [zen] and seed hosts providers [settings]
[2020-08-18T10:32:53,583][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] initialized
[2020-08-18T10:32:53,583][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] starting ...
[2020-08-18T10:32:53,806][INFO ][o.e.t.TransportService   ] [macadmins-mbp-5.lan] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-08-18T10:32:54,345][WARN ][o.e.b.BootstrapChecks    ] [macadmins-mbp-5.lan] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-08-18T10:32:54,347][INFO ][o.e.c.c.Coordinator      ] [macadmins-mbp-5.lan] cluster UUID [xZcDd9BURe6rrELNXbtoOw]
[2020-08-18T10:32:54,367][INFO ][o.e.c.c.ClusterBootstrapService] [macadmins-mbp-5.lan] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-08-18T10:32:54,522][INFO ][o.e.c.s.MasterService    ] [macadmins-mbp-5.lan] elected-as-master ([1] nodes joined)[{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{AXO8YTBuQmWGeTKpYMmpOw}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 5, version: 32, delta: master node changed {previous [], current [{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{AXO8YTBuQmWGeTKpYMmpOw}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}
[2020-08-18T10:32:54,615][INFO ][o.e.c.s.ClusterApplierService] [macadmins-mbp-5.lan] master node changed {previous [], current [{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{AXO8YTBuQmWGeTKpYMmpOw}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}, term: 5, version: 32, reason: Publication{term=5, version=32}
[2020-08-18T10:32:54,663][INFO ][o.e.h.AbstractHttpServerTransport] [macadmins-mbp-5.lan] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-08-18T10:32:54,664][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] started
[2020-08-18T10:32:54,935][INFO ][o.e.l.LicenseService     ] [macadmins-mbp-5.lan] license [4cc6db56-a089-4d47-909a-075c08860a84] mode [basic] - valid
[2020-08-18T10:32:54,937][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [macadmins-mbp-5.lan] Active license is now [BASIC]; Security is disabled
[2020-08-18T10:32:54,948][INFO ][o.e.g.GatewayService     ] [macadmins-mbp-5.lan] recovered [0] indices into cluster_state
^C[2020-08-18T13:49:03,713][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] stopping ...
[2020-08-18T13:49:03,730][INFO ][o.e.x.w.WatcherService   ] [macadmins-mbp-5.lan] stopping watch service, reason [shutdown initiated]
[2020-08-18T13:49:03,731][INFO ][o.e.x.w.WatcherLifeCycleService] [macadmins-mbp-5.lan] watcher has stopped and shutdown
^C^C[2020-08-18T13:49:04,172][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] stopped
[2020-08-18T13:49:04,172][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] closing ...
[2020-08-18T13:49:04,189][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] closed
macadmins-mbp-5:bin pkasireddy$ clear
macadmins-mbp-5:bin pkasireddy$ ./elasticsearch
[2020-08-18T13:49:12,456][INFO ][o.e.e.NodeEnvironment    ] [macadmins-mbp-5.lan] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s1)]], net usable_space [355.4gb], net total_space [465.6gb], types [apfs]
[2020-08-18T13:49:12,461][INFO ][o.e.e.NodeEnvironment    ] [macadmins-mbp-5.lan] heap size [1gb], compressed ordinary object pointers [true]
[2020-08-18T13:49:12,623][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] node name [macadmins-mbp-5.lan], node ID [3RoktXyNTFKukIk0uOfDVw], cluster name [elasticsearch]
[2020-08-18T13:49:12,624][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] version[7.6.0], pid[69493], build[default/tar/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-06T00:09:00.449973Z], OS[Mac OS X/10.15.6/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/14.0.2/14.0.2+12-46]
[2020-08-18T13:49:12,624][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] JVM home [/Library/Java/JavaVirtualMachines/openjdk-14.0.2.jdk/Contents/Home]
[2020-08-18T13:49:12,625][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/folders/74/d360dq8x0b7c83df2nfhtpbs980dz9/T/elasticsearch-7992181741539089573, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/pkasireddy/Projects/elasticsearch-7.6.0, -Des.path.conf=/Users/pkasireddy/Projects/elasticsearch-7.6.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-08-18T13:49:14,545][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [aggs-matrix-stats]
[2020-08-18T13:49:14,546][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [analysis-common]
[2020-08-18T13:49:14,546][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [flattened]
[2020-08-18T13:49:14,547][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [frozen-indices]
[2020-08-18T13:49:14,547][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-common]
[2020-08-18T13:49:14,547][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-geoip]
[2020-08-18T13:49:14,548][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [ingest-user-agent]
[2020-08-18T13:49:14,548][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-expression]
[2020-08-18T13:49:14,549][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-mustache]
[2020-08-18T13:49:14,549][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [lang-painless]
[2020-08-18T13:49:14,549][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [mapper-extras]
[2020-08-18T13:49:14,550][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [parent-join]
[2020-08-18T13:49:14,551][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [percolator]
[2020-08-18T13:49:14,551][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [rank-eval]
[2020-08-18T13:49:14,551][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [reindex]
[2020-08-18T13:49:14,551][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [repository-url]
[2020-08-18T13:49:14,552][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [search-business-rules]
[2020-08-18T13:49:14,552][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [spatial]
[2020-08-18T13:49:14,553][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [transform]
[2020-08-18T13:49:14,553][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [transport-netty4]
[2020-08-18T13:49:14,553][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [vectors]
[2020-08-18T13:49:14,554][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-analytics]
[2020-08-18T13:49:14,554][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ccr]
[2020-08-18T13:49:14,555][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-core]
[2020-08-18T13:49:14,555][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-deprecation]
[2020-08-18T13:49:14,556][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-enrich]
[2020-08-18T13:49:14,556][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-graph]
[2020-08-18T13:49:14,556][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ilm]
[2020-08-18T13:49:14,556][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-logstash]
[2020-08-18T13:49:14,557][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-ml]
[2020-08-18T13:49:14,557][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-monitoring]
[2020-08-18T13:49:14,557][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-rollup]
[2020-08-18T13:49:14,557][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-security]
[2020-08-18T13:49:14,558][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-sql]
[2020-08-18T13:49:14,558][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-voting-only-node]
[2020-08-18T13:49:14,558][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] loaded module [x-pack-watcher]
[2020-08-18T13:49:14,559][INFO ][o.e.p.PluginsService     ] [macadmins-mbp-5.lan] no plugins loaded
[2020-08-18T13:49:19,059][INFO ][o.e.x.s.a.s.FileRolesStore] [macadmins-mbp-5.lan] parsed [0] roles from file [/Users/pkasireddy/Projects/elasticsearch-7.6.0/config/roles.yml]
[2020-08-18T13:49:20,016][DEBUG][o.e.a.ActionModule       ] [macadmins-mbp-5.lan] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-08-18T13:49:20,220][INFO ][o.e.d.DiscoveryModule    ] [macadmins-mbp-5.lan] using discovery type [zen] and seed hosts providers [settings]
[2020-08-18T13:49:20,948][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] initialized
[2020-08-18T13:49:20,948][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] starting ...
[2020-08-18T13:49:21,151][INFO ][o.e.t.TransportService   ] [macadmins-mbp-5.lan] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-08-18T13:49:21,611][WARN ][o.e.b.BootstrapChecks    ] [macadmins-mbp-5.lan] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-08-18T13:49:21,612][INFO ][o.e.c.c.Coordinator      ] [macadmins-mbp-5.lan] cluster UUID [xZcDd9BURe6rrELNXbtoOw]
[2020-08-18T13:49:21,628][INFO ][o.e.c.c.ClusterBootstrapService] [macadmins-mbp-5.lan] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-08-18T13:49:21,791][INFO ][o.e.c.s.MasterService    ] [macadmins-mbp-5.lan] elected-as-master ([1] nodes joined)[{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{VDjnoSJQSlieMQZVWra7fQ}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 6, version: 35, delta: master node changed {previous [], current [{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{VDjnoSJQSlieMQZVWra7fQ}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}
[2020-08-18T13:49:21,881][INFO ][o.e.c.s.ClusterApplierService] [macadmins-mbp-5.lan] master node changed {previous [], current [{macadmins-mbp-5.lan}{3RoktXyNTFKukIk0uOfDVw}{VDjnoSJQSlieMQZVWra7fQ}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}, term: 6, version: 35, reason: Publication{term=6, version=35}
[2020-08-18T13:49:21,951][INFO ][o.e.h.AbstractHttpServerTransport] [macadmins-mbp-5.lan] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-08-18T13:49:21,953][INFO ][o.e.n.Node               ] [macadmins-mbp-5.lan] started
[2020-08-18T13:49:22,167][INFO ][o.e.l.LicenseService     ] [macadmins-mbp-5.lan] license [4cc6db56-a089-4d47-909a-075c08860a84] mode [basic] - valid
[2020-08-18T13:49:22,169][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [macadmins-mbp-5.lan] Active license is now [BASIC]; Security is disabled
[2020-08-18T13:49:22,178][INFO ][o.e.g.GatewayService     ] [macadmins-mbp-5.lan] recovered [0] indices into cluster_state

11 . In a browser, verify the elasticsearch service76 is UP and running on your local:

http://localhost:9200/
Screen Shot 2020-08-18 at 1 52 27 PM

  1. Attempted to run the management task create_docs_index on local elasticsearch7.6 service

(api377) macadmins-mbp-5:openFEC pkasireddy$ ./manage.py create_docs_index

INFO:webservices.legal_docs.index_management:Delete index 'docs'
WARNING:elasticsearch:DELETE http://localhost:9200/docs [status:404 request:0.008s]
INFO:webservices.legal_docs.index_management:Delete index 'docs_index'
WARNING:elasticsearch:DELETE http://localhost:9200/docs_index [status:404 request:0.002s]
INFO:webservices.legal_docs.index_management:Create index 'docs'
WARNING:elasticsearch:PUT http://localhost:9200/docs [status:400 request:0.208s]
Traceback (most recent call last):
  File "./manage.py", line 195, in <module>
    manager.run()
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/flask_script/__init__.py", line 417, in run
    result = self.handle(argv[0], argv[1:])
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/flask_script/__init__.py", line 386, in handle
    res = handle(*args, **config)
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/Users/pkasireddy/Projects/api/openFEC/webservices/legal_docs/index_management.py", line 225, in create_docs_index
    "aliases": {'docs_index': {}, 'docs_search': {}},
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 104, in create
    "PUT", _make_path(index), params=params, headers=headers, body=body
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/transport.py", line 362, in perform_request
    timeout=timeout,
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 248, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/connection/base.py", line 244, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters:  [_default_ : {properties={sort1={include_in_all=false, type=integer}, sort2={include_in_all=false, type=integer}}}] [citations : {properties={citation_text={index=not_analyzed, type=string}, citation_type={index=not_analyzed, type=string}}}] [admin_fines : {properties={no={index=not_analyzed, type=string}, petition_court_decision_date={format=dateOptionalTime, type=date}, treasury_referral_amount={index=no, type=long}, documents={type=nested, properties={document_date={format=dateOptionalTime, type=date}, length={index=no, type=long}, description={type=string}, text={type=string}, category={index=not_analyzed, type=string}, document_id={index=no, type=long}, url={index=no, type=string}}}, challenge_outcome={index=no, type=string}, report_type={index=no, type=string}, reason_to_believe_action_date={format=dateOptionalTime, type=date}, doc_id={index=no, type=string}, url={index=no, type=string}, treasury_referral_date={format=dateOptionalTime, type=date}, challenge_receipt_date={format=dateOptionalTime, type=date}, final_determination_date={format=dateOptionalTime, type=date}, name={analyzer=english, type=string}, report_year={type=string}, commission_votes={properties={vote_date={format=dateOptionalTime, type=date}, text={type=string}}}, final_determination_amount={type=long}, check_amount={index=no, type=long}, petition_court_filing_date={format=dateOptionalTime, type=date}, committee_id={index=not_analyzed, type=string}, reason_to_believe_fine_amount={type=long}}}] [advisory_opinions : {properties={summary={analyzer=english, type=string}, no={index=not_analyzed, type=string}, ao_citations={properties={no={type=string}, name={type=string}}}, requestor_names={type=string}, documents={type=nested, properties={date={format=dateOptionalTime, type=date}, description={type=string}, text={type=string}, document_id={index=no, type=long}, category={index=not_analyzed, type=string}, url={index=no, type=string}}}, regulatory_citations={type=nested, properties={part={type=long}, section={type=long}, title={type=long}}}, aos_cited_by={properties={no={type=string}, name={type=string}}}, issue_date={format=dateOptionalTime, type=date}, statutory_citations={type=nested, properties={section={type=string}, title={type=long}}}, name={analyzer=english, type=string}, requestor_types={index=not_analyzed, type=string}, is_pending={type=boolean}, status={type=string}}}] [regulations : {properties={no={index=not_analyzed, type=string}, name={analyzer=english, type=string}, text={analyzer=english, type=string}, doc_id={index=no, type=string}, url={index=no, type=string}}}] [statutes : {properties={no={index=not_analyzed, type=string}, chapter={type=string}, name={analyzer=english, type=string}, subchapter={type=string}, text={analyzer=english, type=string}, title={type=string}, doc_id={index=no, type=string}, url={index=no, type=string}}}] [adrs : {properties={open_date={format=dateOptionalTime, type=date}, no={index=not_analyzed, type=string}, close_date={format=dateOptionalTime, type=date}, documents={type=nested, properties={document_date={format=dateOptionalTime, type=date}, length={index=no, type=long}, description={type=string}, text={type=string}, category={index=not_analyzed, type=string}, document_id={index=no, type=long}, url={index=no, type=string}}}, subjects={type=string}, dispositions={properties={disposition={index=not_analyzed, type=string}, citations={properties={text={type=string}, title={type=string}, type={type=string}, url={type=string}}}, penalty={type=double}, respondent={type=string}}}, doc_id={index=no, type=string}, url={index=no, type=string}, respondents={type=string}, name={analyzer=english, type=string}, election_cycles={type=long}, commission_votes={properties={vote_date={format=dateOptionalTime, type=date}, text={type=string}}}, participants={properties={citations={type=object}, role={type=string}, name={type=string}}}}}] [murs : {properties={open_date={format=dateOptionalTime, type=date}, no={index=not_analyzed, type=string}, close_date={format=dateOptionalTime, type=date}, documents={type=nested, properties={document_date={format=dateOptionalTime, type=date}, length={index=no, type=long}, description={type=string}, text={type=string}, category={index=not_analyzed, type=string}, document_id={index=no, type=long}, url={index=no, type=string}}}, subjects={type=string}, dispositions={properties={disposition={index=not_analyzed, type=string}, citations={properties={text={type=string}, title={type=string}, type={type=string}, url={type=string}}}, penalty={type=double}, respondent={type=string}}}, doc_id={index=no, type=string}, url={index=no, type=string}, respondents={type=string}, name={analyzer=english, type=string}, election_cycles={type=long}, commission_votes={properties={vote_date={format=dateOptionalTime, type=date}, text={type=string}}}, mur_type={type=string}, participants={properties={citations={type=object}, role={type=string}, name={type=string}}}}}]')

  1. Removed the default tag from MAPPINGS as its deprecated in ES7.* service:
    See Elasticsearch 7.x - https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_schedule_for_removal_of_mapping_types

  2. Datatype string is not supported in 7.x* service. Replaced all string type to text in the source code here:
    https://github.com/fecgov/openFEC/blob/develop/webservices/legal_docs/index_management.py#L89
    See https://www.elastic.co/guide/en/elasticsearch/reference/7.x/sql-data-types.html

  3. Still getting RequestError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters:

(api377) macadmins-MacBook-Pro-5:openFEC pkasireddy$ ./manage.py create_docs_index

WARNING:elasticsearch:DELETE http://localhost:9200/docs [status:404 request:0.004s]
INFO:webservices.legal_docs.index_management:Delete index 'docs_index'
WARNING:elasticsearch:DELETE http://localhost:9200/docs_index [status:404 request:0.002s]
INFO:webservices.legal_docs.index_management:Before creating 'docs' index.
WARNING:elasticsearch:PUT http://localhost:9200/docs [status:400 request:0.012s]
Traceback (most recent call last):
  File "./manage.py", line 195, in <module>
    manager.run()
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/flask_script/__init__.py", line 417, in run
    result = self.handle(argv[0], argv[1:])
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/flask_script/__init__.py", line 386, in handle
    res = handle(*args, **config)
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/Users/pkasireddy/Projects/api/openFEC/webservices/legal_docs/index_management.py", line 232, in create_docs_index
    "aliases": {'docs_index': {}, 'docs_search': {}},
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 104, in create
    "PUT", _make_path(index), params=params, headers=headers, body=body
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/transport.py", line 362, in perform_request
    timeout=timeout,
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 248, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/pkasireddy/.pyenv/versions/api377/lib/python3.7/site-packages/elasticsearch/connection/base.py", line 244, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters:  [citations : {_doc={properties={citation_text={type=keyword}, citation_type={type=keyword}}}}] [admin_fines : {properties={no={index=true, type=text}, petition_court_decision_date={format=dateOptionalTime, type=date}, treasury_referral_amount={index=false, type=long}, documents={type=nested, properties={document_date={format=dateOptionalTime, type=date}, length={index=false, type=long}, description={type=text}, text={type=text}, category={index=true, type=keyword}, document_id={index=false, type=long}, url={index=false, type=text}}}, challenge_outcome={index=false, type=text}, report_type={index=false, type=text}, reason_to_believe_action_date={format=dateOptionalTime, type=date}, doc_id={index=false, type=text}, url={index=false, type=text}, treasury_referral_date={format=dateOptionalTime, type=date}, challenge_receipt_date={format=dateOptionalTime, type=date}, final_determination_date={format=dateOptionalTime, type=date}, name={analyzer=english, type=text}, report_year={type=text}, commission_votes={properties={vote_date={format=dateOptionalTime, type=date}, text={type=text}}}, final_determination_amount={type=long}, check_amount={index=false, type=long}, petition_court_filing_date={format=dateOptionalTime, type=date}, committee_id={index=true, type=keyword}, reason_to_believe_fine_amount={type=long}}}] [advisory_opinions : {properties={summary={analyzer=english, type=text}, no={index=true, type=keyword}, ao_citations={properties={no={type=text}, name={type=text}}}, requestor_names={type=text}, documents={type=nested, properties={date={format=dateOptionalTime, type=date}, description={type=text}, text={type=text}, document_id={index=false, type=long}, category={index=true, type=keyword}, url={index=false, type=text}}}, regulatory_citations={type=nested, properties={part={type=long}, section={type=long}, title={type=long}}}, aos_cited_by={properties={no={type=text}, name={type=text}}}, issue_date={format=dateOptionalTime, type=date}, statutory_citations={type=nested, properties={section={type=text}, title={type=long}}}, name={analyzer=english, type=text}, requestor_types={index=true, type=keyword}, is_pending={type=boolean}, status={type=text}}}] [regulations : {properties={no={index=true, type=keyword}, name={analyzer=english, type=text}, text={analyzer=english, type=text}, doc_id={index=false, type=text}, url={index=false, type=text}}}] [statutes : {properties={no={index=true, type=keyword}, chapter={type=text}, name={analyzer=english, type=text}, subchapter={type=text}, text={analyzer=english, type=text}, title={type=text}, doc_id={index=false, type=text}, url={index=false, type=text}}}] [adrs : {properties={open_date={format=dateOptionalTime, type=date}, no={index=true, type=keyword}, close_date={format=dateOptionalTime, type=date}, documents={type=nested, properties={document_date={format=dateOptionalTime, type=date}, length={index=false, type=long}, description={type=text}, text={type=text}, category={index=true, type=keyword}, document_id={index=false, type=long}, url={index=false, type=text}}}, subjects={type=text}, dispositions={properties={disposition={index=true, type=keyword}, citations={properties={text={type=text}, title={type=text}, type={type=text}, url={type=text}}}, penalty={type=double}, respondent={type=text}}}, doc_id={index=true, type=keyword}, url={index=false, type=text}, respondents={type=text}, name={analyzer=english, type=text}, election_cycles={type=long}, commission_votes={properties={vote_date={format=dateOptionalTime, type=date}, text={type=text}}}, participants={properties={citations={type=object}, role={type=text}, name={type=text}}}}}] [murs : {properties={open_date={format=dateOptionalTime, type=date}, no={index=true, type=keyword}, close_date={format=dateOptionalTime, type=date}, documents={type=nested, properties={document_date={format=dateOptionalTime, type=date}, length={index=false, type=long}, description={type=text}, text={type=text}, category={index=true, type=keyword}, document_id={index=false, type=long}, url={index=false, type=text}}}, subjects={type=text}, dispositions={properties={disposition={index=true, type=keyword}, citations={properties={text={type=text}, title={type=text}, type={type=text}, url={type=text}}}, penalty={type=double}, respondent={type=text}}}, doc_id={index=true, type=keyword}, url={index=false, type=text}, respondents={type=text}, name={analyzer=english, type=text}, election_cycles={type=long}, commission_votes={properties={vote_date={format=dateOptionalTime, type=date}, text={type=text}}}, mur_type={type=text}, participants={properties={citations={type=object}, role={type=text}, name={type=text}}}}}]')

@pkfec
Copy link
Contributor Author

pkfec commented Aug 24, 2020

mapping type(s) disabled in ES7.*. Tried creating a sample students index on my local ES74 service.

curl -XPOST 'http://localhost:9200/students/student' -H 'Content-Type: application/json' -d'
{
"student": {
"properties": {
"name": "keyword",
"degree": "keyword",
"age": "integer"
}
}
}'

@JonellaCulmer JonellaCulmer mentioned this issue Aug 26, 2020
Closed
@lbeaufort lbeaufort modified the milestones: Sprint 13.3, Sprint 13.4 Aug 28, 2020
@fec-jli
Copy link
Contributor

fec-jli commented Sep 14, 2020

To do list:
1)revise test case;
2)modify “docs” mappings
3)adjust index_management.py to split separated files
4)revise backup/snapshot code;
5)revise s3 code for local, add try; check s3 on cloud
6)revise the legal api;
7)make sure frontend works;
8)modify other management commands ;
9)revise schedule task code.
10)…

@fec-jli
Copy link
Contributor

fec-jli commented Sep 25, 2020

09/24/2020 update:
finished:
1)Modify "mappings" and create new ES connection function with AWS4Auth, ES connection successfully.
2)upload all legal docs to ‘feature’ space successfully (except big arch-mur253 hit size limitation issue)
3)create es_management.py to put all manage functions in this file(separated from other upload code)
4)create new command: check_legal_doc to check all data on ES
5)create a public S3 bucket to save legal pdf file, config it to be viewed on browser successfully
6)setup circleci env variable and deploy api on feature space successfully
7)create a new private s3 bucket to save ES snapshot.(separated from public s3 bucket)
8)config ES snapshot successfully

to do list:
1)revise the legal api
2)make sure frontend works
3)revise schedule task code.
4)revise test case
5)make decision for big arch-mur253 pdf ocr text.

@pkfec
Copy link
Contributor Author

pkfec commented Sep 28, 2020 •
edited
Loading

Tracking all code changes in a WIP PR here #4596.

Rebased off develop branch and opened a new WIP PR #4657 and closed #4596

@pkfec pkfec mentioned this issue Oct 7, 2020
Merged
28 tasks
@rfultz rfultz modified the milestones: Sprint 13.6, PI 13 innovation 1 Oct 9, 2020
@jason-upchurch jason-upchurch mentioned this issue Oct 29, 2020
Closed
@jason-upchurch jason-upchurch changed the title Test Elasticsearch v7.4 beta service upgrade to Elasticsearch v7.4 beta service Oct 29, 2020
@rfultz rfultz modified the milestones: Sprint 13.8, Sprint 13.9 Nov 30, 2020
@patphongs patphongs mentioned this issue Mar 4, 2024
Closed
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pkfec pkfec

@fec-jli fec-jli

Labels
None yet
Projects
None yet
Milestone
Sprint 13.9
Development

Successfully merging a pull request may close this issue.

Upgrade es7 (elasticsearch) service fecgov/openFEC
7 participants
@pkfec @fec-jli @dorothyyeager @rfultz @lbeaufort @JonellaCulmer @jason-upchurch