Remove direct uses of hppc

[rjernst]

This is a meta issue for removing most uses of hppc from Elasticsearch.

Like any complex software project, Elasticsearch uses various data structures from computer science: lists, maps, sets, etc. Java provides interfaces for these structures (Java Collections Framework, JCF), and concrete implementations using well known techniques, like a hashtable based Map implementation, HashMap. The Java provided implementations are used throughout Elasticsearch.

Additionally, Elasticsearch server depends on the HPPC library. It provides alternatives to Java’s builtin collections that attempt to provide efficient, fast and “open” implementations (open here means easily hackable). The library is mostly wrapped with our own classes, namely ImmutableOpenMap and ImmutableOpenIntMap, but it is also used sporadically throughout the codebase. But when to use the HPPC based collections vs the Java ones has never been well defined within the team.

HPPC has been around for more than a decade, and while it once provided major performance benefits over JCF, Java has changed a lot in that time, especially in hotspot. It is unclear if HPPC actually provides advantages over JCF anymore. On top of this, the HPPC based collections are more difficult to use, and don’t easily interoperate with JCF interfaces, requiring conversion, or more often, forcing a developer to use HPPC because an upstream object used it, like when dealing with cluster state.

We have decided to try to move away from using hppc. Doing so is tricky since swapping out JCF classes are not trivial in most uses. This meta issue lists direct uses of hppc that should be either removed entirely, or evaluated for performance differences with JCF. Whether performance tests are needed would depend on the use case, and whether it is expected to be sensitive to memory footprint, and the objects lifetime.

• server
  • o.e.action.admin.cluster.stats (Remove hppc from ClusterStatsNodes #85639)
  • o.e.action Multi* request/response (Remove hppc from multi*shard request and responses #85888)
  • o.e.action.admin.indices (Remove unnecessary uses of ObjectHashSet #85911)
  • o.e.common (Remove hppc from some "common" classes #85957)
  • o.e.http (Remove hppc from http server transport #86024)
  • o.e.index.engine (Remove hppc from deletion policy #86072)
  • o.e.index.mapper.BinaryFieldMapper (Rewrite CollectionUtils dedup to work with any type #85352)
  • o.e.index.mapper (Use Set instead of LongSet in long script field #85417, Use Set instead of LongSet in double script field #85475)
  • o.e.index.seqno (Remove hppc from LocalCheckpointTracker #86073)
  • o.e.index.translog (Remove hppc from index translog #85657)
  • o.e.indices (Remove unnecessary uses of ObjectHashSet #85911)
  • o.e.rest.action.cat (Remove hppc from cat allocation api #85842)
  • o.e.search.aggregations.bucket.nested (Remove hppc from nested aggs #86078)
  • o.e.search.aggregations.bucket.terms (Remove hppc from terms agg #86151)
  • o.e.search.aggregations.metrics (Remove hppc from top hits agg #86150)
  • o.e.search.aggregations.pipeline (Remove hppc from percentiles agg #86023)
  • o.e.search.dfs (Remove hppc from dfs search #84688)
  • o.e.tasks (Remove hppc from task manager #85889)
  • o.e.transport (Remove hppc from tcp transport #85843)
• server unit tests
  • o.e.action.fieldcaps (Remove hppc from RequestDispatcherTests #85236)
  • FieldMemoryStatsTests (Remove hppc from FieldMemoryStats #85240)
  • o.e.common.util (Remove hppc from o.e.c.util tests #85406)
  • o.e.index (Remove hppc from o.e.index tests #85297)
  • SearchServiceTests (Remove hppc from FetchSearchPhase #85188)
  • BinaryRangeAggregatorTests (Remove hppc from search and aggs tests #85468)
  • o.e.search.aggregations.metrics (Remove hppc from search and aggs tests #85468)
  • o.e.search.runtime
  • o.e.search.slice
• server integ tests
  • AllocationAwarenessIT (Remove hppc uses from AwarenessAllocationIT #84736)
  • aggs and search tests (Remove hppc uses from server integ tests #84963)
• test framework
  • LinearizabilityChecker (Remove hppc from LinearizabilityChecker #84878)
  • InternalTestCluster (Convert ReplicationTracker insync global checkpoints getter to Map #84882)
  • TestCluster (Remove hppc use from TestCluster #84850)
• x-pack
  • core module (Remove hppc use from xpack core #84783)
  • analytics module (Remove hppc uses from histogram field mapper #84849)
  • ccr module (Remove hppc uses from xpack tests #84782)
  • ml module ([ML] remove use of hppc collections from NLP tokenization #84815)
  • sql qa (JdbcAssert) (Remove hppc uses from xpack tests #84782)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

With the exception of ImmutableOpenIntMap and ImmutableOpenMap, the uses of hppc have been removed. I opened a followup for those classes, as they are used quite extensively: #86239