GET index/_settings?include_defaults returns the wrong default value for settings with default values dependent on other settings

[gwbrown]

The Get Index Settings API returns the wrong default for index.soft_deletes.enabled in the 7.x series.

This is because the default is computed based on the version the index was created in:

elasticsearch/server/src/main/java/org/elasticsearch/index/IndexSettings.java

Lines 233 to 235 in 2488214

	public static final Setting<Boolean> INDEX_SOFT_DELETES_SETTING = Setting.boolSetting("index.soft_deletes.enabled",
	settings -> Boolean.toString(IndexMetaData.SETTING_INDEX_VERSION_CREATED.get(settings).onOrAfter(Version.V_7_0_0)),
	Property.IndexScope, Property.Final);

But the version is not included in the Settings object passed to the default calculation function when called by the Get Index Settings API with the include_defaults flag. It looks like the Settings object passed to calculate the defaults in this circumstance is always empty, so any setting whose defaults depend on another setting is likely to be wrong as well.

---

Update:

This definitely impacts other settings, as can be seen in #51194 where an incorrect default value of index.merge.scheduler.max_merge_count is returned from GET index/settings?include_defaults because its default value depends on the value of index.merge.scheduler.max_thread_count.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Settings)

[gwbrown]

In particular, I think this line is the problem:

elasticsearch/server/src/main/java/org/elasticsearch/action/admin/indices/settings/get/TransportGetSettingsAction.java

Line 108 in 2488214

Settings defaultSettings = settingsFilter.filter(indexScopedSettings.diff(indexSettings, Settings.EMPTY));

Specifically, the Settings.EMPTY there is what ends up getting passed to the function that calculates the setting default.

[ajoliveira]

+1 The CCR docs indicate that since this is default, nothing else is required if you have 7.X indices. However, no replication starts until you explicitly set index.soft_deletes.enabled to true ... which means you can't use the current index and have to go through steps to clone or re-index your data even though you would believe that it is defaulted to true.

[gwbrown]

@ajoliveira I believed this was a cosmetic bug - i.e. the underlying setting is true, it's just displayed as false. Soft deletes should be enabled by default, and behave as such, for indices created in all 7.x releases.

If you need to explicitly set the setting when creating an index for CCR to work, especially if CCR just silently doesn't work, you may have discovered a different (and much more severe) bug. Do you have reproduction steps?

[ajoliveira]

To reproduce:

• 2 single node clusters, both running 7.5.1, configured with CCR (bi-directional - but this shouldn't matter)
• to simulate data, enabled monitoring in clusters
• create auto-follow pattern

PUT /_ccr/auto_follow/monitoring_data
{
  "remote_cluster" : "<your_remote_cluster>",
  "leader_index_patterns" :
  [
    ".monitoring-es-7-*"
  ],
  "follow_index_pattern" : "follow{{leader_index}}"
}

GET _ccr/auto_follow
{
  "patterns" : [
    {
      "name" : "monitoring_data",
      "pattern" : {
        "active" : true,
        "remote_cluster" : "<your_remote_cluster>",
        "leader_index_patterns" : [
          ".monitoring-es-7-*"
        ],
        "follow_index_pattern" : "follow{{leader_index}}"
      }
    }
  ]
}

• follower index is never created
• output of GET _ccr/stats also never reports any stats
• to remedy: on '<your_remote_cluster>' cluster, I disabled monitoring, deleted current .monitoring-es-* index, updated .monitoring-es template to include index.soft_deletes.enabled: true and enabled monitoring again
• as soon as .monitoring-es-* index is created, you start to see the follower index on other cluster

Hope that helps.

[gwbrown]

@ajoliveira I'm not able to reproduce the issue, although I admit I've been trying with regular indices, rather than with monitoring for simplicity's sake (though I did use the index settings from the monitoring template).

I do note that you enabled monitoring prior to creating the auto-follow pattern, but per the docs auto-follow patterns only follow new indices:

Any time a new index with a name that matches one of the patterns in the collection is created in the remote cluster, a follower index is configured in the local cluster. The follower index uses the new index as its leader index.

That would explain why you see the follower index show up when deleting and re-creating the monitoring index. Are you sure that isn't the root cause of the issue?

[ajoliveira]

@gwbrown Doh! You are right, appears I overlooked that and when I came across this bug, thought it was the root cause of the issue. I reset everything on my system and see that follower indices are created correctly even with soft_deletes showing incorrectly. Apologies for side tracking this.