Adjust SyntheticSourceLicenseService #116647

martijnvg · 2024-11-12T09:52:53Z

Allow gold and platinum license to use synthetic source for a limited time. If the start time of a license is before the cut off date, then gold and platinum licenses will not fallback to stored source if synthetic source is used.

Allow gold and platinum license to use synthetic source for a limited time. If the start time of a license if before a cut off date, then gold and platinum licenses will not fallback to stored source if synthetic source is used.

…tinum_licenses

...ogsdb/src/main/java/org/elasticsearch/xpack/logsdb/SyntheticSourceIndexSettingsProvider.java

salvatore-campagna · 2024-11-21T10:38:51Z

Do we also need to test for a customer switching license?

…cutoff date

…tinum_licenses

n1v0lg

Made a first pass and this is looking great. I left a few suggestions -- as discussed on Slack, it'd be great to convert the new IT case to use more realistic licensing primitives by extending AbstractLicensesIntegrationTestCase

I'll make another pass over the PR once that's ready.

...lugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/SyntheticSourceLicenseService.java

n1v0lg · 2024-11-21T14:53:33Z

...lugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/SyntheticSourceLicenseService.java

+    private static final String CUTOFF_DATE_SYS_PROP_NAME = "es.mapping.synthetic_source_fallback_to_stored_source.cutoff_date";
+    private static final Logger LOGGER = LogManager.getLogger(SyntheticSourceLicenseService.class);
+    static final long DEFAULT_CUTOFF_DATE = LocalDateTime.of(2024, 12, 12, 0, 0).toInstant(ZoneOffset.UTC).toEpochMilli();
+    private static final long MAX_CUTOFF_DATE = LocalDateTime.of(2026, 12, 12, 0, 0).toInstant(ZoneOffset.UTC).toEpochMilli();


Unless this is already confirmed, lets validate with product that this is the max cut off we want -- I don't have a very good sense for that this value should be, personally.

...ogsdb/src/main/java/org/elasticsearch/xpack/logsdb/SyntheticSourceIndexSettingsProvider.java

...lugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/SyntheticSourceLicenseService.java

.../logsdb/src/test/java/org/elasticsearch/xpack/logsdb/SyntheticSourceLicenseServiceTests.java

…Case

Co-authored-by: Nikolaj Volgushev <[email protected]>

martijnvg · 2024-11-21T18:17:56Z

Do we also need to test for a customer switching license?

@salvatore-campagna No, I don't think we need to test switching to gold/platinum license. We already have this test coverage by testing going from basic to enterprise and the other way around.

martijnvg · 2024-11-21T18:32:00Z

...ogsdb/src/main/java/org/elasticsearch/xpack/logsdb/SyntheticSourceIndexSettingsProvider.java

+        }
+        // To allow the following patterns: metrics-apm.transaction.*, metrics-apm.service_transaction.*, metrics-apm.service_summary.*,
+        // metrics-apm.service_destination.*, "metrics-apm.internal-* and metrics-apm.app.*
+        if (dataStreamName != null && dataStreamName.startsWith("metrics-apm.")) {


@felixbarny Double checking, is the pattern sufficient in selecting the apm data streams that currently use synthetic source? Synthetic source will require an enterprise license from 8.17.0, but for apm's synthetic source usage we will allow gold/platinum license for a little longer.

Yes, the pattern looks good. For reference, all the APM templates are in Elasticsearch: https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/apm-data/src/main/resources/index-templates

kkrik-es · 2024-11-25T15:08:10Z

...lugin/logsdb/src/test/java/org/elasticsearch/xpack/logsdb/LegacyLicenceIntegrationTests.java

+        putLicense(createGoldOrPlatinumLicense(startPastCutoff));
+        ensureGreen();
+
+        createIndexWithSyntheticSourceAndAssertExpectedType(".profiling-stacktraces", "STORED");


We should test with synthetic?

yes, this method creates an index with source mode set to synthetic. The last argument here is what the expected source mode should be.

kkrik-es · 2024-11-25T15:10:40Z

...a/org/elasticsearch/xpack/logsdb/SyntheticSourceIndexSettingsProviderLegacyLicenseTests.java

+        String indexName = DataStream.getDefaultBackingIndexName(dataStreamName, 0);
+        var result = provider.getAdditionalIndexSettings(indexName, dataStreamName, IndexMode.TIME_SERIES, null, null, settings, List.of());
+        assertThat(result.size(), equalTo(0));
+    }


Add tests with license at a later date?

kkrik-es · 2024-11-25T15:14:19Z

.../logsdb/src/test/java/org/elasticsearch/xpack/logsdb/SyntheticSourceLicenseServiceTests.java

+        return createGoldOrPlatinumLicense(start);
+    }
+
+    static License createGoldOrPlatinumLicense(long start) throws Exception {


Nit: remove this, pass the license string to the function above to deduplicate. Consider adding a boolean "isBeforeCutoff" to avoid passing dates.

kkrik-es

Some comments, ptal.

…tinum_licenses

elasticsearchmachine · 2024-11-26T07:03:19Z

💔 Backport failed

Status	Branch	Result
❌	8.17	Commit could not be cherrypicked due to conflicts
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 116647

Allow gold and platinum license to use synthetic source for a limited time. If the start time of a license is before the cut off date, then gold and platinum licenses will not fallback to stored source if synthetic source is used. Co-authored-by: Nikolaj Volgushev <[email protected]>

* Adjust SyntheticSourceLicenseService (#116647) Allow gold and platinum license to use synthetic source for a limited time. If the start time of a license is before the cut off date, then gold and platinum licenses will not fallback to stored source if synthetic source is used. Co-authored-by: Nikolaj Volgushev <[email protected]> * spotless --------- Co-authored-by: Nikolaj Volgushev <[email protected]>

Indicates whether es.mapping.synthetic_source_fallback_to_stored_source.cutoff_date_restricted_override system property has been configured. A follow up from #116647