Support filtering data in Auto Compaction

[maytasm]

Support filtering data in Auto Compaction

Description

This PR adds filter field via the transformSpec to manual compaction and auto compaction.
By setting filter, the user will be able to drop data matching the filter from the datasource.

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[abhishekagarwal87]

is the PR going to add more changes? One thing I found interesting is that we are re-compacting the data if the filter changes. But the data we are compacting has already passed through a filter. Those records are gone and are not going to come back. Say if I first do a filter on dim = a and then change the filter to dim = b. It might be nice to add a warning in the UI for this when we eventually add those changes.

[abhishekagarwal87]

is it missing JsonProperty annotation?

[maytasm]

Yes. Fixed

[abhishekagarwal87]

Suggested change

	ClientCompactionTaskTransformSpec transformSpec;
	if (config.getTransformSpec() != null) {
	transformSpec = new ClientCompactionTaskTransformSpec(
	config.getTransformSpec().getFilter()
	);
	} else {
	transformSpec = null;
	}
	ClientCompactionTaskTransformSpec transformSpec = null;
	if (config.getTransformSpec() != null) {
	transformSpec = new ClientCompactionTaskTransformSpec(
	config.getTransformSpec().getFilter()
	);
	}

[abhishekagarwal87]

nit - could be simplified.

[maytasm]

Done

[maytasm]

is the PR going to add more changes? One thing I found interesting is that we are re-compacting the data if the filter changes. But the data we are compacting has already passed through a filter. Those records are gone and are not going to come back. Say if I first do a filter on dim = a and then change the filter to dim = b. It might be nice to add a warning in the UI for this when we eventually add those changes.

I have finish adding all changes to this PR (implementation, unit tests, docs, integration tests). Your understanding is correct. That is now true for other auto compaction config that can remove data. For example, if you remove one of your dimension using auto compaction then add that dimension back later, the removed dimension will not come back (although that dimension in future data will not be removed). I'll work on clarifying this in the docs to make sure that user are well aware of this new behavior in a separate PR.

[suneet-s]

Overall LGTM.

2 high level comments since you are making a lot of changes in this area:

1. It would be good to have some sort of integration test with segments in different compaction states so that we can simulate an upgrade where users have segments with different compaction states and validate that we don't re-compact segments each time we add a new field here.
2. It would be good to add a warning in the docs that filtering data that matches an entire segment may not work as expected since Druid doesn't handle "empty segments" too well today.

All my comments do not need to block this PR, and can be addressed in future changes.

[suneet-s]

nit: I think this should be done at the test class level.