Remove index_realtime and index_realtime_appenderator tasks #16602
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
index_realtime tasks were removed from the documentation in apache#13107. Even at that time, they weren't really documented per se— just mentioned. They existed solely to support Tranquility, which is an obsolete ingestion method that predates migration of Druid to ASF and is no longer being maintained. Tranquility docs were also de-linked from the sidebars and the other doc pages in apache#11134. Only a stub remains, so people with links to the page can see that it's no longer recommended. index_realtime_appenderator tasks existed in the code base, but were never documented, nor as far as I am aware were they used for any purpose. This patch removes both task types completely, as well as removes all supporting code that was otherwise unused. It also updates the stub doc for Tranquility to be firmer that it is not compatible. (Previously, the stub doc said it wasn't recommended, and pointed out that it is built against an ancient 0.9.2 version of Druid.)
github-actions
bot
added
Area - Documentation
Area - Batch Ingestion
Area - Querying
Area - Segment Format and Ser/De
Area - Streaming Ingestion
Area - Web Console
Area - Ingestion
Area - MSQ
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
integration-tests-ex/cases/src/test/java/org/apache/druid/testsEx/query/ITUnionQueryTest.java
Fixed
Show fixed
Hide fixed
integration-tests-ex/cases/src/test/java/org/apache/druid/testsEx/query/ITUnionQueryTest.java
Fixed
Show fixed
Hide fixed
integration-tests-ex/cases/src/test/java/org/apache/druid/testsEx/query/ITUnionQueryTest.java
Fixed
Show fixed
Hide fixed
kfaraz
approved these changes
Jun 18, 2024
There was a problem hiding this comment.
Thanks for reviving this, @clintropolis ! The changes look clean to me, we can proceed with merging this PR.
I have not reviewed the new IT very closely, but that can always be revisited later.
| labels: | ||
| druid-int-test: "true" | ||
| ports: | ||
| - 9092:9092 | ||
| - 9093:9093 | ||
| - 9094:9094 |
There was a problem hiding this comment.
i was having trouble getting the docker kafka with zookeeper working so i switched to using no zookeeper, which runs a controller on 9093, see https://github.com/bitnami/containers/blob/main/bitnami/kafka/README.md#using-a-docker-compose-file
There was a problem hiding this comment.
Ah, thanks for the clarification! Maybe we can include this info in the IT readme or somewhere in a follow up PR.
Comment on lines
+164
to
+168
| emitter = new ServiceEmitter( | ||
| "test", | ||
| "test", | ||
| new NoopEmitter() | ||
| ); |
There was a problem hiding this comment.
Nit:
Suggested change
| emitter = new ServiceEmitter( | |
| "test", | |
| "test", | |
| new NoopEmitter() | |
| ); | |
| emitter = new StubServiceEmitter(); |
1 task
clintropolis
added a commit
to clintropolis/druid
that referenced
this pull request
Jun 25, 2024
changes: * `FireHydrant` is now `PartialSegment`. This name much more clearly describes what this class does, and with all the other fireman terminology removed it didn't even fit a theme anymore. * `Sink` is now `AppendableSegment`. This name also much more clearly describes what this class does, and is composed of `PartialSegments` per the previous `FireHydrant` rename. * Additionally, `SinkQuerySegmentWalker` -> `AppendableSegmentQuerySegmentWalker`, and `SinkQueryRunner` -> `AppendableSegmentQueryRunner` * Remove `Firehose`, `IngestSegmentFirehose` was only used by Hadoop indexing `DruidRecordReader`, moved to internal class of `DruidRecordReader` as `SegmentReader` * Remove `FirehoseFactory` and remaining implementations, after apache#16602 they were no longer used
clintropolis
added a commit
to clintropolis/druid
that referenced
this pull request
Jun 25, 2024
changes: * `FireHydrant` is now `PartialSegment`. This name much more clearly describes what this class does, and with all the other fireman terminology removed it didn't even fit a theme anymore. * `Sink` is now `AppendableSegment`. This name also much more clearly describes what this class does, and is composed of `PartialSegments` per the previous `FireHydrant` rename. * Additionally, `SinkQuerySegmentWalker` -> `AppendableSegmentQuerySegmentWalker`, and `SinkQueryRunner` -> `AppendableSegmentQueryRunner` * Remove `Firehose`, `IngestSegmentFirehose` was only used by Hadoop indexing `DruidRecordReader`, moved to internal class of `DruidRecordReader` as `SegmentReader` * Remove `FirehoseFactory` and remaining implementations, after apache#16602 they were no longer used * Moved things from `org.apache.druid.segment.realtime.sink` and `org.apache.druid.segment.realtime.firehose` up to `org.apache.druid.segment.realtime`.
clintropolis
added a commit
to clintropolis/druid
that referenced
this pull request
Jun 25, 2024
changes: * `FireHydrant` is now `PartialSegment`. This name much more clearly describes what this class does, and with all the other fireman terminology removed it didn't even fit a theme anymore. * `Sink` is now `AppendableSegment`. This name also much more clearly describes what this class does, and is composed of `PartialSegments` per the previous `FireHydrant` rename. * Additionally, `SinkQuerySegmentWalker` -> `AppendableSegmentQuerySegmentWalker`, and `SinkQueryRunner` -> `AppendableSegmentQueryRunner` * Remove `Firehose`, `IngestSegmentFirehose` was only used by Hadoop indexing `DruidRecordReader`, moved to internal class of `DruidRecordReader` as `SegmentReader` * Remove `FirehoseFactory` and remaining implementations, after apache#16602 they were no longer used * Moved things from `org.apache.druid.segment.realtime.sink` and `org.apache.druid.segment.realtime.firehose` up to `org.apache.druid.segment.realtime`.
9 tasks
clintropolis
added a commit
to clintropolis/druid
that referenced
this pull request
Jul 19, 2024
changes: * removed `Firehose` and `FirehoseFactory` and remaining implementations which were mostly no longer used after apache#16602 * Moved `IngestSegmentFirehose` which was still used internally by Hadoop ingestion to `DatasourceRecordReader.SegmentReader` * Rename `SQLFirehoseFactoryDatabaseConnector` to `SQLInputSourceDatabaseConnector` and similar renames for sub-classes * Moved anything remaining in a 'firehose' package somewhere else * Clean up docs on firehose stuff
clintropolis
added a commit
that referenced
this pull request
Jul 19, 2024
changes: * removed `Firehose` and `FirehoseFactory` and remaining implementations which were mostly no longer used after #16602 * Moved `IngestSegmentFirehose` which was still used internally by Hadoop ingestion to `DatasourceRecordReader.SegmentReader` * Rename `SQLFirehoseFactoryDatabaseConnector` to `SQLInputSourceDatabaseConnector` and similar renames for sub-classes * Moved anything remaining in a 'firehose' package somewhere else * Clean up docs on firehose stuff
sreemanamala
pushed a commit
to sreemanamala/druid
that referenced
this pull request
Aug 6, 2024
changes: * removed `Firehose` and `FirehoseFactory` and remaining implementations which were mostly no longer used after apache#16602 * Moved `IngestSegmentFirehose` which was still used internally by Hadoop ingestion to `DatasourceRecordReader.SegmentReader` * Rename `SQLFirehoseFactoryDatabaseConnector` to `SQLInputSourceDatabaseConnector` and similar renames for sub-classes * Moved anything remaining in a 'firehose' package somewhere else * Clean up docs on firehose stuff
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
#15717 + fixed up conflicts +
ITUnionQueryTestusing the new integration test framework and kafkaRelease Notes
Druid
index_realtimeandindex_realtime_appenderatortasks have been removed and can no longer be used to ingest data. Docs forindex_realtimewere completely removed in Druid 0.22 in #11134, and prior to that we have been advising that these tasks were obsolete for much longer.index_realtime_appenderatortasks appear to have never been documented.index_realtimewas used to support Tranquility, which docs have been advising as unsupported and untested since 0.9.2.