Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Publish segment schema from MSQ engine #16463

Closed
wants to merge 7 commits into from
Closed

Publish segment schema from MSQ engine #16463

wants to merge 7 commits into from

Conversation

findingrish
Copy link
Contributor

@findingrish findingrish commented May 16, 2024 •
edited
Loading

Parent issue: #14989

  • Introduce a new POJO class which extends DataSegment to include schema information. This will be sent by the workers to the Controller.
  • Changes in SegmentGeneratorFrameProcessor to generate segment schema.
  • Changes in ControllerImpl to publish segment schema along with segment metadata.

@github-actions github-actions bot added Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels May 16, 2024
@findingrish findingrish changed the title Msq schema publish Publish segment schema from MSQ engine May 16, 2024
kfaraz
kfaraz reviewed May 17, 2024
View reviewed changes
processing/src/main/java/org/apache/druid/timeline/DataSegmentExtendedWithSchema.java
import java.util.Map;
import java.util.Objects;

public class DataSegmentExtendedWithSchema extends DataSegment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already a class DataSegmentWithMetadata, why not reuse that?

kfaraz
kfaraz reviewed May 17, 2024
View reviewed changes
...g-service/src/main/java/org/apache/druid/indexing/common/actions/RemoteTaskActionClient.java
@@ -57,7 +57,7 @@ public RemoteTaskActionClient(
@Override
public <RetType> RetType submit(TaskAction<RetType> taskAction) throws IOException
{
log.debug("Performing action for task[%s]: %s", task.getId(), taskAction);
log.info("Performing action for task[%s]: %s", task.getId(), taskAction);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed? This can cause a lot of noise in task logs.

@github-actions GitHub Actions
Copy link

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the [email protected] list.
Thank you for your contributions.

@github-actions github-actions bot added the stale label Jul 17, 2024
@github-actions GitHub Actions
Copy link

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kfaraz kfaraz kfaraz left review comments

Assignees
No one assigned
Labels
Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@findingrish @kfaraz