Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated the PipelineModel to return a SinkModel for sinks #1802

Merged

Conversation

dlvenable
Copy link
Member

@dlvenable dlvenable commented Sep 24, 2022

Description

The primary goal of this PR is to build up the changes that propagate routes from the PipelineModel from the API level to the PipelineConfiguration in data-prepper-core. This is accomplished mainly through two changes.

  1. Updated PipelineModel to return SinkModel for the sinks. This is an API change.
  2. Added RoutedPluginSettings within data-prepper-core to support propagation of this information.

Issues Resolved

Contributes toward #1337 .

Check List

  • New functionality includes testing.
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…des fields for conditional routing.

Signed-off-by: David Venable <[email protected]>
@dlvenable dlvenable requested a review from a team as a code owner September 24, 2022 14:33
@@ -144,6 +146,11 @@ void deserialize_PipelinesDataFlowModel_with_route() throws IOException {
assertThat(pipelineModel.getSinks().size(), equalTo(1));
assertThat(pipelineModel.getSinks().get(0), notNullValue());
assertThat(pipelineModel.getSinks().get(0).getPluginName(), equalTo("testSink"));
assertThat(pipelineModel.getSinks().get(0).getRoutes(), notNullValue());
assertAll(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -116,6 +117,11 @@ private static PluginSetting getPluginSettingFromPluginModel(final PluginModel p
return new PluginSetting(pluginModel.getPluginName(), settingsMap == null ? new HashMap<>() : settingsMap);
}

private static RoutedPluginSetting getRoutedPluginSettingFromSinkModel(final SinkModel sinkModel) {
final Map<String, Object> settingsMap = sinkModel.getPluginSettings();
return new RoutedPluginSetting(sinkModel.getPluginName(), settingsMap == null ? new HashMap<>() : settingsMap, sinkModel.getRoutes());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The inline operator argument could be moved to its own line.

Another alternative is Optional

    public static String bar(final String arg) {
        return arg;
    }

    public static void main(String[] args) {
        final var foo = Optional.ofNullable(bar(args[0]));
        System.out.println(foo.orElse("default"));
    }

@Test
void testSinksWithRoutes() {
final List<Collection<String>> orderedSinkRoutes = new ArrayList<>();
for (SinkModel sink : sinks) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sink can be final

@codecov-commenter
Copy link

Codecov Report

Merging #1802 (53ebb6c) into main (715c7df) will decrease coverage by 0.02%.
The diff coverage is 96.42%.

@@             Coverage Diff              @@
##               main    #1802      +/-   ##
============================================
- Coverage     93.96%   93.94%   -0.03%     
- Complexity     1457     1478      +21     
============================================
  Files           188      189       +1     
  Lines          4275     4326      +51     
  Branches        347      354       +7     
============================================
+ Hits           4017     4064      +47     
- Misses          177      179       +2     
- Partials         81       83       +2     
Impacted Files Coverage Δ
...dataprepper/model/configuration/PipelineModel.java 100.00% <ø> (ø)
...opensearch/dataprepper/pipeline/ProcessWorker.java 73.52% <ø> (-0.76%) ⬇️
...prepper/parser/PipelineConfigurationValidator.java 71.18% <50.00%> (ø)
...zon/dataprepper/model/configuration/SinkModel.java 100.00% <100.00%> (ø)
.../opensearch/dataprepper/parser/PipelineParser.java 84.89% <100.00%> (ø)
...ataprepper/parser/model/PipelineConfiguration.java 98.18% <100.00%> (+2.26%) ⬆️
.../dataprepper/parser/model/RoutedPluginSetting.java 100.00% <100.00%> (ø)
.../org/opensearch/dataprepper/pipeline/Pipeline.java 92.75% <100.00%> (ø)
...h/dataprepper/logstash/mapping/LogstashMapper.java 100.00% <100.00%> (ø)
.../processor/parsejson/ParseJsonProcessorConfig.java 87.50% <0.00%> (-12.50%) ⬇️
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@dlvenable dlvenable merged commit b580a12 into opensearch-project:main Sep 28, 2022
@dlvenable dlvenable deleted the conditional-routing-internal-model branch September 28, 2022 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants