Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Logstash OpenSearch output plugin mapping to converter #756

Merged
merged 2 commits into from
Dec 22, 2021
Merged

Added Logstash OpenSearch output plugin mapping to converter #756

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a429acc
Added support for negating boolean logstash attribute value
asifsmohammed Dec 16, 2021
84eaa84
Updated support for negating boolean logstash attribute value
asifsmohammed Dec 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions ...a/org/opensearch/dataprepper/logstash/mapping/AbstractLogstashPluginAttributesMapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,15 +36,22 @@ public Map<String, Object> mapAttributes(final List<LogstashAttribute> logstashA
.stream()
.filter(logstashAttribute -> !customMappedAttributeNames.contains(logstashAttribute.getAttributeName()))
.forEach(logstashAttribute -> {
if (mappedAttributeNames.containsKey(logstashAttribute.getAttributeName())) {
pluginSettings.put(
mappedAttributeNames.get(logstashAttribute.getAttributeName()),
logstashAttribute.getAttributeValue().getValue()
);
}
else {
LOG.warn("Attribute name {} is not found in mapping file.", logstashAttribute.getAttributeName());
}
final String logstashAttributeName = logstashAttribute.getAttributeName();
final String dataPrepperAttributeName = mappedAttributeNames.get(logstashAttributeName);

if (mappedAttributeNames.containsKey(logstashAttributeName)) {
if (dataPrepperAttributeName.startsWith("!") && logstashAttribute.getAttributeValue().getValue() instanceof Boolean) {
sbayer55 marked this conversation as resolved.
Show resolved Hide resolved
pluginSettings.put(
dataPrepperAttributeName.substring(1), !(Boolean) logstashAttribute.getAttributeValue().getValue()
);
}
else {
pluginSettings.put(dataPrepperAttributeName, logstashAttribute.getAttributeValue().getValue());
}
}
else {
LOG.warn("Attribute name {} is not found in mapping file.", logstashAttributeName);
}
});

if (!customMappedAttributeNames.isEmpty()) {
Expand Down
7 changes: 7 additions & 0 deletions ...on/src/main/resources/org/opensearch/dataprepper/logstash/mapping/opensearch.mapping.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
pluginName: opensearch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an integration test case for this.

See the LogstashConfigConverterIT Javadocs for how to add a new case. It only requires creating input and expected files.

mappedAttributeNames:
hosts: hosts
user: username
password: password
index: index
ssl_certificate_verification: "!insecure"
20 changes: 19 additions & 1 deletion ...rg/opensearch/dataprepper/logstash/mapping/DefaultLogstashPluginAttributesMapperTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.opensearch.dataprepper.logstash.model.LogstashAttribute;
import org.opensearch.dataprepper.logstash.model.LogstashAttributeValue;

Expand All @@ -28,14 +30,15 @@ class DefaultLogstashPluginAttributesMapperTest {
private String logstashAttributeName;
private String value;
private List<LogstashAttribute> logstashAttributes;
private LogstashAttributeValue logstashAttributeValue;
private LogstashAttributesMappings mappings;

@BeforeEach
void setUp() {
value = UUID.randomUUID().toString();
logstashAttributeName = UUID.randomUUID().toString();
final LogstashAttribute logstashAttribute = mock(LogstashAttribute.class);
final LogstashAttributeValue logstashAttributeValue = mock(LogstashAttributeValue.class);
logstashAttributeValue = mock(LogstashAttributeValue.class);
when(logstashAttributeValue.getValue()).thenReturn(value);
when(logstashAttribute.getAttributeName()).thenReturn(logstashAttributeName);
when(logstashAttribute.getAttributeValue()).thenReturn(logstashAttributeValue);
Expand Down Expand Up @@ -114,4 +117,19 @@ void mapAttributes_sets_additional_attributes_to_those_values() {
assertThat(actualPluginSettings, hasKey(additionalAttributeName));
assertThat(actualPluginSettings.get(additionalAttributeName), equalTo(additionalAttributeValue));
}

@ParameterizedTest
@ValueSource(booleans = { true, false } )
void mapAttributes_with_negation_expression_negates_boolean_value(boolean inputValue) {
final String dataPrepperAttribute = "!".concat(UUID.randomUUID().toString());

when(logstashAttributeValue.getValue()).thenReturn(inputValue);
when(mappings.getMappedAttributeNames()).thenReturn(Collections.singletonMap(logstashAttributeName, dataPrepperAttribute));

final Map<String, Object> actualPluginSettings = createObjectUnderTest().mapAttributes(logstashAttributes, mappings);

assertThat(actualPluginSettings, notNullValue());
assertThat(actualPluginSettings.size(), equalTo(1));
assertThat(actualPluginSettings.get(dataPrepperAttribute.substring(1)), equalTo(!inputValue));
}
}
18 changes: 18 additions & 0 deletions .../opensearch/dataprepper/logstash/log-ingest-to-logstash-opensearch-insecure.expected.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
logstash-converted-pipeline:
source:
http:
max_connection_count: 500
request_timeout: 10000
processor:
- grok:
match:
log:
- "%{COMBINEDAPACHELOG}"
sink:
- opensearch:
hosts:
- "fakedomain.us-east-1.es.amazonaws.com"
username: "myuser"
password: "mypassword"
index: "my-index"
insecure: true
18 changes: 18 additions & 0 deletions ...test/resources/org/opensearch/dataprepper/logstash/log-ingest-to-logstash-opensearch.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
input {
http {
}
}
filter {
grok {
match => {"log" => "%{COMBINEDAPACHELOG}"}
}
}
output {
opensearch {
hosts => ["fakedomain.us-east-1.es.amazonaws.com"]
user => myuser
password => mypassword
index => "my-index"
ssl_certificate_verification => true
}
}
18 changes: 18 additions & 0 deletions ...urces/org/opensearch/dataprepper/logstash/log-ingest-to-logstash-opensearch.expected.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
logstash-converted-pipeline:
source:
http:
max_connection_count: 500
request_timeout: 10000
processor:
- grok:
match:
log:
- "%{COMBINEDAPACHELOG}"
sink:
- opensearch:
hosts:
- "fakedomain.us-east-1.es.amazonaws.com"
username: "myuser"
password: "mypassword"
index: "my-index"
insecure: false
Copy link
Member

@graytaylor0 graytaylor0 Dec 21, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I would rather see an integration test with a non-default value for insecure, as it feels like there is some chance that it isn't actually taking into consideration and negating the value from ssl_certificate_verification

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having two test cases here makes sense.

By the way, default values shouldn't end up in the generated YAML. So there is a good indication that this is working. But, I'm still for adding another input file.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. Yeah, as David mentioned default values won't be there in YAML files.
I'll add one more integration test.

18 changes: 18 additions & 0 deletions ...rces/org/opensearch/dataprepper/logstash/log-ingest-to-logtstash-opensearch-insecure.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
input {
http {
}
}
filter {
grok {
match => {"log" => "%{COMBINEDAPACHELOG}"}
}
}
output {
opensearch {
hosts => ["fakedomain.us-east-1.es.amazonaws.com"]
user => myuser
password => mypassword
index => "my-index"
ssl_certificate_verification => false
}
}