diff --git a/CONFIGURE.md b/CONFIGURE.md
index 48c1c2186d..507500b0f0 100644
--- a/CONFIGURE.md
+++ b/CONFIGURE.md
@@ -20,6 +20,7 @@
- [Table of Contents](#table-of-contents)
- [Configuring](#configuring)
- [Parameter Contexts](#parameter-contexts)
+ - [Parameter Providers](#parameter-providers)
- [Configuring flow configuration format](#configuring-flow-configuration-format)
- [Scheduling strategies](#scheduling-strategies)
- [Configuring encryption for flow configuration](#configuring-encryption-for-flow-configuration)
@@ -79,49 +80,50 @@ MiNiFi Toolkit Converter (version 0.0.1 - schema version 1) is considered as dep
It's recommended to create your configuration in YAML format or configure the agent via Command and Control protocol (see below)
-
- Flow Controller:
- id: 471deef6-2a6e-4a7d-912a-81cc17e3a205
- name: MiNiFi Flow
-
- Processors:
- - name: GetFile
- id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
- class: org.apache.nifi.processors.standard.GetFile
- max concurrent tasks: 1
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1 sec
- penalization period: 30 sec
- yield period: 1 sec
- run duration nanos: 0
- auto-terminated relationships list:
- Properties:
- Input Directory: /tmp/getfile
- Keep Source File: true
-
- Connections:
- - name: TransferFilesToRPG
- id: 471deef6-2a6e-4a7d-912a-81cc17e3a207
- source name: GetFile
- source id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
- source relationship name: success
- destination id: 471deef6-2a6e-4a7d-912a-81cc17e3a204
- max work queue size: 0
- max work queue data size: 1 MB
- flowfile expiration: 60 sec
- drop empty: false
-
- Remote Processing Groups:
- - name: NiFi Flow
- id: 471deef6-2a6e-4a7d-912a-81cc17e3a208
- url: http://localhost:8080/nifi
- timeout: 30 secs
- yield period: 10 sec
- Input Ports:
- - id: 471deef6-2a6e-4a7d-912a-81cc17e3a204
- name: From Node A
- max concurrent tasks: 1
- Properties:
+```yaml
+Flow Controller:
+ id: 471deef6-2a6e-4a7d-912a-81cc17e3a205
+ name: MiNiFi Flow
+
+Processors:
+ - name: GetFile
+ id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
+ class: org.apache.nifi.processors.standard.GetFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ penalization period: 30 sec
+ yield period: 1 sec
+ run duration nanos: 0
+ auto-terminated relationships list:
+ Properties:
+ Input Directory: /tmp/getfile
+ Keep Source File: true
+
+Connections:
+ - name: TransferFilesToRPG
+ id: 471deef6-2a6e-4a7d-912a-81cc17e3a207
+ source name: GetFile
+ source id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
+ source relationship name: success
+ destination id: 471deef6-2a6e-4a7d-912a-81cc17e3a204
+ max work queue size: 0
+ max work queue data size: 1 MB
+ flowfile expiration: 60 sec
+ drop empty: false
+
+Remote Processing Groups:
+ - name: NiFi Flow
+ id: 471deef6-2a6e-4a7d-912a-81cc17e3a208
+ url: http://localhost:8080/nifi
+ timeout: 30 secs
+ yield period: 10 sec
+ Input Ports:
+ - id: 471deef6-2a6e-4a7d-912a-81cc17e3a204
+ name: From Node A
+ max concurrent tasks: 1
+ Properties:
+```
Besides YAML configuration format, MiNiFi C++ also supports JSON configuration. To see different uses cases in both formats, please refer to the [examples page](examples/README.md) for flow config examples.
@@ -204,46 +206,170 @@ An example for using parameters in a JSON configuration file:
An example for using parameters in a YAML configuration file:
```yaml
- MiNiFi Config Version: 3
- Flow Controller:
- name: MiNiFi Flow
- Parameter Contexts:
- - id: 235e6b47-ea22-45cd-a472-545801db98e6
- name: common-parameter-context
- description: Common parameter context
- Parameters:
- - name: common_timeout
- description: 'Common timeout seconds'
- sensitive: false
- value: 30
- - id: 804e6b47-ea22-45cd-a472-545801db98e6
- name: root-process-group-context
- description: Root process group parameter context
- Parameters:
- - name: tail_base_dir
- description: 'Base dir of tailed files'
- sensitive: false
- value: /tmp/tail/file/path
- Inherited Parameter Contexts:
- - common-parameter-context
- Processors:
- - name: Tail test_file1.log
- id: 83b58f9f-e661-4634-96fb-0e82b92becdf
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file1.log"
- - name: Tail test_file2.log
- id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file2.log"
- Parameter Context Name: root-process-group-context
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Parameter Contexts:
+ - id: 235e6b47-ea22-45cd-a472-545801db98e6
+ name: common-parameter-context
+ description: Common parameter context
+ Parameters:
+ - name: common_timeout
+ description: 'Common timeout seconds'
+ sensitive: false
+ value: 30
+ - id: 804e6b47-ea22-45cd-a472-545801db98e6
+ name: root-process-group-context
+ description: Root process group parameter context
+ Parameters:
+ - name: tail_base_dir
+ description: 'Base dir of tailed files'
+ sensitive: false
+ value: /tmp/tail/file/path
+ Inherited Parameter Contexts:
+ - common-parameter-context
+Processors:
+- name: Tail test_file1.log
+ id: 83b58f9f-e661-4634-96fb-0e82b92becdf
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file1.log"
+- name: Tail test_file2.log
+ id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file2.log"
+Parameter Context Name: root-process-group-context
+```
+
+### Parameter Providers
+
+Parameter contexts can be generated by Parameter Providers. Parameter Providers can be added to the flow configuration, after which parameter contexts and parameters generated by these providers can be referenced in the properties. The parameter contexts generated are persisted in the flow configuration file and are only regenerated on MiNiFi C++ restart if the context is removed from the flow configuration. Other parameter contexts can be also inherited from provider generated parameter contexts.
+
+There are two properties that can be set for all parameter providers for selecting which properties should be sensitive parameters:
+
+- `Sensitive Parameter Scope`: This property can be set to `none`, `selected` or `all`. If set to `All`, all parameters generated by the provider will be marked as sensitive. If set to `none`, all parameters generated by the provider will be marked as non-sensitive. If set to `selected`, the `Sensitive Parameter List` property should be set to a list of parameter names that should be marked as sensitive.
+- `Sensitive Parameter List`: This property should be set to a comma-separated list of parameter names that should be marked as sensitive. This property is only used if the `Sensitive Parameter Scope` property is set to `selected`.
+
+An example for using parameter providers in a JSON configuration file:
+
+```json
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Parameter Group Name": "environment-variable-parameter-context",
+ "Environment Variable Inclusion Strategy": "Regular Expression",
+ "Include Environment Variables": "INPUT_.*"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Input Directory": "#{INPUT_DIR}"
+ }
+ }
+ ],
+ "parameterContextName": "environment-variable-parameter-context"
+ }
+}
+```
+
+The same example in YAML configuration file:
+
+```yaml
+MiNiFi Config Version: 3
+Flow Controller:
+name: MiNiFi Flow
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: EnvironmentVariableParameterProvider
+ type: EnvironmentVariableParameterProvider
+ Properties:
+ Parameter Group Name: environment-variable-parameter-context
+ Environment Variable Inclusion Strategy: Regular Expression
+ Include Environment Variables: INPUT_.*
+Processors:
+ - name: MyProcessor
+ id: 00000000-0000-0000-0000-000000000001
+ class: org.apache.nifi.processors.GetFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 3 sec
+ Properties:
+ Input Directory: "#{INPUT_DIR}"
+Parameter Context Name: environment-variable-parameter-context
```
+In the above example, the `EnvironmentVariableParameterProvider` is used to generate a parameter context with the name `environment-variable-parameter-context` that includes all environment variables starting with `INPUT_`. The generated parameter context is assigned to the root process group and the `INPUT_DIR` environment variable is used in the `Input Directory` property of the `MyProcessor` processor which is a generated parameter in the `environment-variable-parameter-context` parameter context.
+
+After the parameter contexts are generated successfully, the parameter contexts are persisted in the flow configuration file, which looks like this for the above example:
+
+```json
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Parameter Group Name": "environment-variable-parameter-context",
+ "Environment Variable Inclusion Strategy": "Regular Expression",
+ "Include Environment Variables": "INPUT_.*"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Input Directory": "#{INPUT_DIR}"
+ }
+ }
+ ],
+ "parameterContextName": "environment-variable-parameter-context"
+ },
+ "parameterContexts": [
+ {
+ "identifier": "a48df754-a0f4-11ef-ae56-10f60a596f64",
+ "name": "environment-variable-parameter-context",
+ "parameterProvider": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "parameters": [
+ {
+ "name": "INPUT_DIR",
+ "description": "",
+ "sensitive": false,
+ "provided": true,
+ "value": "/tmp/input/"
+ }
+ ]
+ }
+ ]
+}
+```
+
+To see the full list of available parameter providers and their properties, please refer to the [Parameter Providers documentation](PARAMETER_PROVIDERS.md).
+
### Configuring flow configuration format
MiNiFi supports YAML and JSON configuration formats. The desired configuration format can be set in the minifi.properties file, but it is automatically identified by default. The default value is `adaptiveconfiguration`, but we can force to use YAML with the `yamlconfiguration` value.
diff --git a/PARAMETER_PROVIDERS.md b/PARAMETER_PROVIDERS.md
new file mode 100644
index 0000000000..bfe9184bcb
--- /dev/null
+++ b/PARAMETER_PROVIDERS.md
@@ -0,0 +1,37 @@
+
+## Table of Contents
+
+- [EnvironmentVariableParameterProvider](#EnvironmentVariableParameterProvider)
+
+## EnvironmentVariableParameterProvider
+
+### Description
+
+Fetches parameters from environment variables
+
+### Properties
+
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.
+
+| Name | Default Value | Allowable Values | Description |
+|---------------------------------------------|---------------|--------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Environment Variable Inclusion Strategy** | Include All | Include All
Comma-Separated
Regular Expression | Indicates how Environment Variables should be included |
+| Include Environment Variables | | | Specifies comma separated environment variable names or regular expression (depending on the Environment Variable Inclusion Strategy) that should be used to fetch environment variables. |
+| **Parameter Group Name** | | | The name of the parameter group that will be fetched. This indicates the name of the Parameter Context that may receive the fetched parameters. |
+
+### Generated Parameter Contexts
+
+This provider generates a single Parameter Context with the name specified in the `Parameter Group Name` property. The parameters generated match the name of the environment variables that are included.
diff --git a/cmake/BuildTests.cmake b/cmake/BuildTests.cmake
index 37e19fa106..6254fe1719 100644
--- a/cmake/BuildTests.cmake
+++ b/cmake/BuildTests.cmake
@@ -58,6 +58,7 @@ function(appendIncludes testName)
target_include_directories(${testName} BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/libminifi/include/core/yaml")
target_include_directories(${testName} BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/libminifi/include/core/statemanagement")
target_include_directories(${testName} BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/libminifi/include/core/statemanagement/metrics")
+ target_include_directories(${testName} BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/libminifi/include/core/parameter-providers")
target_include_directories(${testName} BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/libminifi/include/io")
if(WIN32)
target_include_directories(${testName} BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/libminifi/opsys/win")
diff --git a/extensions/standard-processors/tests/unit/FlowJsonTests.cpp b/extensions/standard-processors/tests/unit/FlowJsonTests.cpp
index 6dc048f71f..396c3139d3 100644
--- a/extensions/standard-processors/tests/unit/FlowJsonTests.cpp
+++ b/extensions/standard-processors/tests/unit/FlowJsonTests.cpp
@@ -30,6 +30,7 @@
#include "core/Resource.h"
#include "utils/crypto/property_encryption/PropertyEncryptionUtils.h"
#include "unit/TestUtils.h"
+#include "unit/DummyParameterProvider.h"
using namespace std::literals::chrono_literals;
@@ -334,31 +335,6 @@ TEST_CASE("Cannot use the same parameter name within a parameter context twice")
REQUIRE_THROWS_WITH(config.getRootFromPayload(CONFIG_JSON), "Parameter Operation: Parameter name 'file_size' already exists, parameter names must be unique within a parameter context!");
}
-class DummyFlowJsonProcessor : public core::Processor {
- public:
- using core::Processor::Processor;
-
- static constexpr const char* Description = "A processor that does nothing.";
- static constexpr auto SimpleProperty = core::PropertyDefinitionBuilder<>::createProperty("Simple Property")
- .withDescription("Just a simple string property")
- .build();
- static constexpr auto SensitiveProperty = core::PropertyDefinitionBuilder<>::createProperty("Sensitive Property")
- .withDescription("Sensitive property")
- .isSensitive(true)
- .build();
- static constexpr auto Properties = std::to_array({SimpleProperty, SensitiveProperty});
- static constexpr auto Relationships = std::array{};
- static constexpr bool SupportsDynamicProperties = true;
- static constexpr bool SupportsDynamicRelationships = true;
- static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_ALLOWED;
- static constexpr bool IsSingleThreaded = false;
- ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS
-
- void initialize() override { setSupportedProperties(Properties); }
-};
-
-REGISTER_RESOURCE(DummyFlowJsonProcessor, Processor);
-
TEST_CASE("Cannot use non-sensitive parameter in sensitive property") {
ConfigurationTestController test_controller;
@@ -387,7 +363,7 @@ TEST_CASE("Cannot use non-sensitive parameter in sensitive property") {
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyGenFF",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -430,7 +406,7 @@ TEST_CASE("Cannot use non-sensitive parameter in sensitive property value sequen
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyGenFF",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -641,7 +617,7 @@ TEST_CASE("Cannot use parameters if no parameter context is defined") {
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyGenFF",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -667,7 +643,7 @@ TEST_CASE("Cannot use parameters in property value sequences if no parameter con
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyGenFF",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -717,7 +693,7 @@ TEST_CASE("Property value sequences can use parameters") {
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -778,7 +754,7 @@ TEST_CASE("Dynamic properties can use parameters") {
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -839,7 +815,7 @@ TEST_CASE("Test sensitive parameters in sensitive properties") {
"processors": [{{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {{
@@ -896,7 +872,7 @@ TEST_CASE("Test sensitive parameters in sensitive property value sequence") {
"processors": [{{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {{
@@ -1073,7 +1049,7 @@ TEST_CASE("Test parameter context inheritance") {
"processors": [{{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {{
@@ -1123,7 +1099,7 @@ TEST_CASE("Parameter context can not inherit from a itself") {
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -1166,7 +1142,7 @@ TEST_CASE("Parameter context can not inherit from non-existing parameter context
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -1251,7 +1227,7 @@ TEST_CASE("Cycles are not allowed in parameter context inheritance") {
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -1332,7 +1308,7 @@ TEST_CASE("Parameter context inheritance order is respected") {
"processors": [{
"identifier": "00000000-0000-0000-0000-000000000001",
"name": "MyProcessor",
- "type": "org.apache.nifi.processors.DummyFlowJsonProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
"schedulingStrategy": "TIMER_DRIVEN",
"schedulingPeriod": "3 sec",
"properties": {
@@ -1358,4 +1334,263 @@ TEST_CASE("Parameter context inheritance order is respected") {
CHECK(value == "5");
}
+TEST_CASE("Parameter providers can be used for parameter values") {
+ ConfigurationTestController test_controller;
+ core::flow::AdaptiveConfiguration config(test_controller.getContext());
+
+ static const std::string CONFIG_JSON =
+ R"(
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "DummyParameterProvider",
+ "type": "DummyParameterProvider",
+ "properties": {
+ "Dummy1 Value": "value1",
+ "Dummy2 Value": "value2",
+ "Dummy3 Value": "value3"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [{
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Simple Property": "#{dummy1}",
+ "My Dynamic Property Sequence": [
+ {"value": "#{dummy2}"},
+ {"value": "#{dummy3}"}
+ ]
+ }
+ }],
+ "parameterContextName": "dummycontext"
+ }
+})";
+
+ std::unique_ptr flow = config.getRootFromPayload(CONFIG_JSON);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("MyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Simple Property") == "value1");
+ core::Property property("My Dynamic Property Sequence", "");
+ proc->getDynamicProperty("My Dynamic Property Sequence", property);
+ auto values = property.getValues();
+ REQUIRE(values.size() == 2);
+ CHECK(values[0] == "value2");
+ CHECK(values[1] == "value3");
+}
+
+TEST_CASE("Parameter providers can be configured to select which parameters to be sensitive") {
+ ConfigurationTestController test_controller;
+ auto context = test_controller.getContext();
+ auto encrypted_sensitive_property_value = minifi::utils::crypto::property_encryption::encrypt("#{dummy1}", *context.sensitive_values_encryptor);
+ core::flow::AdaptiveConfiguration config(context);
+
+ static const std::string CONFIG_JSON =
+ fmt::format(R"(
+{{
+ "parameterProviders": [
+ {{
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "DummyParameterProvider",
+ "type": "DummyParameterProvider",
+ "properties": {{
+ "Sensitive Parameter Scope": "selected",
+ "Sensitive Parameter List": "dummy1",
+ "Dummy1 Value": "value1",
+ "Dummy3 Value": "value3"
+ }}
+ }}
+ ],
+ "rootGroup": {{
+ "name": "MiNiFi Flow",
+ "processors": [{{
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {{
+ "Simple Property": "#{{dummy3}}",
+ "Sensitive Property": "{}"
+ }}
+ }}],
+ "parameterContextName": "dummycontext"
+ }}
+}})", encrypted_sensitive_property_value);
+
+ std::unique_ptr flow = config.getRootFromPayload(CONFIG_JSON);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("MyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Sensitive Property") == "value1");
+ REQUIRE(proc->getProperty("Simple Property") == "value3");
+}
+
+TEST_CASE("Parameter providers can be configured to make all parameters sensitive") {
+ ConfigurationTestController test_controller;
+ auto context = test_controller.getContext();
+ auto encrypted_sensitive_property_value = minifi::utils::crypto::property_encryption::encrypt("#{dummy1}", *context.sensitive_values_encryptor);
+ core::flow::AdaptiveConfiguration config(context);
+
+ static const std::string CONFIG_JSON =
+ fmt::format(R"(
+{{
+ "parameterProviders": [
+ {{
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "DummyParameterProvider",
+ "type": "DummyParameterProvider",
+ "properties": {{
+ "Sensitive Parameter Scope": "all",
+ "Dummy1 Value": "value1",
+ "Dummy3 Value": "value3"
+ }}
+ }}
+ ],
+ "rootGroup": {{
+ "name": "MiNiFi Flow",
+ "processors": [{{
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {{
+ "Sensitive Property": "{}"
+ }}
+ }}],
+ "parameterContextName": "dummycontext"
+ }}
+}})", encrypted_sensitive_property_value);
+
+ std::unique_ptr flow = config.getRootFromPayload(CONFIG_JSON);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("MyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Sensitive Property") == "value1");
+}
+
+TEST_CASE("Parameter context can be inherited from parameter provider generated parameter context") {
+ ConfigurationTestController test_controller;
+ core::flow::AdaptiveConfiguration config(test_controller.getContext());
+
+ static const std::string CONFIG_JSON =
+ R"(
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "DummyParameterProvider",
+ "type": "DummyParameterProvider",
+ "properties": {
+ "Dummy1 Value": "value1",
+ "Dummy2 Value": "value2",
+ "Dummy3 Value": "value3"
+ }
+ }
+ ],
+ "parameterContexts": [
+ {
+ "identifier": "721e10b7-8e00-3188-9a27-476cca376978",
+ "name": "my-context",
+ "description": "my parameter context",
+ "parameters": [
+ {
+ "name": "file_size",
+ "description": "",
+ "sensitive": false,
+ "value": "10 B"
+ }
+ ],
+ "inheritedParameterContexts": ["dummycontext"]
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [{
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Simple Property": "#{dummy1}"
+ }
+ }],
+ "parameterContextName": "my-context"
+ }
+})";
+
+ std::unique_ptr flow = config.getRootFromPayload(CONFIG_JSON);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("MyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Simple Property") == "value1");
+}
+
+TEST_CASE("Parameter context names cannot conflict with parameter provider generated parameter context names") {
+ ConfigurationTestController test_controller;
+ core::flow::AdaptiveConfiguration config(test_controller.getContext());
+
+ static const std::string CONFIG_JSON =
+ R"(
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "DummyParameterProvider",
+ "type": "DummyParameterProvider",
+ "properties": {
+ "Dummy1 Value": "value1",
+ "Dummy2 Value": "value2",
+ "Dummy3 Value": "value3"
+ }
+ }
+ ],
+ "parameterContexts": [
+ {
+ "identifier": "721e10b7-8e00-3188-9a27-476cca376978",
+ "name": "dummycontext",
+ "description": "my parameter context",
+ "parameters": [
+ {
+ "name": "file_size",
+ "description": "",
+ "sensitive": false,
+ "value": "10 B"
+ }
+ ]
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [{
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.DummyProcessor",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Simple Property": "#{dummy1}"
+ }
+ }],
+ "parameterContextName": "dummycontext"
+ }
+})";
+
+ REQUIRE_THROWS_WITH(config.getRootFromPayload(CONFIG_JSON), "Parameter provider 'DummyParameterProvider' cannot create parameter context 'dummycontext' because parameter context already exists "
+ "with no parameter provider or generated by other parameter provider");
+}
+
} // namespace org::apache::nifi::minifi::test
diff --git a/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp b/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp
index 0506c75ca8..1988a3e6fa 100644
--- a/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp
+++ b/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp
@@ -31,6 +31,7 @@
#include "unit/TestUtils.h"
#include "core/Resource.h"
#include "utils/crypto/property_encryption/PropertyEncryptionUtils.h"
+#include "unit/DummyProcessor.h"
using namespace std::literals::chrono_literals;
@@ -1261,31 +1262,6 @@ NiFi Properties Overrides: {}
"Parameter Operation: Parameter name 'lookup.frequency' already exists, parameter names must be unique within a parameter context!");
}
-class DummyFlowYamlProcessor : public core::Processor {
- public:
- using core::Processor::Processor;
-
- static constexpr const char* Description = "A processor that does nothing.";
- static constexpr auto SimpleProperty = core::PropertyDefinitionBuilder<>::createProperty("Simple Property")
- .withDescription("Just a simple string property")
- .build();
- static constexpr auto SensitiveProperty = core::PropertyDefinitionBuilder<>::createProperty("Sensitive Property")
- .withDescription("Sensitive property")
- .isSensitive(true)
- .build();
- static constexpr auto Properties = std::to_array({SimpleProperty, SensitiveProperty});
- static constexpr auto Relationships = std::array{};
- static constexpr bool SupportsDynamicProperties = true;
- static constexpr bool SupportsDynamicRelationships = true;
- static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_ALLOWED;
- static constexpr bool IsSingleThreaded = false;
- ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS
-
- void initialize() override { setSupportedProperties(Properties); }
-};
-
-REGISTER_RESOURCE(DummyFlowYamlProcessor, Processor);
-
TEST_CASE("Cannot use non-sensitive parameter in sensitive property", "[YamlConfiguration]") {
ConfigurationTestController test_controller;
core::YamlConfiguration yaml_config(test_controller.getContext());
@@ -1307,7 +1283,7 @@ Parameter Contexts:
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
name: TailFile
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1349,7 +1325,7 @@ Parameter Contexts:
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
name: TailFile
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1531,7 +1507,7 @@ Flow Controller:
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
name: TailFile
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1555,7 +1531,7 @@ Flow Controller:
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
name: TailFile
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1594,7 +1570,7 @@ Parameter Contexts:
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
name: DummyProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1643,7 +1619,7 @@ Parameter Contexts:
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
name: DummyProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1695,8 +1671,8 @@ Parameter Contexts:
value: {}
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
- name: DummyFlowYamlProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1709,7 +1685,7 @@ Parameter Context Name: my-context
std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
REQUIRE(flow);
- auto* proc = flow->findProcessorByName("DummyFlowYamlProcessor");
+ auto* proc = flow->findProcessorByName("DummyProcessor");
REQUIRE(proc);
REQUIRE(proc->getProperty("Simple Property") == "simple");
REQUIRE(proc->getProperty("Sensitive Property") == "value1");
@@ -1744,8 +1720,8 @@ Parameter Contexts:
value: {}
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
- name: DummyFlowYamlProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1760,7 +1736,7 @@ Parameter Context Name: my-context
std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
REQUIRE(flow);
- auto* proc = flow->findProcessorByName("DummyFlowYamlProcessor");
+ auto* proc = flow->findProcessorByName("DummyProcessor");
core::Property property("Sensitive Property", "");
proc->getProperty("Sensitive Property", property);
auto values = property.getValues();
@@ -1801,8 +1777,8 @@ Parameter Contexts:
value: old_value
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
- name: DummyFlowYamlProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1815,7 +1791,7 @@ Parameter Context Name: inherited-context
std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
REQUIRE(flow);
- auto* proc = flow->findProcessorByName("DummyFlowYamlProcessor");
+ auto* proc = flow->findProcessorByName("DummyProcessor");
REQUIRE(proc);
REQUIRE(proc->getProperty("Simple Property") == "old_value");
REQUIRE(proc->getProperty("Sensitive Property") == "value1");
@@ -1843,8 +1819,8 @@ Parameter Contexts:
- base-context
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
- name: DummyFlowYamlProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1878,8 +1854,8 @@ Parameter Contexts:
Inherited Parameter Contexts: [unknown]
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
- name: DummyFlowYamlProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -1944,8 +1920,8 @@ Parameter Contexts:
Inherited Parameter Contexts: []
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
- name: DummyFlowYamlProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -2007,8 +1983,8 @@ Parameter Contexts:
- a-context
Processors:
- id: b0c04f28-0158-1000-0000-000000000000
- name: DummyFlowYamlProcessor
- class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
@@ -2022,7 +1998,7 @@ Parameter Context Name: c-context
std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
REQUIRE(flow);
- auto* proc = flow->findProcessorByName("DummyFlowYamlProcessor");
+ auto* proc = flow->findProcessorByName("DummyProcessor");
REQUIRE(proc);
std::string value;
REQUIRE(proc->getDynamicProperty("My A Property", value));
@@ -2033,4 +2009,224 @@ Parameter Context Name: c-context
CHECK(value == "5");
}
+TEST_CASE("Parameter providers can be used for parameter values", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flow
+Parameter Providers:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: DummyParameterProvider
+ type: DummyParameterProvider
+ Properties:
+ Dummy1 Value: value1
+ Dummy2 Value: value2
+ Dummy3 Value: value3
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: "#{dummy1}"
+ My Dynamic Property Sequence:
+ - value: "#{dummy2}"
+ - value: "#{dummy3}"
+Parameter Context Name: dummycontext
+ )";
+
+ std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("DummyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Simple Property") == "value1");
+ core::Property property("My Dynamic Property Sequence", "");
+ proc->getDynamicProperty("My Dynamic Property Sequence", property);
+ auto values = property.getValues();
+ REQUIRE(values.size() == 2);
+ CHECK(values[0] == "value2");
+ CHECK(values[1] == "value3");
+}
+
+TEST_CASE("Parameter providers can be configured to select which parameters to be sensitive", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ auto context = test_controller.getContext();
+ auto encrypted_sensitive_property_value = minifi::utils::crypto::property_encryption::encrypt("#{dummy1}", *context.sensitive_values_encryptor);
+ core::YamlConfiguration yaml_config(context);
+
+ static const std::string TEST_CONFIG_YAML =
+ fmt::format(R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Providers:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: DummyParameterProvider
+ type: DummyParameterProvider
+ Properties:
+ Sensitive Parameter Scope: selected
+ Sensitive Parameter List: dummy1
+ Dummy1 Value: value1
+ Dummy3 Value: value3
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: "#{{dummy3}}"
+ Sensitive Property: {}
+Parameter Context Name: dummycontext
+ )", encrypted_sensitive_property_value);
+
+ std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("DummyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Sensitive Property") == "value1");
+ REQUIRE(proc->getProperty("Simple Property") == "value3");
+}
+
+TEST_CASE("Parameter providers can be configured to make all parameters sensitive", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ auto context = test_controller.getContext();
+ auto encrypted_sensitive_property_value = minifi::utils::crypto::property_encryption::encrypt("#{dummy1}", *context.sensitive_values_encryptor);
+ core::YamlConfiguration yaml_config(context);
+
+ static const std::string TEST_CONFIG_YAML =
+ fmt::format(R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Providers:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: DummyParameterProvider
+ type: DummyParameterProvider
+ Properties:
+ Sensitive Parameter Scope: all
+ Dummy1 Value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Sensitive Property: {}
+Parameter Context Name: dummycontext
+ )", encrypted_sensitive_property_value);
+
+ std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("DummyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Sensitive Property") == "value1");
+}
+
+TEST_CASE("Parameter context can be inherited from parameter provider generated parameter context", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flow
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: DummyParameterProvider
+ type: DummyParameterProvider
+ Properties:
+ Dummy1 Value: value1
+ Dummy2 Value: value2
+ Dummy3 Value: value3
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: file_size
+ description: ''
+ sensitive: false
+ value: 10 B
+ Inherited Parameter Contexts: [dummycontext]
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: "#{dummy1}"
+Parameter Context Name: my-context
+ )";
+
+ std::unique_ptr flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+
+ auto* proc = flow->findProcessorByName("DummyProcessor");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Simple Property") == "value1");
+}
+
+TEST_CASE("Parameter context names cannot conflict with parameter provider generated parameter context names", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flow
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: DummyParameterProvider
+ type: DummyParameterProvider
+ Properties:
+ Dummy1 Value: value1
+ Dummy2 Value: value2
+ Dummy3 Value: value3
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: dummycontext
+ description: my parameter context
+ Parameters:
+ - name: file_size
+ description: ''
+ sensitive: false
+ value: 10 B
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: DummyProcessor
+ class: org.apache.nifi.processors.DummyProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: "#{dummy1}"
+Parameter Context Name: dummycontext
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter provider 'DummyParameterProvider' cannot create parameter context 'dummycontext' because parameter context already "
+ "exists with no parameter provider or generated by other parameter provider");
+}
+
} // namespace org::apache::nifi::minifi::test
diff --git a/libminifi/CMakeLists.txt b/libminifi/CMakeLists.txt
index 73a3fc2628..2649f44455 100644
--- a/libminifi/CMakeLists.txt
+++ b/libminifi/CMakeLists.txt
@@ -42,7 +42,7 @@ endif()
set(TLS_SOURCES "src/utils/tls/*.cpp" "src/io/tls/*.cpp")
-file(GLOB SOURCES "src/agent/*.cpp" "src/properties/*.cpp" "src/utils/file/*.cpp" "src/sitetosite/*.cpp" "src/core/logging/*.cpp" "src/core/logging/internal/*.cpp" "src/core/logging/alert/*.cpp" "src/core/state/*.cpp" "src/core/state/nodes/*.cpp" "src/c2/protocols/*.cpp" "src/c2/triggers/*.cpp" "src/c2/*.cpp" "src/io/*.cpp" ${SOCKET_SOURCES} ${TLS_SOURCES} "src/core/controller/*.cpp" "src/controllers/*.cpp" "src/controllers/keyvalue/*.cpp" "src/core/*.cpp" "src/core/repository/*.cpp" "src/core/flow/*.cpp" "src/core/json/*.cpp" "src/core/yaml/*.cpp" "src/core/reporting/*.cpp" "src/core/extension/*.cpp" "src/serialization/*.cpp" "src/provenance/*.cpp" "src/utils/*.cpp" "src/utils/crypto/*.cpp" "src/utils/crypto/ciphers/*.cpp" "src/utils/crypto/property_encryption/*.cpp" "src/*.cpp" "src/utils/net/*.cpp" "src/http/*.cpp")
+file(GLOB SOURCES "src/agent/*.cpp" "src/properties/*.cpp" "src/utils/file/*.cpp" "src/sitetosite/*.cpp" "src/core/logging/*.cpp" "src/core/logging/internal/*.cpp" "src/core/logging/alert/*.cpp" "src/core/state/*.cpp" "src/core/state/nodes/*.cpp" "src/c2/protocols/*.cpp" "src/c2/triggers/*.cpp" "src/c2/*.cpp" "src/io/*.cpp" ${SOCKET_SOURCES} ${TLS_SOURCES} "src/core/controller/*.cpp" "src/controllers/*.cpp" "src/controllers/keyvalue/*.cpp" "src/core/*.cpp" "src/core/repository/*.cpp" "src/core/flow/*.cpp" "src/core/json/*.cpp" "src/core/yaml/*.cpp" "src/core/reporting/*.cpp" "src/core/extension/*.cpp" "src/serialization/*.cpp" "src/provenance/*.cpp" "src/utils/*.cpp" "src/utils/crypto/*.cpp" "src/utils/crypto/ciphers/*.cpp" "src/utils/crypto/property_encryption/*.cpp" "src/*.cpp" "src/utils/net/*.cpp" "src/http/*.cpp" "src/parameter-providers/*.cpp")
# manually add this as it might not yet be present when this executes
list(APPEND SOURCES "${CMAKE_CURRENT_BINARY_DIR}/agent_version.cpp")
diff --git a/libminifi/include/agent/agent_docs.h b/libminifi/include/agent/agent_docs.h
index a234478c35..0be4aaab5e 100644
--- a/libminifi/include/agent/agent_docs.h
+++ b/libminifi/include/agent/agent_docs.h
@@ -34,7 +34,7 @@
namespace org::apache::nifi::minifi {
enum class ResourceType {
- Processor, ControllerService, InternalResource, DescriptionOnly
+ Processor, ControllerService, InternalResource, DescriptionOnly, ParameterProvider
};
struct ClassDescription {
diff --git a/libminifi/include/core/FlowConfiguration.h b/libminifi/include/core/FlowConfiguration.h
index c73f0d9347..5289013f5c 100644
--- a/libminifi/include/core/FlowConfiguration.h
+++ b/libminifi/include/core/FlowConfiguration.h
@@ -42,6 +42,7 @@
#include "utils/file/FileSystem.h"
#include "utils/ChecksumCalculator.h"
#include "ParameterContext.h"
+#include "ParameterProvider.h"
namespace org::apache::nifi::minifi::core {
@@ -95,6 +96,8 @@ class FlowConfiguration : public CoreComponent {
// Create Provenance Report Task
std::unique_ptr createProvenanceReportTask();
+ static std::unique_ptr createParameterProvider(const std::string &class_name, const std::string &full_class_name, const utils::Identifier& uuid);
+
[[nodiscard]] std::shared_ptr getFlowVersion() const {
return flow_version_;
}
@@ -140,6 +143,7 @@ class FlowConfiguration : public CoreComponent {
protected:
std::unordered_map>> parameter_contexts_;
+ std::vector>> parameter_providers_;
std::optional config_path_;
std::shared_ptr flow_file_repo_;
std::shared_ptr content_repo_;
diff --git a/libminifi/include/core/ParameterContext.h b/libminifi/include/core/ParameterContext.h
index c7a11b5930..64ce25c5e7 100644
--- a/libminifi/include/core/ParameterContext.h
+++ b/libminifi/include/core/ParameterContext.h
@@ -36,6 +36,7 @@ struct Parameter {
std::string name;
std::string description;
bool sensitive = false;
+ bool provided = false;
std::string value;
};
@@ -66,10 +67,19 @@ class ParameterContext : public CoreComponent {
return inherited_parameter_contexts_;
}
+ void setParameterProvider(const std::string ¶meter_provider) {
+ parameter_provider_ = parameter_provider;
+ }
+
+ std::string getParameterProvider() const {
+ return parameter_provider_;
+ }
+
private:
std::string description_;
std::unordered_map parameters_;
std::vector> inherited_parameter_contexts_;
+ std::string parameter_provider_;
};
} // namespace org::apache::nifi::minifi::core
diff --git a/libminifi/include/core/ParameterProvider.h b/libminifi/include/core/ParameterProvider.h
new file mode 100644
index 0000000000..9dd78ab717
--- /dev/null
+++ b/libminifi/include/core/ParameterProvider.h
@@ -0,0 +1,78 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "core/Core.h"
+#include "core/ConfigurableComponent.h"
+#include "core/ParameterContext.h"
+#include "core/PropertyDefinitionBuilder.h"
+
+namespace org::apache::nifi::minifi::core {
+
+struct ParameterGroup {
+ std::string name;
+ std::unordered_map parameters;
+};
+
+enum class SensitiveParameterScopeOptions {
+ none,
+ all,
+ selected
+};
+
+struct ParameterProviderConfig {
+ SensitiveParameterScopeOptions sensitive_parameter_scope = SensitiveParameterScopeOptions::none;
+ std::unordered_set sensitive_parameters;
+};
+
+class ParameterProvider : public ConfigurableComponent, public CoreComponent {
+ public:
+ using CoreComponent::CoreComponent;
+ ParameterProvider(const ParameterProvider &other) = delete;
+ ParameterProvider &operator=(const ParameterProvider &other) = delete;
+
+ MINIFIAPI static constexpr auto SensitiveParameterScope = core::PropertyDefinitionBuilder()>::createProperty("Sensitive Parameter Scope")
+ .withDescription("Define which parameters are considered sensitive, being either 'none', 'all' or 'selected'. If 'selected' is chosen, the 'Sensitive Parameter List' property must be set.")
+ .isRequired(true)
+ .withDefaultValue(magic_enum::enum_name(SensitiveParameterScopeOptions::none))
+ .withAllowedValues(magic_enum::enum_names())
+ .build();
+ MINIFIAPI static constexpr auto SensitiveParameterList = core::PropertyDefinitionBuilder<>::createProperty("Sensitive Parameter List")
+ .withDescription("List of sensitive parameters, if 'Sensitive Parameter Scope' is set to 'selected'.")
+ .build();
+
+ MINIFIAPI static constexpr auto Properties = std::to_array({SensitiveParameterScope, SensitiveParameterList});
+
+ bool supportsDynamicProperties() const override { return false; }
+ bool supportsDynamicRelationships() const override { return false; };
+
+ std::vector>> createParameterContexts();
+
+ protected:
+ ParameterProviderConfig readParameterProviderConfig() const;
+ virtual std::vector buildParameterGroups() = 0;
+
+ bool canEdit() override { return true; }
+};
+
+} // namespace org::apache::nifi::minifi::core
diff --git a/libminifi/include/core/flow/FlowSchema.h b/libminifi/include/core/flow/FlowSchema.h
index 9860cb86e8..e751d7bff7 100644
--- a/libminifi/include/core/flow/FlowSchema.h
+++ b/libminifi/include/core/flow/FlowSchema.h
@@ -86,7 +86,11 @@ struct FlowSchema {
Keys value;
Keys parameter_context_name;
Keys sensitive;
+ Keys provided;
Keys inherited_parameter_contexts;
+ Keys parameter_providers;
+ Keys parameter_provider_properties;
+ Keys parameter_provider;
static FlowSchema getDefault();
static FlowSchema getNiFiFlowJson();
diff --git a/libminifi/include/core/flow/FlowSerializer.h b/libminifi/include/core/flow/FlowSerializer.h
index 05f374fe8c..19c365c32a 100644
--- a/libminifi/include/core/flow/FlowSerializer.h
+++ b/libminifi/include/core/flow/FlowSerializer.h
@@ -23,6 +23,7 @@
#include "core/ProcessGroup.h"
#include "utils/crypto/EncryptionProvider.h"
#include "utils/Id.h"
+#include "core/ParameterContext.h"
namespace org::apache::nifi::minifi::core::flow {
@@ -52,7 +53,7 @@ class FlowSerializer {
FlowSerializer& operator=(FlowSerializer&&) = delete;
[[nodiscard]] virtual std::string serialize(const core::ProcessGroup& process_group, const FlowSchema& schema, const utils::crypto::EncryptionProvider& encryption_provider,
- const std::unordered_map& overrides) const = 0;
+ const std::unordered_map& overrides, const std::unordered_map>>& parameter_contexts) const = 0;
};
} // namespace org::apache::nifi::minifi::core::flow
diff --git a/libminifi/include/core/flow/StructuredConfiguration.h b/libminifi/include/core/flow/StructuredConfiguration.h
index 09fd6b3f58..1536366337 100644
--- a/libminifi/include/core/flow/StructuredConfiguration.h
+++ b/libminifi/include/core/flow/StructuredConfiguration.h
@@ -108,7 +108,7 @@ class StructuredConfiguration : public FlowConfiguration {
*/
std::unique_ptr parseRootProcessGroup(const Node& root_flow_node);
- void parseParameterContexts(const Node& parameter_contexts_node);
+ void parseParameterContexts(const Node& parameter_contexts_node, const Node& parameter_providers_node);
void parseControllerServices(const Node& controller_services_node);
/**
@@ -234,6 +234,9 @@ class StructuredConfiguration : public FlowConfiguration {
* @param reason
*/
void raiseComponentError(const std::string &component_name, const std::string §ion, const std::string &reason) const;
+ void parseParameterProvidersNode(const Node& parameter_providers_node);
+ void parseParameterContextsNode(const Node& parameter_contexts_node);
+ void parseParameterContextInheritance(const Node& parameter_contexts_node);
void verifyNoInheritanceCycles() const;
};
diff --git a/libminifi/include/core/json/JsonFlowSerializer.h b/libminifi/include/core/json/JsonFlowSerializer.h
index b5042ed54c..8dfe0430f4 100644
--- a/libminifi/include/core/json/JsonFlowSerializer.h
+++ b/libminifi/include/core/json/JsonFlowSerializer.h
@@ -25,10 +25,14 @@ class JsonFlowSerializer : public core::flow::FlowSerializer {
public:
explicit JsonFlowSerializer(rapidjson::Document document) : flow_definition_json_(std::move(document)) {}
- [[nodiscard]] std::string serialize(const core::ProcessGroup& process_group, const core::flow::FlowSchema& schema, const utils::crypto::EncryptionProvider& encryption_provider,
- const std::unordered_map& overrides) const override;
+ [[nodiscard]] std::string serialize(const core::ProcessGroup& process_group, const core::flow::FlowSchema& schema,
+ const utils::crypto::EncryptionProvider& encryption_provider,
+ const std::unordered_map& overrides,
+ const std::unordered_map>>& parameter_contexts) const override;
private:
+ void addProviderCreatedParameterContexts(rapidjson::Value& flow_definition_json, rapidjson::Document::AllocatorType& alloc, const core::flow::FlowSchema& schema,
+ const std::unordered_map>>& parameter_contexts) const;
void encryptSensitiveParameters(rapidjson::Value& flow_definition_json, rapidjson::Document::AllocatorType& alloc, const core::flow::FlowSchema& schema,
const utils::crypto::EncryptionProvider& encryption_provider, const std::unordered_map& overrides) const;
void encryptSensitiveProcessorProperties(rapidjson::Value& root_group, rapidjson::Document::AllocatorType& alloc, const core::ProcessGroup& process_group,
diff --git a/libminifi/include/core/yaml/YamlFlowSerializer.h b/libminifi/include/core/yaml/YamlFlowSerializer.h
index 240d112c53..ebf5b2ff00 100644
--- a/libminifi/include/core/yaml/YamlFlowSerializer.h
+++ b/libminifi/include/core/yaml/YamlFlowSerializer.h
@@ -26,9 +26,12 @@ class YamlFlowSerializer : public core::flow::FlowSerializer {
explicit YamlFlowSerializer(const YAML::Node& flow_definition_yaml) : flow_definition_yaml_(flow_definition_yaml) {}
[[nodiscard]] std::string serialize(const core::ProcessGroup& process_group, const core::flow::FlowSchema& schema, const utils::crypto::EncryptionProvider& encryption_provider,
- const std::unordered_map& overrides) const override;
+ const std::unordered_map& overrides,
+ const std::unordered_map>>& parameter_contexts) const override;
private:
+ void addProviderCreatedParameterContexts(YAML::Node flow_definition_yaml, const core::flow::FlowSchema& schema,
+ const std::unordered_map>>& parameter_contexts) const;
void encryptSensitiveParameters(YAML::Node& flow_definition_yaml, const core::flow::FlowSchema& schema, const utils::crypto::EncryptionProvider& encryption_provider,
const std::unordered_map& overrides) const;
void encryptSensitiveProcessorProperties(YAML::Node& flow_definition_yaml, const core::ProcessGroup& process_group, const core::flow::FlowSchema& schema,
diff --git a/libminifi/include/parameter-providers/EnvironmentVariableParameterProvider.h b/libminifi/include/parameter-providers/EnvironmentVariableParameterProvider.h
new file mode 100644
index 0000000000..29af6947fe
--- /dev/null
+++ b/libminifi/include/parameter-providers/EnvironmentVariableParameterProvider.h
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include "core/ParameterProvider.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org::apache::nifi::minifi::parameter_providers {
+
+enum class EnvironmentVariableInclusionStrategyOptions {
+ include_all,
+ comma_separated,
+ regular_expression
+};
+
+} // namespace org::apache::nifi::minifi::parameter_providers
+
+namespace magic_enum::customize {
+using EnvironmentVariableInclusionStrategyOptions = org::apache::nifi::minifi::parameter_providers::EnvironmentVariableInclusionStrategyOptions;
+
+template <>
+constexpr customize_t enum_name(EnvironmentVariableInclusionStrategyOptions value) noexcept {
+ switch (value) {
+ case EnvironmentVariableInclusionStrategyOptions::include_all:
+ return "Include All";
+ case EnvironmentVariableInclusionStrategyOptions::comma_separated:
+ return "Comma-Separated";
+ case EnvironmentVariableInclusionStrategyOptions::regular_expression:
+ return "Regular Expression";
+ }
+ return invalid_tag;
+}
+} // namespace magic_enum::customize
+
+namespace org::apache::nifi::minifi::parameter_providers {
+
+class EnvironmentVariableParameterProvider final : public core::ParameterProvider {
+ public:
+ using ParameterProvider::ParameterProvider;
+
+ MINIFIAPI static constexpr const char* Description = "Fetches parameters from environment variables";
+
+ MINIFIAPI static constexpr auto EnvironmentVariableInclusionStrategy =
+ core::PropertyDefinitionBuilder()>::createProperty("Environment Variable Inclusion Strategy")
+ .withDescription("Indicates how Environment Variables should be included")
+ .isRequired(true)
+ .withDefaultValue(magic_enum::enum_name(EnvironmentVariableInclusionStrategyOptions::include_all))
+ .withAllowedValues(magic_enum::enum_names())
+ .build();
+ MINIFIAPI static constexpr auto IncludeEnvironmentVariables = core::PropertyDefinitionBuilder<>::createProperty("Include Environment Variables")
+ .withDescription("Specifies comma separated environment variable names or regular expression (depending on the Environment Variable Inclusion Strategy) "
+ "that should be used to fetch environment variables.")
+ .build();
+ MINIFIAPI static constexpr auto ParameterGroupName = core::PropertyDefinitionBuilder<>::createProperty("Parameter Group Name")
+ .withDescription("The name of the parameter group that will be fetched. This indicates the name of the Parameter Context that may receive the fetched parameters.")
+ .isRequired(true)
+ .build();
+
+ void initialize() override {
+ setSupportedProperties(minifi::utils::array_cat(core::ParameterProvider::Properties,
+ std::to_array({EnvironmentVariableInclusionStrategy, IncludeEnvironmentVariables, ParameterGroupName})));
+ }
+
+ private:
+ std::string readParameterGroupName() const;
+ EnvironmentVariableInclusionStrategyOptions readEnvironmentVariableInclusionStrategy() const;
+ void filterEnvironmentVariablesByCommaSeparatedList(std::unordered_map& environment_variables) const;
+ void filterEnvironmentVariablesByRegularExpression(std::unordered_map& environment_variables) const;
+ std::vector buildParameterGroups() override;
+
+ std::shared_ptr logger_{core::logging::LoggerFactory::getLogger(uuid_)};
+};
+
+} // namespace org::apache::nifi::minifi::parameter_providers
diff --git a/libminifi/include/utils/Environment.h b/libminifi/include/utils/Environment.h
index 262df02ab7..6f47f27ddd 100644
--- a/libminifi/include/utils/Environment.h
+++ b/libminifi/include/utils/Environment.h
@@ -15,8 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#ifndef LIBMINIFI_INCLUDE_UTILS_ENVIRONMENT_H_
-#define LIBMINIFI_INCLUDE_UTILS_ENVIRONMENT_H_
+#pragma once
#include
#include
@@ -31,11 +30,6 @@ namespace org::apache::nifi::minifi::utils {
* functions are called natively, and not through this class), then this class can't guarantee thread safety.
*/
class Environment {
- private:
- static bool runningAsService_;
-
- static void accessEnvironment(const std::function& func);
-
public:
/**
* Gets an environment variable using the native OS API
@@ -72,8 +66,13 @@ class Environment {
* @return true if the current process is running as a service
*/
static bool isRunningAsService();
+
+ static std::unordered_map getEnvironmentVariables();
+
+ private:
+ static void accessEnvironment(const std::function& func);
+
+ static bool runningAsService_;
};
} // namespace org::apache::nifi::minifi::utils
-
-#endif // LIBMINIFI_INCLUDE_UTILS_ENVIRONMENT_H_
diff --git a/libminifi/src/core/FlowConfiguration.cpp b/libminifi/src/core/FlowConfiguration.cpp
index fd46a064a1..6925d40f55 100644
--- a/libminifi/src/core/FlowConfiguration.cpp
+++ b/libminifi/src/core/FlowConfiguration.cpp
@@ -177,4 +177,22 @@ std::shared_ptr FlowConfiguration::crea
return controllerServicesNode;
}
+std::unique_ptr FlowConfiguration::createParameterProvider(const std::string &class_name, const std::string &full_class_name, const utils::Identifier& uuid) {
+ auto ptr = core::ClassLoader::getDefaultClassLoader().instantiate(class_name, uuid);
+ if (ptr == nullptr) {
+ ptr = core::ClassLoader::getDefaultClassLoader().instantiate(full_class_name, uuid);
+ }
+ if (ptr == nullptr) {
+ return nullptr;
+ }
+
+ auto returnPtr = utils::dynamic_unique_cast(std::move(ptr));
+ if (!returnPtr) {
+ throw std::runtime_error("Invalid return from the classloader");
+ }
+
+ returnPtr->initialize();
+ return returnPtr;
+}
+
} // namespace org::apache::nifi::minifi::core
diff --git a/libminifi/src/core/ParameterProvider.cpp b/libminifi/src/core/ParameterProvider.cpp
new file mode 100644
index 0000000000..01cfd7aaa0
--- /dev/null
+++ b/libminifi/src/core/ParameterProvider.cpp
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterProvider.h"
+
+namespace org::apache::nifi::minifi::core {
+
+ParameterProviderConfig ParameterProvider::readParameterProviderConfig() const {
+ ParameterProviderConfig config;
+
+ auto sensitive_parameter_scope_str = getProperty(SensitiveParameterScope);
+ if (!sensitive_parameter_scope_str) {
+ throw ParameterException("Sensitive Parameter Scope is required");
+ }
+ auto sensitive_parameter_scope = magic_enum::enum_cast(*sensitive_parameter_scope_str);
+ if (!sensitive_parameter_scope) {
+ throw ParameterException("Sensitive Parameter Scope has invalid value: '" + *sensitive_parameter_scope_str + "'");
+ }
+ config.sensitive_parameter_scope = sensitive_parameter_scope.value();
+
+ if (config.sensitive_parameter_scope == SensitiveParameterScopeOptions::selected) {
+ if (auto sensitive_parameter_list = getProperty(SensitiveParameterList)) {
+ for (const auto& sensitive_parameter : minifi::utils::string::split(*sensitive_parameter_list, ",")) {
+ config.sensitive_parameters.insert(sensitive_parameter);
+ }
+ }
+ }
+
+ return config;
+}
+
+std::vector>> ParameterProvider::createParameterContexts() {
+ auto config = readParameterProviderConfig();
+
+ auto parameter_groups = buildParameterGroups();
+ std::vector>> result;
+ for (const auto& parameter_group : parameter_groups) {
+ auto parameter_context = std::make_unique(parameter_group.name);
+ parameter_context->setParameterProvider(getUUIDStr());
+ for (const auto& [name, value] : parameter_group.parameters) {
+ bool sensitive = config.sensitive_parameter_scope == SensitiveParameterScopeOptions::all ||
+ (config.sensitive_parameter_scope == SensitiveParameterScopeOptions::selected && config.sensitive_parameters.contains(name));
+ parameter_context->addParameter(Parameter{
+ .name = name,
+ .description = "",
+ .sensitive = sensitive,
+ .provided = true,
+ .value = value
+ });
+ }
+ result.push_back(gsl::make_not_null(std::move(parameter_context)));
+ }
+
+ return result;
+}
+
+} // namespace org::apache::nifi::minifi::core
diff --git a/libminifi/src/core/flow/AdaptiveConfiguration.cpp b/libminifi/src/core/flow/AdaptiveConfiguration.cpp
index 3431e92235..194963bf11 100644
--- a/libminifi/src/core/flow/AdaptiveConfiguration.cpp
+++ b/libminifi/src/core/flow/AdaptiveConfiguration.cpp
@@ -77,7 +77,7 @@ void AdaptiveConfiguration::setSensitivePropertiesEncryptor(utils::crypto::Encry
std::string AdaptiveConfiguration::serializeWithOverrides(const core::ProcessGroup& process_group, const std::unordered_map& overrides) const {
gsl_Expects(flow_serializer_);
- return flow_serializer_->serialize(process_group, schema_, sensitive_values_encryptor_, overrides);
+ return flow_serializer_->serialize(process_group, schema_, sensitive_values_encryptor_, overrides, parameter_contexts_);
}
} // namespace org::apache::nifi::minifi::core::flow
diff --git a/libminifi/src/core/flow/FlowSchema.cpp b/libminifi/src/core/flow/FlowSchema.cpp
index b58edc6982..812b578f02 100644
--- a/libminifi/src/core/flow/FlowSchema.cpp
+++ b/libminifi/src/core/flow/FlowSchema.cpp
@@ -83,7 +83,11 @@ FlowSchema FlowSchema::getDefault() {
.value = {"value"},
.parameter_context_name = {"Parameter Context Name"},
.sensitive = {"sensitive"},
- .inherited_parameter_contexts = {"Inherited Parameter Contexts"}
+ .provided = {"provided"},
+ .inherited_parameter_contexts = {"Inherited Parameter Contexts"},
+ .parameter_providers = {"Parameter Providers"},
+ .parameter_provider_properties = {"Properties"},
+ .parameter_provider{"Parameter Provider"}
};
}
@@ -151,7 +155,11 @@ FlowSchema FlowSchema::getNiFiFlowJson() {
.value = {"value"},
.parameter_context_name = {"parameterContextName"},
.sensitive = {"sensitive"},
- .inherited_parameter_contexts = {"inheritedParameterContexts"}
+ .provided = {"provided"},
+ .inherited_parameter_contexts = {"inheritedParameterContexts"},
+ .parameter_providers = {"parameterProviders"},
+ .parameter_provider_properties = {"properties"},
+ .parameter_provider{"parameterProvider"}
};
}
diff --git a/libminifi/src/core/flow/StructuredConfiguration.cpp b/libminifi/src/core/flow/StructuredConfiguration.cpp
index 4ec53a416e..9c612972f7 100644
--- a/libminifi/src/core/flow/StructuredConfiguration.cpp
+++ b/libminifi/src/core/flow/StructuredConfiguration.cpp
@@ -121,10 +121,11 @@ std::unique_ptr StructuredConfiguration::getRootFrom(const N
schema_ = std::move(schema);
uuids_.clear();
Node parameterContextsNode = root_node[schema_.parameter_contexts];
+ Node parameterProvidersNode = root_node[schema_.parameter_providers];
Node controllerServiceNode = root_node[schema_.root_group][schema_.controller_services];
Node provenanceReportNode = root_node[schema_.provenance_reporting];
- parseParameterContexts(parameterContextsNode);
+ parseParameterContexts(parameterContextsNode, parameterProvidersNode);
parseControllerServices(controllerServiceNode);
// Create the root process group
std::unique_ptr root = parseRootProcessGroup(root_node);
@@ -180,7 +181,7 @@ void StructuredConfiguration::verifyNoInheritanceCycles() const {
}
}
-void StructuredConfiguration::parseParameterContexts(const Node& parameter_contexts_node) {
+void StructuredConfiguration::parseParameterContextsNode(const Node& parameter_contexts_node) {
if (!parameter_contexts_node || !parameter_contexts_node.isSequence()) {
return;
}
@@ -188,7 +189,7 @@ void StructuredConfiguration::parseParameterContexts(const Node& parameter_conte
checkRequiredField(parameter_context_node, schema_.name);
auto name = parameter_context_node[schema_.name].getString().value();
- if (parameter_contexts_.find(name) != parameter_contexts_.end()) {
+ if (parameter_contexts_.contains(name)) {
throw std::invalid_argument("Parameter context name '" + name + "' already exists, parameter context names must be unique!");
}
auto id = getRequiredIdField(parameter_context_node);
@@ -197,6 +198,7 @@ void StructuredConfiguration::parseParameterContexts(const Node& parameter_conte
uuid = id;
auto parameter_context = std::make_unique(name, uuid);
parameter_context->setDescription(getOptionalField(parameter_context_node, schema_.description, ""));
+ parameter_context->setParameterProvider(getOptionalField(parameter_context_node, schema_.parameter_provider, ""));
for (const auto& parameter_node : parameter_context_node[schema_.parameters]) {
checkRequiredField(parameter_node, schema_.name);
checkRequiredField(parameter_node, schema_.value);
@@ -204,16 +206,72 @@ void StructuredConfiguration::parseParameterContexts(const Node& parameter_conte
auto parameter_name = parameter_node[schema_.name].getString().value();
auto parameter_value = parameter_node[schema_.value].getString().value();
auto sensitive = parameter_node[schema_.sensitive].getBool().value();
+ auto provided = parameter_node[schema_.provided].getBool().value_or(false);
auto parameter_description = getOptionalField(parameter_node, schema_.description, "");
if (sensitive) {
parameter_value = utils::crypto::property_encryption::decrypt(parameter_value, sensitive_values_encryptor_);
}
- parameter_context->addParameter(Parameter{parameter_name, parameter_description, sensitive, parameter_value});
+ parameter_context->addParameter(Parameter{
+ .name = parameter_name,
+ .description = parameter_description,
+ .sensitive = sensitive,
+ .provided = provided,
+ .value = parameter_value});
}
parameter_contexts_.emplace(name, gsl::make_not_null(std::move(parameter_context)));
}
+}
+
+void StructuredConfiguration::parseParameterProvidersNode(const Node& parameter_providers_node) {
+ if (!parameter_providers_node || !parameter_providers_node.isSequence()) {
+ return;
+ }
+ for (const auto& parameter_provider_node : parameter_providers_node) {
+ checkRequiredField(parameter_provider_node, schema_.name);
+ auto type = getRequiredField(parameter_provider_node, schema_.type);
+ logger_->log_debug("Using type {} for parameter provider node", type);
+
+ std::string fullType = type;
+ auto lastOfIdx = type.find_last_of('.');
+ if (lastOfIdx != std::string::npos) {
+ lastOfIdx++; // if a value is found, increment to move beyond the .
+ type = type.substr(lastOfIdx);
+ }
+
+ auto name = parameter_provider_node[schema_.name].getString().value();
+ auto id = getRequiredIdField(parameter_provider_node);
+
+ utils::Identifier uuid;
+ uuid = id;
+ auto parameter_provider = createParameterProvider(type, fullType, uuid);
+ if (nullptr != parameter_provider) {
+ logger_->log_debug("Created Parameter Provider with UUID {} and name {}", id, name);
+ if (Node propertiesNode = parameter_provider_node[schema_.parameter_provider_properties]) {
+ parsePropertiesNode(propertiesNode, *parameter_provider, name, nullptr);
+ }
+ } else {
+ logger_->log_debug("Could not locate {}", type);
+ }
+ parameter_provider->setName(name);
+ auto parameter_contexts = parameter_provider->createParameterContexts();
+ for (auto& parameter_context : parameter_contexts) {
+ if (!parameter_contexts_.contains(parameter_context->getName())) {
+ parameter_contexts_.emplace(parameter_context->getName(), std::move(parameter_context));
+ } else if (parameter_contexts_.at(parameter_context->getName())->getParameterProvider() != parameter_provider->getUUIDStr()) {
+ throw std::invalid_argument(fmt::format("Parameter provider '{}' cannot create parameter context '{}' because parameter context already exists "
+ "with no parameter provider or generated by other parameter provider", parameter_provider->getName(), parameter_context->getName()));
+ }
+ }
+ parameter_providers_.push_back(gsl::make_not_null(std::move(parameter_provider)));
+ }
+}
+
+void StructuredConfiguration::parseParameterContextInheritance(const Node& parameter_contexts_node) {
+ if (!parameter_contexts_node || !parameter_contexts_node.isSequence()) {
+ return;
+ }
for (const auto& parameter_context_node : parameter_contexts_node) {
if (!isFieldPresent(parameter_context_node, schema_.inherited_parameter_contexts[0])) {
continue;
@@ -236,6 +294,12 @@ void StructuredConfiguration::parseParameterContexts(const Node& parameter_conte
verifyNoInheritanceCycles();
}
+void StructuredConfiguration::parseParameterContexts(const Node& parameter_contexts_node, const Node& parameter_providers_node) {
+ parseParameterContextsNode(parameter_contexts_node);
+ parseParameterProvidersNode(parameter_providers_node);
+ parseParameterContextInheritance(parameter_contexts_node);
+}
+
void StructuredConfiguration::parseProcessorNode(const Node& processors_node, core::ProcessGroup* parentGroup) {
int64_t runDurationNanos = -1;
utils::Identifier uuid;
@@ -1067,7 +1131,7 @@ void StructuredConfiguration::addNewId(const std::string& uuid) {
std::string StructuredConfiguration::serialize(const core::ProcessGroup& process_group) {
gsl_Expects(flow_serializer_);
- return flow_serializer_->serialize(process_group, schema_, sensitive_values_encryptor_, {});
+ return flow_serializer_->serialize(process_group, schema_, sensitive_values_encryptor_, {}, parameter_contexts_);
}
} // namespace org::apache::nifi::minifi::core::flow
diff --git a/libminifi/src/core/json/JsonFlowSerializer.cpp b/libminifi/src/core/json/JsonFlowSerializer.cpp
index 08aeeda94b..7bb95110c5 100644
--- a/libminifi/src/core/json/JsonFlowSerializer.cpp
+++ b/libminifi/src/core/json/JsonFlowSerializer.cpp
@@ -39,6 +39,56 @@ rapidjson::Value& getMember(rapidjson::Value& node, const std::string& member_na
}
}
+void JsonFlowSerializer::addProviderCreatedParameterContexts(rapidjson::Value& flow_definition_json, rapidjson::Document::AllocatorType& alloc, const core::flow::FlowSchema& schema,
+ const std::unordered_map>>& parameter_contexts) const {
+ std::vector> provided_parameter_contexts;
+ for (const auto& [parameter_context_name, parameter_context] : parameter_contexts) {
+ if (!parameter_context->getParameterProvider().empty()) {
+ provided_parameter_contexts.push_back(gsl::make_not_null(parameter_context.get()));
+ }
+ }
+
+ if (provided_parameter_contexts.empty()) {
+ return;
+ }
+
+ std::unordered_set parameter_context_names;
+ if (flow_definition_json.HasMember(schema.parameter_contexts[0])) {
+ auto parameter_contexts_node = getMember(flow_definition_json, schema.parameter_contexts[0]).GetArray();
+ for (auto& parameter_context : parameter_contexts_node) {
+ parameter_context_names.insert(getMember(parameter_context, schema.name[0]).GetString());
+ }
+ } else {
+ flow_definition_json.AddMember(rapidjson::Value(schema.parameter_contexts[0].c_str(), alloc), rapidjson::Value(rapidjson::kArrayType), alloc);
+ }
+
+ for (const auto& parameter_context : provided_parameter_contexts) {
+ if (parameter_context_names.contains(parameter_context->getName())) {
+ logger_->log_warn("Parameter context '{}' already exists in the flow definition, will not be updated!", parameter_context->getName());
+ continue;
+ }
+ rapidjson::Value parameter_context_json(rapidjson::kObjectType);
+ parameter_context_json.AddMember(rapidjson::Value(schema.identifier[0], alloc), rapidjson::Value(parameter_context->getUUIDStr(), alloc), alloc);
+ parameter_context_json.AddMember(rapidjson::Value(schema.name[0], alloc), rapidjson::Value(parameter_context->getName(), alloc), alloc);
+ parameter_context_json.AddMember(rapidjson::Value(schema.parameter_provider[0], alloc), rapidjson::Value(parameter_context->getParameterProvider(), alloc), alloc);
+ auto parameters = parameter_context->getParameters();
+ rapidjson::Value parameters_json(rapidjson::kArrayType);
+ for (const auto& [name, parameter] : parameters) {
+ rapidjson::Value parameter_json(rapidjson::kObjectType);
+ parameter_json.AddMember(rapidjson::Value(schema.name[0], alloc), rapidjson::Value(name, alloc), alloc);
+ parameter_json.AddMember(rapidjson::Value(schema.description[0], alloc), rapidjson::Value(parameter.description, alloc), alloc);
+ parameter_json.AddMember(rapidjson::Value(schema.sensitive[0], alloc), rapidjson::Value(parameter.sensitive), alloc);
+ parameter_json.AddMember(rapidjson::Value(schema.provided[0], alloc), rapidjson::Value(parameter.provided), alloc);
+ parameter_json.AddMember(rapidjson::Value(schema.value[0], alloc), rapidjson::Value(parameter.value, alloc), alloc);
+ parameters_json.PushBack(parameter_json, alloc);
+ }
+ parameter_context_json.AddMember(rapidjson::Value(schema.parameters[0], alloc), parameters_json, alloc);
+
+ auto& parameter_contexts_node = getMember(flow_definition_json, schema.parameter_contexts[0]);
+ parameter_contexts_node.PushBack(parameter_context_json, alloc);
+ }
+}
+
void JsonFlowSerializer::encryptSensitiveProperties(rapidjson::Value& property_jsons, rapidjson::Document::AllocatorType& alloc,
const std::map& properties, const utils::crypto::EncryptionProvider& encryption_provider,
const core::flow::Overrides& overrides) const {
@@ -125,6 +175,9 @@ void JsonFlowSerializer::encryptSensitiveProcessorProperties(rapidjson::Value& r
void JsonFlowSerializer::encryptSensitiveControllerServiceProperties(rapidjson::Value& root_group, rapidjson::Document::AllocatorType& alloc, const core::ProcessGroup& process_group,
const core::flow::FlowSchema& schema, const utils::crypto::EncryptionProvider& encryption_provider, const std::unordered_map& overrides) const {
+ if (!root_group.HasMember(schema.controller_services[0])) {
+ return;
+ }
auto controller_services = getMember(root_group, schema.controller_services[0]).GetArray();
for (auto &controller_service_json : controller_services) {
const std::string controller_service_id_str{getMember(controller_service_json, schema.identifier[0]).GetString(), getMember(controller_service_json, schema.identifier[0]).GetStringLength()};
@@ -150,17 +203,18 @@ void JsonFlowSerializer::encryptSensitiveControllerServiceProperties(rapidjson::
}
std::string JsonFlowSerializer::serialize(const core::ProcessGroup& process_group, const core::flow::FlowSchema& schema, const utils::crypto::EncryptionProvider& encryption_provider,
- const std::unordered_map& overrides) const {
+ const std::unordered_map& overrides, const std::unordered_map>>& parameter_contexts) const {
gsl_Expects(schema.root_group.size() == 1 && schema.identifier.size() == 1 &&
schema.processors.size() == 1 && schema.processor_properties.size() == 1 &&
schema.controller_services.size() == 1 && schema.controller_service_properties.size() == 1);
rapidjson::Document doc;
- auto alloc = doc.GetAllocator();
+ auto& alloc = doc.GetAllocator();
rapidjson::Value flow_definition_json;
flow_definition_json.CopyFrom(flow_definition_json_, alloc);
auto& root_group = getMember(flow_definition_json, schema.root_group[0]);
+ addProviderCreatedParameterContexts(flow_definition_json, alloc, schema, parameter_contexts);
encryptSensitiveParameters(flow_definition_json, alloc, schema, encryption_provider, overrides);
encryptSensitiveProcessorProperties(root_group, alloc, process_group, schema, encryption_provider, overrides);
encryptSensitiveControllerServiceProperties(root_group, alloc, process_group, schema, encryption_provider, overrides);
diff --git a/libminifi/src/core/yaml/YamlFlowSerializer.cpp b/libminifi/src/core/yaml/YamlFlowSerializer.cpp
index 7ac4ceebd3..7fa252e360 100644
--- a/libminifi/src/core/yaml/YamlFlowSerializer.cpp
+++ b/libminifi/src/core/yaml/YamlFlowSerializer.cpp
@@ -23,6 +23,59 @@
namespace org::apache::nifi::minifi::core::yaml {
+void YamlFlowSerializer::addProviderCreatedParameterContexts(YAML::Node flow_definition_yaml, const core::flow::FlowSchema& schema,
+ const std::unordered_map>>& parameter_contexts) const {
+ std::vector> provided_parameter_contexts;
+ for (const auto& [parameter_context_name, parameter_context] : parameter_contexts) {
+ if (!parameter_context->getParameterProvider().empty()) {
+ provided_parameter_contexts.push_back(gsl::make_not_null(parameter_context.get()));
+ }
+ }
+
+ if (provided_parameter_contexts.empty()) {
+ return;
+ }
+
+ std::unordered_set parameter_context_names;
+ auto parameter_contexts_node = flow_definition_yaml[schema.parameter_contexts[0]];
+
+ if (parameter_contexts_node.IsDefined() && parameter_contexts_node.IsSequence()) {
+ for (const auto& parameter_context : parameter_contexts_node) {
+ parameter_context_names.insert(parameter_context[schema.name[0]].as());
+ }
+ } else {
+ parameter_contexts_node = YAML::Node(YAML::NodeType::Sequence);
+ }
+
+ for (const auto& parameter_context : provided_parameter_contexts) {
+ if (parameter_context_names.contains(parameter_context->getName())) {
+ logger_->log_warn("Parameter context '{}' already exists in the flow definition, will not be updated!", parameter_context->getName());
+ continue;
+ }
+
+ YAML::Node parameter_context_node;
+ parameter_context_node[schema.identifier[0]] = std::string(parameter_context->getUUIDStr());
+ parameter_context_node[schema.name[0]] = parameter_context->getName();
+ parameter_context_node[schema.parameter_provider[0]] = parameter_context->getParameterProvider();
+
+ YAML::Node parameters_node = YAML::Node(YAML::NodeType::Sequence);
+ auto parameters = parameter_context->getParameters();
+ for (const auto& [name, parameter] : parameters) {
+ YAML::Node parameter_node;
+ parameter_node[schema.name[0]] = name;
+ parameter_node[schema.description[0]] = parameter.description;
+ parameter_node[schema.sensitive[0]] = parameter.sensitive;
+ parameter_node[schema.provided[0]] = parameter.provided;
+ parameter_node[schema.value[0]] = parameter.value;
+
+ parameters_node.push_back(parameter_node);
+ }
+
+ parameter_context_node[schema.parameters[0]] = parameters_node;
+ parameter_contexts_node.push_back(parameter_context_node);
+ }
+}
+
void YamlFlowSerializer::encryptSensitiveProperties(YAML::Node property_yamls, const std::map& properties, const utils::crypto::EncryptionProvider& encryption_provider,
const core::flow::Overrides& overrides) const {
std::unordered_set processed_property_names;
@@ -128,13 +181,15 @@ void YamlFlowSerializer::encryptSensitiveControllerServiceProperties(YAML::Node&
}
std::string YamlFlowSerializer::serialize(const core::ProcessGroup& process_group, const core::flow::FlowSchema& schema, const utils::crypto::EncryptionProvider& encryption_provider,
- const std::unordered_map& overrides) const {
+ const std::unordered_map& overrides,
+ const std::unordered_map>>& parameter_contexts) const {
gsl_Expects(schema.identifier.size() == 1 &&
schema.processors.size() == 1 && schema.processor_properties.size() == 1 &&
schema.controller_services.size() == 1 && schema.controller_service_properties.size() == 1);
auto flow_definition_yaml = YAML::Clone(flow_definition_yaml_);
+ addProviderCreatedParameterContexts(flow_definition_yaml, schema, parameter_contexts);
encryptSensitiveParameters(flow_definition_yaml, schema, encryption_provider, overrides);
encryptSensitiveProcessorProperties(flow_definition_yaml, process_group, schema, encryption_provider, overrides);
encryptSensitiveControllerServiceProperties(flow_definition_yaml, process_group, schema, encryption_provider, overrides);
diff --git a/libminifi/src/parameter-providers/EnvironmentVariableParameterProvider.cpp b/libminifi/src/parameter-providers/EnvironmentVariableParameterProvider.cpp
new file mode 100644
index 0000000000..fe66431b7a
--- /dev/null
+++ b/libminifi/src/parameter-providers/EnvironmentVariableParameterProvider.cpp
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "parameter-providers/EnvironmentVariableParameterProvider.h"
+#include "core/Resource.h"
+#include "utils/Environment.h"
+#include "Exception.h"
+#include "range/v3/view/filter.hpp"
+#include "utils/RegexUtils.h"
+
+namespace org::apache::nifi::minifi::parameter_providers {
+
+std::string EnvironmentVariableParameterProvider::readParameterGroupName() const {
+ auto parameter_group_name = getProperty(ParameterGroupName);
+ if (!parameter_group_name || parameter_group_name.value().empty()) {
+ throw core::ParameterException("Parameter Group Name is required");
+ }
+ logger_->log_debug("Building Parameter Context with name: {}", parameter_group_name.value());
+ return parameter_group_name.value();
+}
+
+EnvironmentVariableInclusionStrategyOptions EnvironmentVariableParameterProvider::readEnvironmentVariableInclusionStrategy() const {
+ auto env_variable_inclusion_strategy_str = getProperty(EnvironmentVariableInclusionStrategy);
+ if (!env_variable_inclusion_strategy_str) {
+ throw core::ParameterException("Environment Variable Inclusion Strategy is required");
+ }
+ auto env_variable_inclusion_strategy = magic_enum::enum_cast(*env_variable_inclusion_strategy_str);
+ if (!env_variable_inclusion_strategy) {
+ throw core::ParameterException("Environment Variable Inclusion Strategy has invalid value: '" + *env_variable_inclusion_strategy_str + "'");
+ }
+ logger_->log_debug("Environment Variable Inclusion Strategy: {}", *env_variable_inclusion_strategy_str);
+ return env_variable_inclusion_strategy.value();
+}
+
+void EnvironmentVariableParameterProvider::filterEnvironmentVariablesByCommaSeparatedList(std::unordered_map& environment_variables) const {
+ std::unordered_set included_environment_variables;
+ if (auto incuded_environment_variables_str = getProperty(IncludeEnvironmentVariables)) {
+ for (const auto& included_environment_variable : minifi::utils::string::split(*incuded_environment_variables_str, ",")) {
+ included_environment_variables.insert(included_environment_variable);
+ }
+ logger_->log_debug("Filtering environment variables by comma separated list: {}", *incuded_environment_variables_str);
+ }
+ if (!included_environment_variables.empty()) {
+ environment_variables = environment_variables
+ | ranges::views::filter([&included_environment_variables](const auto& kvp) { return included_environment_variables.contains(kvp.first); })
+ | ranges::to>();
+ } else {
+ throw core::ParameterException("Environment Variable Inclusion Strategy is set to Comma-Separated, but no value is defined in Include Environment Variables property");
+ }
+}
+
+void EnvironmentVariableParameterProvider::filterEnvironmentVariablesByRegularExpression(std::unordered_map& environment_variables) const {
+ auto env_variable_regex_str = getProperty(IncludeEnvironmentVariables);
+ if (env_variable_regex_str && !env_variable_regex_str->empty()) {
+ logger_->log_debug("Filtering environment variables using regular expression: {}", *env_variable_regex_str);
+ utils::Regex regex(*env_variable_regex_str);
+ environment_variables = environment_variables
+ | ranges::views::filter([®ex](const auto& kvp) { return minifi::utils::regexMatch(kvp.first, regex); })
+ | ranges::to>();
+ } else {
+ throw core::ParameterException("Environment Variable Inclusion Strategy is set to Regular Expression, but no regex is defined in Include Environment Variables property");
+ }
+}
+
+std::vector EnvironmentVariableParameterProvider::buildParameterGroups() {
+ core::ParameterGroup group;
+ group.name = readParameterGroupName();
+ group.parameters = utils::Environment::getEnvironmentVariables();
+
+ auto env_variable_inclusion_strategy = readEnvironmentVariableInclusionStrategy();
+ if (env_variable_inclusion_strategy == EnvironmentVariableInclusionStrategyOptions::comma_separated) {
+ filterEnvironmentVariablesByCommaSeparatedList(group.parameters);
+ } else if (env_variable_inclusion_strategy == EnvironmentVariableInclusionStrategyOptions::regular_expression) {
+ filterEnvironmentVariablesByRegularExpression(group.parameters);
+ }
+ return {group};
+}
+
+REGISTER_RESOURCE(EnvironmentVariableParameterProvider, ParameterProvider);
+
+} // namespace org::apache::nifi::minifi::parameter_providers
diff --git a/libminifi/src/utils/Environment.cpp b/libminifi/src/utils/Environment.cpp
index b1724a657b..b9bfdcdfcd 100644
--- a/libminifi/src/utils/Environment.cpp
+++ b/libminifi/src/utils/Environment.cpp
@@ -27,9 +27,16 @@
#include
#include
#include
+#include
#include "utils/gsl.h"
+// Apple doesn't provide the environ global variable
+#if defined(__APPLE__) && !defined(environ)
+#include
+#define environ (*_NSGetEnviron())
+#endif
+
namespace org::apache::nifi::minifi::utils {
bool Environment::runningAsService_(false);
@@ -123,4 +130,49 @@ bool Environment::isRunningAsService() {
return runningAsService;
}
+std::unordered_map Environment::getEnvironmentVariables() {
+ std::unordered_map env_var_map;
+
+#ifdef WIN32
+ LPWCH env_strings = GetEnvironmentStringsW();
+ if (!env_strings) {
+ return env_var_map;
+ }
+
+ LPWCH env = env_strings;
+
+ while (*env) {
+ std::wstring wstring_variable_key_value_pair(env);
+
+ int size_needed = WideCharToMultiByte(CP_UTF8, 0, wstring_variable_key_value_pair.c_str(), -1, nullptr, 0, nullptr, nullptr);
+ std::vector buffer(size_needed);
+ WideCharToMultiByte(CP_UTF8, 0, wstring_variable_key_value_pair.c_str(), -1, buffer.data(), size_needed, nullptr, nullptr);
+
+ std::string variable_key_value_pair(buffer.data());
+ size_t pos = variable_key_value_pair.find('=');
+ if (pos != std::string::npos) {
+ std::string key = variable_key_value_pair.substr(0, pos);
+ std::string value = variable_key_value_pair.substr(pos + 1);
+ env_var_map[key] = value;
+ }
+
+ env += wcslen(env) + 1;
+ }
+
+ FreeEnvironmentStringsW(env_strings);
+#else
+ for (char **env = environ; *env != nullptr; ++env) {
+ std::string variable_key_value_pair(*env);
+ size_t pos = variable_key_value_pair.find('=');
+ if (pos != std::string::npos) {
+ std::string key = variable_key_value_pair.substr(0, pos);
+ std::string value = variable_key_value_pair.substr(pos + 1);
+ env_var_map[key] = value;
+ }
+ }
+#endif
+
+ return env_var_map;
+}
+
} // namespace org::apache::nifi::minifi::utils
diff --git a/libminifi/test/libtest/unit/DummyParameterProvider.cpp b/libminifi/test/libtest/unit/DummyParameterProvider.cpp
new file mode 100644
index 0000000000..f31940dd08
--- /dev/null
+++ b/libminifi/test/libtest/unit/DummyParameterProvider.cpp
@@ -0,0 +1,25 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "DummyParameterProvider.h"
+
+#include "core/Resource.h"
+
+namespace org::apache::nifi::minifi::test {
+
+REGISTER_RESOURCE(DummyParameterProvider, ParameterProvider);
+
+} // namespace org::apache::nifi::minifi::test
diff --git a/libminifi/test/libtest/unit/DummyParameterProvider.h b/libminifi/test/libtest/unit/DummyParameterProvider.h
new file mode 100644
index 0000000000..eec3ed0011
--- /dev/null
+++ b/libminifi/test/libtest/unit/DummyParameterProvider.h
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include "core/ParameterProvider.h"
+#include "utils/ArrayUtils.h"
+
+namespace org::apache::nifi::minifi::test {
+
+class DummyParameterProvider : public core::ParameterProvider {
+ public:
+ using core::ParameterProvider::ParameterProvider;
+
+ static constexpr const char* Description = "A parameter provider that returns dummy1, dummy2, and dummy3 parameters in the dummycontext.";
+ static constexpr auto Dummy1Value = core::PropertyDefinitionBuilder<>::createProperty("Dummy1 Value")
+ .withDescription("Value of dummy1 parameter")
+ .withDefaultValue("default1")
+ .build();
+ static constexpr auto Dummy2Value = core::PropertyDefinitionBuilder<>::createProperty("Dummy2 Value")
+ .withDescription("Value of dummy2 parameter")
+ .withDefaultValue("default2")
+ .build();
+ static constexpr auto Dummy3Value = core::PropertyDefinitionBuilder<>::createProperty("Dummy3 Value")
+ .withDescription("Value of dummy3 parameter")
+ .withDefaultValue("default3")
+ .build();
+
+ void initialize() override {
+ setSupportedProperties(minifi::utils::array_cat(core::ParameterProvider::Properties, std::to_array({Dummy1Value, Dummy2Value, Dummy3Value})));
+ }
+
+ protected:
+ std::vector buildParameterGroups() override {
+ core::ParameterGroup group;
+ std::unordered_map parameter_map;
+ auto dummy_value = getProperty("Dummy1 Value");
+ parameter_map.emplace("dummy1", dummy_value.value());
+ dummy_value = getProperty("Dummy2 Value");
+ parameter_map.emplace("dummy2", dummy_value.value());
+ dummy_value = getProperty("Dummy3 Value");
+ parameter_map.emplace("dummy3", dummy_value.value());
+ group.parameters = parameter_map;
+ group.name = "dummycontext";
+ return {group};
+ }
+};
+
+} // namespace org::apache::nifi::minifi::test
diff --git a/libminifi/test/libtest/unit/DummyProcessor.h b/libminifi/test/libtest/unit/DummyProcessor.h
index b120d0eca0..46cb0ca3c0 100644
--- a/libminifi/test/libtest/unit/DummyProcessor.h
+++ b/libminifi/test/libtest/unit/DummyProcessor.h
@@ -22,6 +22,7 @@
#include "core/Processor.h"
#include "agent/agent_docs.h"
+#include "core/PropertyDefinitionBuilder.h"
namespace org::apache::nifi::minifi::test {
@@ -32,13 +33,22 @@ class DummyProcessor : public minifi::core::Processor {
DummyProcessor(std::string_view name, const minifi::utils::Identifier& uuid) : Processor(name, uuid) {}
explicit DummyProcessor(std::string_view name) : Processor(name) {}
static constexpr const char* Description = "A processor that does nothing.";
- static constexpr auto Properties = std::array{};
+ static constexpr auto SimpleProperty = core::PropertyDefinitionBuilder<>::createProperty("Simple Property")
+ .withDescription("Just a simple string property")
+ .build();
+ static constexpr auto SensitiveProperty = core::PropertyDefinitionBuilder<>::createProperty("Sensitive Property")
+ .withDescription("Sensitive property")
+ .isSensitive(true)
+ .build();
+ static constexpr auto Properties = std::array{SimpleProperty, SensitiveProperty};
static constexpr auto Relationships = std::array{};
- static constexpr bool SupportsDynamicProperties = false;
+ static constexpr bool SupportsDynamicProperties = true;
static constexpr bool SupportsDynamicRelationships = false;
static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_ALLOWED;
static constexpr bool IsSingleThreaded = false;
ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS
+
+ void initialize() override { setSupportedProperties(Properties); }
};
} // namespace org::apache::nifi::minifi::test
diff --git a/libminifi/test/unit/EnvironmentVariableParameterProviderTest.cpp b/libminifi/test/unit/EnvironmentVariableParameterProviderTest.cpp
new file mode 100644
index 0000000000..58bad6da6e
--- /dev/null
+++ b/libminifi/test/unit/EnvironmentVariableParameterProviderTest.cpp
@@ -0,0 +1,101 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "unit/TestBase.h"
+#include "unit/Catch.h"
+#include "parameter-providers/EnvironmentVariableParameterProvider.h"
+#include "utils/Environment.h"
+
+namespace org::apache::nifi::minifi::test {
+
+TEST_CASE("Parameter Group Name is required", "[parameterProviders]") {
+ utils::Environment::setEnvironmentVariable("MINIFI_DATA", "minifi_data_value");
+ parameter_providers::EnvironmentVariableParameterProvider provider("EnvironmentVariableParameterProvider");
+ provider.initialize();
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::ParameterGroupName, "");
+ REQUIRE_THROWS_WITH(provider.createParameterContexts(), "Parameter Operation: Parameter Group Name is required");
+}
+
+TEST_CASE("Test EnvironmentVariableParameterProvider with default options", "[parameterProviders]") {
+ utils::Environment::setEnvironmentVariable("MINIFI_DATA", "minifi_data_value");
+ parameter_providers::EnvironmentVariableParameterProvider provider("EnvironmentVariableParameterProvider");
+ provider.initialize();
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::ParameterGroupName, "environment-variable-parameter-context");
+ auto contexts = provider.createParameterContexts();
+ REQUIRE(contexts.size() == 1);
+ REQUIRE(contexts[0]->getName() == "environment-variable-parameter-context");
+ auto enviroment_parameters = contexts[0]->getParameters();
+ REQUIRE(!enviroment_parameters.empty());
+ REQUIRE(enviroment_parameters.contains("MINIFI_DATA"));
+ REQUIRE(enviroment_parameters["MINIFI_DATA"].value == "minifi_data_value");
+}
+
+TEST_CASE("Create parameter context with selected environment variables", "[parameterProviders]") {
+ utils::Environment::setEnvironmentVariable("MINIFI_DATA", "minifi_data_value");
+ utils::Environment::setEnvironmentVariable("MINIFI_NEW_DATA", "minifi_new_data_value");
+ parameter_providers::EnvironmentVariableParameterProvider provider("EnvironmentVariableParameterProvider");
+ provider.initialize();
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::ParameterGroupName, "environment-variable-parameter-context");
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::EnvironmentVariableInclusionStrategy, "Comma-Separated");
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::IncludeEnvironmentVariables, "MINIFI_DATA,MINIFI_NEW_DATA");
+ auto contexts = provider.createParameterContexts();
+ REQUIRE(contexts.size() == 1);
+ REQUIRE(contexts[0]->getName() == "environment-variable-parameter-context");
+ auto enviroment_parameters = contexts[0]->getParameters();
+ REQUIRE(enviroment_parameters.size() == 2);
+ REQUIRE(enviroment_parameters.contains("MINIFI_DATA"));
+ REQUIRE(enviroment_parameters["MINIFI_DATA"].value == "minifi_data_value");
+ REQUIRE(enviroment_parameters.contains("MINIFI_NEW_DATA"));
+ REQUIRE(enviroment_parameters["MINIFI_NEW_DATA"].value == "minifi_new_data_value");
+}
+
+TEST_CASE("Environment variable list must be defined if Comma-Separated inclusion strategy is set", "[parameterProviders]") {
+ parameter_providers::EnvironmentVariableParameterProvider provider("EnvironmentVariableParameterProvider");
+ provider.initialize();
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::ParameterGroupName, "environment-variable-parameter-context");
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::EnvironmentVariableInclusionStrategy, "Comma-Separated");
+ REQUIRE_THROWS_WITH(provider.createParameterContexts(), "Parameter Operation: Environment Variable Inclusion Strategy is set to Comma-Separated, "
+ "but no value is defined in Include Environment Variables property");
+}
+
+TEST_CASE("Create parameter context with regex matching environment variables", "[parameterProviders]") {
+ utils::Environment::setEnvironmentVariable("MINIFI_DATA_1", "minifi_data_value");
+ utils::Environment::setEnvironmentVariable("MINIFI_NEW_DATA", "minifi_new_data_value");
+ parameter_providers::EnvironmentVariableParameterProvider provider("EnvironmentVariableParameterProvider");
+ provider.initialize();
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::ParameterGroupName, "environment-variable-parameter-context");
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::EnvironmentVariableInclusionStrategy, "Regular Expression");
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::IncludeEnvironmentVariables, "MINIFI_DATA_.*");
+ auto contexts = provider.createParameterContexts();
+ REQUIRE(contexts.size() == 1);
+ REQUIRE(contexts[0]->getName() == "environment-variable-parameter-context");
+ auto enviroment_parameters = contexts[0]->getParameters();
+ REQUIRE(enviroment_parameters.size() == 1);
+ REQUIRE(enviroment_parameters.contains("MINIFI_DATA_1"));
+ REQUIRE(enviroment_parameters["MINIFI_DATA_1"].value == "minifi_data_value");
+}
+
+TEST_CASE("Regex must be defined if regex inclusion strategy is set", "[parameterProviders]") {
+ parameter_providers::EnvironmentVariableParameterProvider provider("EnvironmentVariableParameterProvider");
+ provider.initialize();
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::ParameterGroupName, "environment-variable-parameter-context");
+ provider.setProperty(parameter_providers::EnvironmentVariableParameterProvider::EnvironmentVariableInclusionStrategy, "Regular Expression");
+ REQUIRE_THROWS_WITH(provider.createParameterContexts(), "Parameter Operation: Environment Variable Inclusion Strategy is set to Regular Expression, "
+ "but no regex is defined in Include Environment Variables property");
+}
+
+} // namespace org::apache::nifi::minifi::test
diff --git a/libminifi/test/unit/JsonFlowSerializerTests.cpp b/libminifi/test/unit/JsonFlowSerializerTests.cpp
index 6cd1eaaa1e..8911f5fbd5 100644
--- a/libminifi/test/unit/JsonFlowSerializerTests.cpp
+++ b/libminifi/test/unit/JsonFlowSerializerTests.cpp
@@ -26,9 +26,10 @@
#include "utils/crypto/EncryptionProvider.h"
#include "utils/crypto/property_encryption/PropertyEncryptionUtils.h"
#include "utils/StringUtils.h"
+#include "core/Resource.h"
+#include "utils/Environment.h"
-namespace core = org::apache::nifi::minifi::core;
-namespace utils = org::apache::nifi::minifi::utils;
+namespace org::apache::nifi::minifi::test {
constexpr std::string_view config_json_with_default_schema = R"({
"MiNiFi Config Version": 3,
@@ -553,10 +554,10 @@ constexpr std::string_view config_json_with_nifi_schema_part_2 = R"(
}
)";
-const utils::crypto::Bytes secret_key = utils::string::from_hex("75536923a75928a970077f9dae2c2b166a5413e020cb5190de4fb8edce1a38c7");
-const utils::crypto::EncryptionProvider encryption_provider{secret_key};
+const minifi::utils::crypto::Bytes secret_key = minifi::utils::string::from_hex("75536923a75928a970077f9dae2c2b166a5413e020cb5190de4fb8edce1a38c7");
+const minifi::utils::crypto::EncryptionProvider encryption_provider{secret_key};
-using OverridesMap = std::unordered_map;
+using OverridesMap = std::unordered_map;
TEST_CASE("JsonFlowSerializer can encrypt the sensitive properties") {
ConfigurationTestController test_controller;
@@ -564,7 +565,7 @@ TEST_CASE("JsonFlowSerializer can encrypt the sensitive properties") {
const auto [schema, flow_definition] = GENERATE(
std::make_tuple(core::flow::FlowSchema::getDefault(), std::string{config_json_with_default_schema}),
- std::make_tuple(core::flow::FlowSchema::getNiFiFlowJson(), utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2)));
+ std::make_tuple(core::flow::FlowSchema::getNiFiFlowJson(), minifi::utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2)));
const auto process_group = json_configuration.getRootFromPayload(flow_definition);
REQUIRE(process_group);
@@ -574,9 +575,9 @@ TEST_CASE("JsonFlowSerializer can encrypt the sensitive properties") {
REQUIRE(res);
const auto flow_serializer = core::json::JsonFlowSerializer{std::move(doc)};
- const auto processor_id = utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
- const auto controller_service_id = utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
- const auto parameter_id = utils::Identifier::parse("721e10b7-8e00-3188-9a27-476cca376978").value();
+ const auto processor_id = minifi::utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
+ const auto controller_service_id = minifi::utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
+ const auto parameter_id = minifi::utils::Identifier::parse("721e10b7-8e00-3188-9a27-476cca376978").value();
const auto [overrides, expected_results] = GENERATE_REF(
std::make_tuple(OverridesMap{},
@@ -591,7 +592,7 @@ TEST_CASE("JsonFlowSerializer can encrypt the sensitive properties") {
std::make_tuple(OverridesMap{{parameter_id, core::flow::Overrides{}.add("secret_parameter", "param_value_2")}},
std::array{"very_secure_password", "very_secure_passphrase", "param_value_2"}));
- std::string config_json_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides);
+ std::string config_json_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides, {});
{
std::regex regex{R"_("invokehttp-proxy-password": "(.*)",)_"};
@@ -600,7 +601,7 @@ TEST_CASE("JsonFlowSerializer can encrypt the sensitive properties") {
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[0]);
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[0]);
}
{
@@ -610,7 +611,7 @@ TEST_CASE("JsonFlowSerializer can encrypt the sensitive properties") {
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[1]);
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[1]);
}
{
@@ -620,7 +621,7 @@ TEST_CASE("JsonFlowSerializer can encrypt the sensitive properties") {
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[2]);
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[2]);
}
}
@@ -629,7 +630,7 @@ TEST_CASE("JsonFlowSerializer with an override can add a new property to the flo
core::flow::AdaptiveConfiguration json_configuration{test_controller.getContext()};
const auto schema = core::flow::FlowSchema::getNiFiFlowJson();
- const auto config_json_with_nifi_schema = utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2);
+ const auto config_json_with_nifi_schema = minifi::utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2);
const auto process_group = json_configuration.getRootFromPayload(config_json_with_nifi_schema);
REQUIRE(process_group);
@@ -638,12 +639,12 @@ TEST_CASE("JsonFlowSerializer with an override can add a new property to the flo
REQUIRE(res);
const auto flow_serializer = core::json::JsonFlowSerializer{std::move(doc)};
- const auto second_controller_service_id = utils::Identifier::parse("b418f4ff-e598-4ea2-921f-14f9dd864482").value();
+ const auto second_controller_service_id = minifi::utils::Identifier::parse("b418f4ff-e598-4ea2-921f-14f9dd864482").value();
SECTION("with required overrides") {
const OverridesMap overrides{{second_controller_service_id, core::flow::Overrides{}.add("Passphrase", "new passphrase")}};
- std::string config_json_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides);
+ std::string config_json_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides, {});
std::regex regex{R"_("Passphrase": "(.*)")_"};
std::smatch match_results;
@@ -655,15 +656,15 @@ TEST_CASE("JsonFlowSerializer with an override can add a new property to the flo
REQUIRE(std::regex_search(match_results.suffix().first, config_json_encrypted.cend(), match_results, regex));
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "new passphrase");
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "new passphrase");
}
SECTION("with optional overrides: the override is only used if the property is already in the flow config") {
- const auto first_controller_service_id = utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
+ const auto first_controller_service_id = minifi::utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
const OverridesMap overrides{{first_controller_service_id, core::flow::Overrides{}.addOptional("Passphrase", "first new passphrase")},
{second_controller_service_id, core::flow::Overrides{}.addOptional("Passphrase", "second new passphrase")}};
- std::string config_json_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides);
+ std::string config_json_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides, {});
std::regex regex{R"_("Passphrase": "(.*)")_"};
std::smatch match_results;
@@ -672,7 +673,7 @@ TEST_CASE("JsonFlowSerializer with an override can add a new property to the flo
REQUIRE(std::regex_search(config_json_encrypted.cbegin(), config_json_encrypted.cend(), match_results, regex));
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "first new passphrase");
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "first new passphrase");
// check that there is no second match
CHECK_FALSE(std::regex_search(match_results.suffix().first, config_json_encrypted.cend(), match_results, regex));
@@ -686,7 +687,7 @@ TEST_CASE("The encrypted flow configuration can be decrypted with the correct ke
core::flow::AdaptiveConfiguration json_configuration_before{configuration_context};
const auto schema = core::flow::FlowSchema::getNiFiFlowJson();
- const auto config_json_with_nifi_schema = utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2);
+ const auto config_json_with_nifi_schema = minifi::utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2);
const auto process_group_before = json_configuration_before.getRootFromPayload(config_json_with_nifi_schema);
REQUIRE(process_group_before);
@@ -694,13 +695,13 @@ TEST_CASE("The encrypted flow configuration can be decrypted with the correct ke
rapidjson::ParseResult res = doc.Parse(config_json_with_nifi_schema.data(), config_json_with_nifi_schema.size());
REQUIRE(res);
const auto flow_serializer = core::json::JsonFlowSerializer{std::move(doc)};
- std::string config_json_encrypted = flow_serializer.serialize(*process_group_before, schema, encryption_provider, {});
+ std::string config_json_encrypted = flow_serializer.serialize(*process_group_before, schema, encryption_provider, {}, {});
core::flow::AdaptiveConfiguration json_configuration_after{configuration_context};
const auto process_group_after = json_configuration_after.getRootFromPayload(config_json_encrypted);
REQUIRE(process_group_after);
- const auto processor_id = utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
+ const auto processor_id = minifi::utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
const auto* processor_before = process_group_before->findProcessorById(processor_id);
REQUIRE(processor_before);
const auto* processor_after = process_group_after->findProcessorById(processor_id);
@@ -731,7 +732,7 @@ TEST_CASE("The encrypted flow configuration cannot be decrypted with an incorrec
core::flow::AdaptiveConfiguration json_configuration_before{configuration_context};
const auto schema = core::flow::FlowSchema::getNiFiFlowJson();
- const auto config_json_with_nifi_schema = utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2);
+ const auto config_json_with_nifi_schema = minifi::utils::string::join_pack(config_json_with_nifi_schema_part_1, config_json_with_nifi_schema_part_2);
const auto process_group_before = json_configuration_before.getRootFromPayload(config_json_with_nifi_schema);
REQUIRE(process_group_before);
@@ -739,11 +740,165 @@ TEST_CASE("The encrypted flow configuration cannot be decrypted with an incorrec
rapidjson::ParseResult res = doc.Parse(config_json_with_nifi_schema.data(), config_json_with_nifi_schema.size());
REQUIRE(res);
const auto flow_serializer = core::json::JsonFlowSerializer{std::move(doc)};
- std::string config_json_encrypted = flow_serializer.serialize(*process_group_before, schema, encryption_provider, {});
+ std::string config_json_encrypted = flow_serializer.serialize(*process_group_before, schema, encryption_provider, {}, {});
- const utils::crypto::Bytes different_secret_key = utils::string::from_hex("ea55b7d0edc22280c9547e4d89712b3fae74f96d82f240a004fb9fbd0640eec7");
- configuration_context.sensitive_values_encryptor = utils::crypto::EncryptionProvider{different_secret_key};
+ const minifi::utils::crypto::Bytes different_secret_key = minifi::utils::string::from_hex("ea55b7d0edc22280c9547e4d89712b3fae74f96d82f240a004fb9fbd0640eec7");
+ configuration_context.sensitive_values_encryptor = minifi::utils::crypto::EncryptionProvider{different_secret_key};
core::flow::AdaptiveConfiguration json_configuration_after{configuration_context};
- REQUIRE_THROWS_AS(json_configuration_after.getRootFromPayload(config_json_encrypted), utils::crypto::EncryptionError);
+ REQUIRE_THROWS_AS(json_configuration_after.getRootFromPayload(config_json_encrypted), minifi::utils::crypto::EncryptionError);
}
+
+TEST_CASE("Parameter provider generated parameter context is serialized correctly") {
+ ConfigurationTestController test_controller;
+ auto configuration_context = test_controller.getContext();
+ configuration_context.sensitive_values_encryptor = encryption_provider;
+ core::flow::AdaptiveConfiguration json_configuration_before{configuration_context};
+
+ const auto schema = core::flow::FlowSchema::getNiFiFlowJson();
+ static const std::string config_json_with_nifi_schema =
+ R"(
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Environment Variable Inclusion Strategy": "Comma-Separated",
+ "Include Environment Variables": "MINIFI_DATA,SECRET_MINIFI_DATA",
+ "Sensitive Parameter Scope": "selected",
+ "Sensitive Parameter List": "SECRET_MINIFI_DATA",
+ "Parameter Group Name": "environment-variable-parameter-context"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [{
+ "name": "DummyProcessor",
+ "identifier": "aabb6d26-8a8d-4338-92c9-1b8c67ec18e0",
+ "type": "DummyProcessor",
+ "scheduling strategy": "TIMER_DRIVEN",
+ "scheduling period": "15 sec",
+ "properties": {
+ "Simple Property": "#{MINIFI_DATA}",
+ "Sensitive Property": "#{SECRET_MINIFI_DATA}"
+ }
+ }],
+ "parameterContextName": "environment-variable-parameter-context"
+ }
+})";
+
+ minifi::utils::Environment::setEnvironmentVariable("MINIFI_DATA", "minifi_data_value");
+ minifi::utils::Environment::setEnvironmentVariable("SECRET_MINIFI_DATA", "secret_minifi_data_value");
+ const auto process_group_before = json_configuration_before.getRootFromPayload(config_json_with_nifi_schema);
+ REQUIRE(process_group_before);
+
+ std::string reserialized_config = json_configuration_before.serialize(*process_group_before);
+ rapidjson::Document result_doc;
+ rapidjson::ParseResult res = result_doc.Parse(reserialized_config.data(), reserialized_config.size());
+ REQUIRE(res);
+ REQUIRE(result_doc.HasMember("parameterContexts"));
+ auto parameters = result_doc["parameterContexts"].GetArray()[0]["parameters"].GetArray();
+ REQUIRE(parameters.Size() == 2);
+ for (const auto& parameter : parameters) {
+ std::string name = parameter["name"].GetString();
+ std::string value = parameter["value"].GetString();
+ if (name == "MINIFI_DATA") {
+ REQUIRE(value == "minifi_data_value");
+ } else if (name == "SECRET_MINIFI_DATA") {
+ REQUIRE(minifi::utils::crypto::property_encryption::decrypt(value, encryption_provider) == "secret_minifi_data_value");
+ }
+ }
+}
+
+TEST_CASE("Parameter provider generated parameter context is not serialized if parameter context already exists") {
+ ConfigurationTestController test_controller;
+ auto configuration_context = test_controller.getContext();
+ configuration_context.sensitive_values_encryptor = encryption_provider;
+ core::flow::AdaptiveConfiguration json_configuration_before{configuration_context};
+
+ const auto schema = core::flow::FlowSchema::getNiFiFlowJson();
+ static const std::string config_json_with_nifi_schema =
+ R"(
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Environment Variable Inclusion Strategy": "Comma-Separated",
+ "Include Environment Variables": "MINIFI_DATA,SECRET_MINIFI_DATA",
+ "Sensitive Parameter Scope": "selected",
+ "Sensitive Parameter List": "SECRET_MINIFI_DATA",
+ "Parameter Group Name": "environment-variable-parameter-context"
+ }
+ }
+ ],
+ "parameterContexts": [
+ {
+ "identifier": "123ee5f5-0192-1000-0482-4e333725e345",
+ "name": "environment-variable-parameter-context",
+ "description": "my parameter context",
+ "parameters": [
+ {
+ "name": "SECRET_MINIFI_DATA",
+ "description": "",
+ "sensitive": true,
+ "provided": true,
+ "value": "old_secret_minifi_data_value"
+ },
+ {
+ "name": "MINIFI_DATA",
+ "description": "",
+ "sensitive": false,
+ "provided": true,
+ "value": "old_minifi_data_value"
+ }
+ ],
+ "parameterProvider": "d26ee5f5-0192-1000-0482-4e333725e089"
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [{
+ "name": "DummyProcessor",
+ "identifier": "aabb6d26-8a8d-4338-92c9-1b8c67ec18e0",
+ "type": "DummyProcessor",
+ "scheduling strategy": "TIMER_DRIVEN",
+ "scheduling period": "15 sec",
+ "properties": {
+ "Simple Property": "#{MINIFI_DATA}",
+ "Sensitive Property": "#{SECRET_MINIFI_DATA}"
+ }
+ }],
+ "parameterContextName": "environment-variable-parameter-context"
+ }
+})";
+
+ minifi::utils::Environment::setEnvironmentVariable("MINIFI_DATA", "minifi_data_value");
+ minifi::utils::Environment::setEnvironmentVariable("SECRET_MINIFI_DATA", "secret_minifi_data_value");
+ const auto process_group_before = json_configuration_before.getRootFromPayload(config_json_with_nifi_schema);
+ REQUIRE(process_group_before);
+
+ std::string reserialized_config = json_configuration_before.serialize(*process_group_before);
+ rapidjson::Document result_doc;
+ rapidjson::ParseResult res = result_doc.Parse(reserialized_config.data(), reserialized_config.size());
+ REQUIRE(res);
+ REQUIRE(result_doc.HasMember("parameterContexts"));
+ auto parameters = result_doc["parameterContexts"].GetArray()[0]["parameters"].GetArray();
+ REQUIRE(parameters.Size() == 2);
+ for (const auto& parameter : parameters) {
+ std::string name = parameter["name"].GetString();
+ std::string value = parameter["value"].GetString();
+ if (name == "MINIFI_DATA") {
+ REQUIRE(value == "old_minifi_data_value");
+ } else if (name == "SECRET_MINIFI_DATA") {
+ REQUIRE(minifi::utils::crypto::property_encryption::decrypt(value, encryption_provider) == "old_secret_minifi_data_value");
+ }
+ }
+}
+
+} // namespace org::apache::nifi::minifi::test
diff --git a/libminifi/test/unit/ParameterTokenParserTest.cpp b/libminifi/test/unit/ParameterTokenParserTest.cpp
index 9b7d32a3c2..8f21a2cb2f 100644
--- a/libminifi/test/unit/ParameterTokenParserTest.cpp
+++ b/libminifi/test/unit/ParameterTokenParserTest.cpp
@@ -102,8 +102,8 @@ TEST_CASE("Test invalid token names") {
TEST_CASE("Test token replacement") {
core::NonSensitiveParameterTokenParser parser("## What is #{what}, baby don't hurt #{who}, don't hurt #{who}, no more ##");
core::ParameterContext context("test_context");
- context.addParameter(core::Parameter{"what", "", false, "love"});
- context.addParameter(core::Parameter{"who", "", false, "me"});
+ context.addParameter(core::Parameter{"what", "", false, false, "love"});
+ context.addParameter(core::Parameter{"who", "", false, false, "me"});
REQUIRE(parser.replaceParameters(&context) == "## What is love, baby don't hurt me, don't hurt me, no more ##");
}
@@ -111,15 +111,15 @@ TEST_CASE("Test replacement with escaped tokens") {
core::NonSensitiveParameterTokenParser parser("### What is #####{what}, baby don't hurt ###{who}, don't hurt ###{who}, no ####{more} ##{");
REQUIRE(parser.getTokens().size() == 4);
core::ParameterContext context("test_context");
- context.addParameter(core::Parameter{"what", "", false, "love"});
- context.addParameter(core::Parameter{"who", "", false, "me"});
+ context.addParameter(core::Parameter{"what", "", false, false, "love"});
+ context.addParameter(core::Parameter{"who", "", false, false, "me"});
REQUIRE(parser.replaceParameters(&context) == "### What is ##love, baby don't hurt #me, don't hurt #me, no ##{more} ##{");
}
TEST_CASE("Test replacement with missing token in context") {
core::NonSensitiveParameterTokenParser parser("What is #{what}, baby don't hurt #{who}, don't hurt #{who}, no more");
core::ParameterContext context("test_context");
- context.addParameter(core::Parameter{"what", "", false, "love"});
+ context.addParameter(core::Parameter{"what", "", false, false, "love"});
REQUIRE_THROWS_WITH(parser.replaceParameters(&context), "Parameter Operation: Parameter 'who' not found");
}
@@ -128,8 +128,8 @@ TEST_CASE("Sensitive property parameter replacement is not supported") {
utils::crypto::EncryptionProvider encryption_provider{secret_key};
core::SensitiveParameterTokenParser parser("What is #{what}, baby don't hurt #{who}, don't hurt #{who}, no more", encryption_provider);
core::ParameterContext context("test_context");
- context.addParameter(core::Parameter{"what", "", false, "love"});
- context.addParameter(core::Parameter{"who", "", false, "me"});
+ context.addParameter(core::Parameter{"what", "", false, false, "love"});
+ context.addParameter(core::Parameter{"who", "", false, false, "me"});
REQUIRE_THROWS_WITH(parser.replaceParameters(&context), "Parameter Operation: Non-sensitive parameter 'what' cannot be referenced in a sensitive property");
}
@@ -154,8 +154,8 @@ TEST_CASE("Test sensitive token replacement") {
core::SensitiveParameterTokenParser parser("What is #{what}, baby don't hurt #{who}, don't hurt #{who}, no more", encryption_provider);
auto value1 = utils::crypto::property_encryption::encrypt("love", encryption_provider);
auto value2 = utils::crypto::property_encryption::encrypt("me", encryption_provider);
- context.addParameter(core::Parameter{"what", "", true, value1});
- context.addParameter(core::Parameter{"who", "", true, value2});
+ context.addParameter(core::Parameter{"what", "", true, false, value1});
+ context.addParameter(core::Parameter{"who", "", true, false, value2});
REQUIRE(parser.replaceParameters(&context) == "What is love, baby don't hurt me, don't hurt me, no more");
}
diff --git a/libminifi/test/unit/YamlFlowSerializerTests.cpp b/libminifi/test/unit/YamlFlowSerializerTests.cpp
index 75062fa92b..7f3ee5409f 100644
--- a/libminifi/test/unit/YamlFlowSerializerTests.cpp
+++ b/libminifi/test/unit/YamlFlowSerializerTests.cpp
@@ -26,9 +26,10 @@
#include "utils/crypto/EncryptionProvider.h"
#include "utils/crypto/property_encryption/PropertyEncryptionUtils.h"
#include "utils/StringUtils.h"
+#include "core/Resource.h"
+#include "utils/Environment.h"
-namespace core = org::apache::nifi::minifi::core;
-namespace utils = org::apache::nifi::minifi::utils;
+namespace org::apache::nifi::minifi::test {
constexpr std::string_view config_yaml = R"(MiNiFi Config Version: 3
Flow Controller:
@@ -179,10 +180,10 @@ Remote Process Groups: []
NiFi Properties Overrides: {}
)";
-const utils::crypto::Bytes secret_key = utils::string::from_hex("cb76fe6fe4cbfdc3770c0cb0afc910f81ced4d436b11f691395fc2a9dbea27ca");
-const utils::crypto::EncryptionProvider encryption_provider{secret_key};
+const minifi::utils::crypto::Bytes secret_key = minifi::utils::string::from_hex("cb76fe6fe4cbfdc3770c0cb0afc910f81ced4d436b11f691395fc2a9dbea27ca");
+const minifi::utils::crypto::EncryptionProvider encryption_provider{secret_key};
-using OverridesMap = std::unordered_map;
+using OverridesMap = std::unordered_map;
TEST_CASE("YamlFlowSerializer can encrypt the sensitive properties") {
ConfigurationTestController test_controller;
@@ -195,9 +196,9 @@ TEST_CASE("YamlFlowSerializer can encrypt the sensitive properties") {
YAML::Node root_yaml_node = YAML::Load(std::string{config_yaml});
const auto flow_serializer = core::yaml::YamlFlowSerializer{root_yaml_node};
- const auto processor_id = utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
- const auto controller_service_id = utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
- const auto parameter_id = utils::Identifier::parse("721e10b7-8e00-3188-9a27-476cca376978").value();
+ const auto processor_id = minifi::utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
+ const auto controller_service_id = minifi::utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
+ const auto parameter_id = minifi::utils::Identifier::parse("721e10b7-8e00-3188-9a27-476cca376978").value();
const auto [overrides, expected_results] = GENERATE_REF(
std::make_tuple(OverridesMap{},
@@ -212,7 +213,7 @@ TEST_CASE("YamlFlowSerializer can encrypt the sensitive properties") {
std::make_tuple(OverridesMap{{parameter_id, core::flow::Overrides{}.add("secret_parameter", "param_value_2")}},
std::array{"very_secure_password", "very_secure_passphrase", "param_value_2"}));
- std::string config_yaml_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides);
+ std::string config_yaml_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides, {});
{
std::regex regex{R"_(invokehttp-proxy-password: (.*))_"};
@@ -221,7 +222,7 @@ TEST_CASE("YamlFlowSerializer can encrypt the sensitive properties") {
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[0]);
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[0]);
}
{
@@ -231,7 +232,7 @@ TEST_CASE("YamlFlowSerializer can encrypt the sensitive properties") {
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[1]);
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[1]);
}
{
@@ -241,7 +242,7 @@ TEST_CASE("YamlFlowSerializer can encrypt the sensitive properties") {
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[2]);
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == expected_results[2]);
}
}
@@ -256,12 +257,12 @@ TEST_CASE("YamlFlowSerializer with an override can add a new property to the flo
YAML::Node root_yaml_node = YAML::Load(std::string{config_yaml});
const auto flow_serializer = core::yaml::YamlFlowSerializer{root_yaml_node};
- const auto second_controller_service_id = utils::Identifier::parse("b418f4ff-e598-4ea2-921f-14f9dd864482").value();
+ const auto second_controller_service_id = minifi::utils::Identifier::parse("b418f4ff-e598-4ea2-921f-14f9dd864482").value();
SECTION("with required overrides") {
const OverridesMap overrides{{second_controller_service_id, core::flow::Overrides{}.add("Passphrase", "new passphrase")}};
- std::string config_yaml_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides);
+ std::string config_yaml_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides, {});
std::regex regex{R"_(Passphrase: (.*))_"};
std::smatch match_results;
@@ -273,15 +274,15 @@ TEST_CASE("YamlFlowSerializer with an override can add a new property to the flo
REQUIRE(std::regex_search(match_results.suffix().first, config_yaml_encrypted.cend(), match_results, regex));
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "new passphrase");
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "new passphrase");
}
SECTION("with optional overrides: the override is only used if the property is already in the flow config") {
- const auto first_controller_service_id = utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
+ const auto first_controller_service_id = minifi::utils::Identifier::parse("b9801278-7b5d-4314-aed6-713fd4b5f933").value();
const OverridesMap overrides{{first_controller_service_id, core::flow::Overrides{}.addOptional("Passphrase", "first new passphrase")},
{second_controller_service_id, core::flow::Overrides{}.addOptional("Passphrase", "second new passphrase")}};
- std::string config_yaml_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides);
+ std::string config_yaml_encrypted = flow_serializer.serialize(*process_group, schema, encryption_provider, overrides, {});
std::regex regex{R"_(Passphrase: (.*))_"};
std::smatch match_results;
@@ -290,7 +291,7 @@ TEST_CASE("YamlFlowSerializer with an override can add a new property to the flo
REQUIRE(std::regex_search(config_yaml_encrypted.cbegin(), config_yaml_encrypted.cend(), match_results, regex));
REQUIRE(match_results.size() == 2);
std::string encrypted_value = match_results[1];
- CHECK(utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "first new passphrase");
+ CHECK(minifi::utils::crypto::property_encryption::decrypt(encrypted_value, encryption_provider) == "first new passphrase");
// check that there is no second match
CHECK_FALSE(std::regex_search(match_results.suffix().first, config_yaml_encrypted.cend(), match_results, regex));
@@ -309,13 +310,13 @@ TEST_CASE("The encrypted flow configuration can be decrypted with the correct ke
const auto schema = core::flow::FlowSchema::getDefault();
YAML::Node root_yaml_node = YAML::Load(std::string{config_yaml});
const auto flow_serializer_before = core::yaml::YamlFlowSerializer{root_yaml_node};
- std::string config_yaml_encrypted = flow_serializer_before.serialize(*process_group_before, schema, encryption_provider, {});
+ std::string config_yaml_encrypted = flow_serializer_before.serialize(*process_group_before, schema, encryption_provider, {}, {});
core::flow::AdaptiveConfiguration yaml_configuration_after{configuration_context};
const auto process_group_after = yaml_configuration_after.getRootFromPayload(config_yaml_encrypted);
REQUIRE(process_group_after);
- const auto processor_id = utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
+ const auto processor_id = minifi::utils::Identifier::parse("469617f1-3898-4bbf-91fe-27d8f4dd2a75").value();
const auto* processor_before = process_group_before->findProcessorById(processor_id);
REQUIRE(processor_before);
const auto* processor_after = process_group_after->findProcessorById(processor_id);
@@ -351,11 +352,141 @@ TEST_CASE("The encrypted flow configuration cannot be decrypted with an incorrec
const auto schema = core::flow::FlowSchema::getDefault();
YAML::Node root_yaml_node = YAML::Load(std::string{config_yaml});
const auto flow_serializer = core::yaml::YamlFlowSerializer{root_yaml_node};
- std::string config_yaml_encrypted = flow_serializer.serialize(*process_group_before, schema, encryption_provider, {});
+ std::string config_yaml_encrypted = flow_serializer.serialize(*process_group_before, schema, encryption_provider, {}, {});
- const utils::crypto::Bytes different_secret_key = utils::string::from_hex("ea55b7d0edc22280c9547e4d89712b3fae74f96d82f240a004fb9fbd0640eec7");
- configuration_context.sensitive_values_encryptor = utils::crypto::EncryptionProvider{different_secret_key};
+ const minifi::utils::crypto::Bytes different_secret_key = minifi::utils::string::from_hex("ea55b7d0edc22280c9547e4d89712b3fae74f96d82f240a004fb9fbd0640eec7");
+ configuration_context.sensitive_values_encryptor = minifi::utils::crypto::EncryptionProvider{different_secret_key};
core::flow::AdaptiveConfiguration yaml_configuration_after{configuration_context};
- REQUIRE_THROWS_AS(yaml_configuration_after.getRootFromPayload(config_yaml_encrypted), utils::crypto::EncryptionError);
+ REQUIRE_THROWS_AS(yaml_configuration_after.getRootFromPayload(config_yaml_encrypted), minifi::utils::crypto::EncryptionError);
}
+
+TEST_CASE("Parameter provider generated parameter context is serialized correctly") {
+ ConfigurationTestController test_controller;
+ auto configuration_context = test_controller.getContext();
+ configuration_context.sensitive_values_encryptor = encryption_provider;
+ core::flow::AdaptiveConfiguration yaml_configuration_before{configuration_context};
+
+ const auto schema = core::flow::FlowSchema::getDefault();
+ static const std::string config_yaml =
+ R"(
+Flow Controller:
+ name: root
+ comment: ""
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: EnvironmentVariableParameterProvider
+ type: EnvironmentVariableParameterProvider
+ Properties:
+ Environment Variable Inclusion Strategy: Comma-Separated
+ Include Environment Variables: MINIFI_DATA,SECRET_MINIFI_DATA
+ Sensitive Parameter Scope: selected
+ Sensitive Parameter List: SECRET_MINIFI_DATA
+ Parameter Group Name: environment-variable-parameter-context
+Processors:
+ - name: DummyProcessor
+ id: aabb6d26-8a8d-4338-92c9-1b8c67ec18e0
+ type: DummyProcessor
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: "15 sec"
+ Properties:
+ Simple Property: "#{MINIFI_DATA}"
+ Sensitive Property: "#{SECRET_MINIFI_DATA}"
+Parameter Context Name: environment-variable-parameter-context
+)";
+
+ minifi::utils::Environment::setEnvironmentVariable("MINIFI_DATA", "minifi_data_value");
+ minifi::utils::Environment::setEnvironmentVariable("SECRET_MINIFI_DATA", "secret_minifi_data_value");
+ const auto process_group_before = yaml_configuration_before.getRootFromPayload(std::string{config_yaml});
+ REQUIRE(process_group_before);
+
+ std::string reserialized_config = yaml_configuration_before.serialize(*process_group_before);
+ YAML::Node result_yaml_node = YAML::Load(std::string{reserialized_config});
+
+ REQUIRE(result_yaml_node["Parameter Contexts"].IsDefined());
+ auto parameters = result_yaml_node["Parameter Contexts"][0]["Parameters"];
+ REQUIRE(parameters.size() == 2);
+ for (const auto& parameter : parameters) {
+ auto name = parameter["name"].as();
+ auto value = parameter["value"].as();
+ if (name == "MINIFI_DATA") {
+ REQUIRE(value == "minifi_data_value");
+ } else if (name == "SECRET_MINIFI_DATA") {
+ REQUIRE(minifi::utils::crypto::property_encryption::decrypt(value, encryption_provider) == "secret_minifi_data_value");
+ }
+ }
+}
+
+TEST_CASE("Parameter provider generated parameter context is not serialized if parameter context already exists") {
+ ConfigurationTestController test_controller;
+ auto configuration_context = test_controller.getContext();
+ configuration_context.sensitive_values_encryptor = encryption_provider;
+ core::flow::AdaptiveConfiguration yaml_configuration_before{configuration_context};
+
+ const auto schema = core::flow::FlowSchema::getDefault();
+ static const std::string config_yaml =
+ R"(
+Flow Controller:
+ name: root
+ comment: ""
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: EnvironmentVariableParameterProvider
+ type: EnvironmentVariableParameterProvider
+ Properties:
+ Environment Variable Inclusion Strategy: Comma-Separated
+ Include Environment Variables: MINIFI_DATA,SECRET_MINIFI_DATA
+ Sensitive Parameter Scope: selected
+ Sensitive Parameter List: SECRET_MINIFI_DATA
+ Parameter Group Name: environment-variable-parameter-context
+Parameter Contexts:
+ - id: 123ee5f5-0192-1000-0482-4e333725e345
+ name: environment-variable-parameter-context
+ description: my parameter context
+ Parameters:
+ - name: SECRET_MINIFI_DATA
+ description: ''
+ sensitive: true
+ provided: true
+ value: old_secret_minifi_data_value
+ - name: MINIFI_DATA
+ description: ''
+ sensitive: false
+ provided: true
+ value: old_minifi_data_value
+ Parameter Provider: d26ee5f5-0192-1000-0482-4e333725e089
+Processors:
+ - name: DummyProcessor
+ id: aabb6d26-8a8d-4338-92c9-1b8c67ec18e0
+ type: DummyProcessor
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: "15 sec"
+ Properties:
+ Simple Property: "#{MINIFI_DATA}"
+ Sensitive Property: "#{SECRET_MINIFI_DATA}"
+Parameter Context Name: environment-variable-parameter-context
+)";
+
+ minifi::utils::Environment::setEnvironmentVariable("MINIFI_DATA", "minifi_data_value");
+ minifi::utils::Environment::setEnvironmentVariable("SECRET_MINIFI_DATA", "secret_minifi_data_value");
+ const auto process_group_before = yaml_configuration_before.getRootFromPayload(std::string{config_yaml});
+ REQUIRE(process_group_before);
+
+ std::string reserialized_config = yaml_configuration_before.serialize(*process_group_before);
+ YAML::Node result_yaml_node = YAML::Load(std::string{reserialized_config});
+
+ REQUIRE(result_yaml_node["Parameter Contexts"].IsDefined());
+ auto parameters = result_yaml_node["Parameter Contexts"][0]["Parameters"];
+ REQUIRE(parameters.size() == 2);
+ for (const auto& parameter : parameters) {
+ auto name = parameter["name"].as();
+ auto value = parameter["value"].as();
+ if (name == "MINIFI_DATA") {
+ REQUIRE(value == "old_minifi_data_value");
+ } else if (name == "SECRET_MINIFI_DATA") {
+ REQUIRE(minifi::utils::crypto::property_encryption::decrypt(value, encryption_provider) == "old_secret_minifi_data_value");
+ }
+ }
+}
+
+} // namespace org::apache::nifi::minifi::test