Fix --set flag #2162

joe-elliott · 2020-11-16T21:14:14Z

Description:
Re-adds the --set parameter in a way that works with {} being used in configs.

Link to tracking Issue:
Fixes #1907

Testing:
The original tests were added along with the addition of the following:

extensions:
  health_check: {}

This failed the previous tests, but now passes.

Documentation:

codecov · 2020-11-16T21:20:36Z

Codecov Report

Merging #2162 (9720e4c) into master (7c28105) will decrease coverage by 0.02%.
The diff coverage is 76.92%.

@@            Coverage Diff             @@
##           master    #2162      +/-   ##
==========================================
- Coverage   91.97%   91.94%   -0.03%     
==========================================
  Files         271      272       +1     
  Lines       15653    15675      +22     
==========================================
+ Hits        14397    14413      +16     
- Misses        854      857       +3     
- Partials      402      405       +3

Impacted Files	Coverage Δ
service/service.go	`52.67% <50.00%> (-0.23%)`	⬇️
service/set_flag.go	`80.95% <80.95%> (ø)`
config/config.go	`98.87% <100.00%> (ø)`
internal/data/traceid.go	`84.61% <0.00%> (-7.70%)`	⬇️
internal/data/spanid.go	`100.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c28105...9720e4c. Read the comment docs.

tigrannajaryan · 2020-11-18T16:01:36Z

@pavolloffay can you review please?

pavolloffay · 2020-11-19T07:57:59Z

Here is the revert #1905 and the original PR https://github.com/open-telemetry/opentelemetry-collector/pull/1640/files

pavolloffay · 2020-11-19T08:00:47Z

@joe-elliott could you highlight what fixed the issue?

joe-elliott · 2020-11-19T14:12:28Z

@pavolloffay
Previously it was iterating through .AllKeys() in the config file viper and calling .Set() on each one to force them into viper's config "override". This allowed the --set values to nicely merge with the config values. However, .AllKeys() just failed to return a key with an empty object like : healthcheck: {} and so it was never set.

By iterating through all keys and only calling .Set on the root keys we are preserving the entire viper config object faithfully.
https://github.com/open-telemetry/opentelemetry-collector/pull/2162/files#diff-2a9451c3c8352620ac89fc278d2e47f08ce3fa76d427cb1694ff8119003caa9fR68-R77

I also attempted other changes like .MergeConfig() and can confirm they will not work for this.

joe-elliott · 2020-11-24T21:05:04Z

To continue this conversation. If your file looks like this:

a: {}
b:
  c: 2
  d:

If you load this with viper and run AllSettings() you will get the following structure back:

b:
  c: 2

If you run AllKeys() and log each one you get the following keys:

b.c
b.d

Viper does not enumerate keys that are empty objects but it will enumerate nil keys. By calling Get() on the root keys only we are able to faithfully reconstruct the original config in the viper overrides so it will nicely merge with the --set params.

@pavolloffay thoughts?

joe-elliott · 2020-11-27T16:03:36Z

@jpkrohling do you mind taking a look?

tigrannajaryan · 2020-11-27T16:47:06Z

@pavolloffay
Previously it was iterating through .AllKeys() in the config file viper and calling .Set() on each one to force them into viper's config "override". This allowed the --set values to nicely merge with the config values. However, .AllKeys() just failed to return a key with an empty object like : healthcheck: {} and so it was never set.

By iterating through all keys and only calling .Set on the root keys we are preserving the entire viper config object faithfully.
https://github.com/open-telemetry/opentelemetry-collector/pull/2162/files#diff-2a9451c3c8352620ac89fc278d2e47f08ce3fa76d427cb1694ff8119003caa9fR68-R77

I also attempted other changes like .MergeConfig() and can confirm they will not work for this.

To continue this conversation. If your file looks like this:
a: {}
b:
  c: 2
  d:
If you load this with viper and run AllSettings() you will get the following structure back:
b:
  c: 2
If you run AllKeys() and log each one you get the following keys:
b.c
b.d
Viper does not enumerate keys that are empty objects but it will enumerate nil keys. By calling Get() on the root keys only we are able to faithfully reconstruct the original config in the viper overrides so it will nicely merge with the --set params.

@pavolloffay thoughts?

@joe-elliott these ^^^^ are very nice explanations. Would you be able to add them as comments to the relevant parts of the code? Perhaps drop the "previously" part and keep the part that describes how it works now and most importantly why it works correctly.

jpkrohling

LGTM. There are a couple of minor improvements that could be made, but those can be safely ignored...

service/service_test.go

jpkrohling · 2020-11-30T09:49:05Z

service/testdata/otelcol-config.yaml

@@ -17,7 +17,7 @@ processors:
  batch:

 extensions:
-  health_check:


From what I understood, it's still OK to have nil here instead of empty. If that's the case, keep this as nil, as a proof that this change didn't break compat with previous versions.

Yes, nil is fine here as well as empty object. The problem is that empty object was not working and was not being tested for. With these changes this file now tests both empty object and nil object:

... extensions: health_check: {} <- empty object test pprof: <- nil test ...

testbed/testbed/otelcol_runner.go

service/set_flag.go

joe-elliott · 2020-11-30T14:49:33Z

I believe I've addressed all comments and improved the in code comments regarding the behavior of viper and why we are setting the root keys.

I am getting a failed contribtest:

--- FAIL: TestNewLogsExporter (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbde2df]

Could this be due to changes I made? should I dig into this?

tigrannajaryan · 2020-11-30T15:32:15Z

I am getting a failed contribtest:

--- FAIL: TestNewLogsExporter (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbde2df]

Could this be due to changes I made? should I dig into this?

Unlikely that your change could cause this. Looks like a bug in TestNewLogsExporter.

service/service.go

tigrannajaryan · 2020-11-30T17:10:51Z

@joe-elliott please rebase from latest master.

…y#1640)" (open-telemetry#1905)" This reverts commit 2efdb28.

Signed-off-by: Joe Elliott <[email protected]>

Co-authored-by: Juraci Paixão Kröhling <[email protected]> Signed-off-by: Joe Elliott <[email protected]>

Signed-off-by: Joe Elliott <[email protected]>

joe-elliott · 2020-12-01T16:11:27Z

Rebased but now getting a build error on contrib:

# github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter/translator [github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter/translator.test]
translator/segment_test.go:743:6: val1.InitEmpty undefined (type pdata.AttributeValue has no field or method InitEmpty)
translator/segment_test.go:746:6: val2.InitEmpty undefined (type pdata.AttributeValue has no field or method InitEmpty)

tigrannajaryan · 2020-12-01T16:22:10Z

Rebased but now getting a build error on contrib:

# github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter/translator [github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter/translator.test]
translator/segment_test.go:743:6: val1.InitEmpty undefined (type pdata.AttributeValue has no field or method InitEmpty)
translator/segment_test.go:746:6: val2.InitEmpty undefined (type pdata.AttributeValue has no field or method InitEmpty)

This is expected, ignore it.

* Deprecate Array attribute in favor of *Slice types * Use new attr types in Jaeger exporter * Use slice attr types in otlpmetric * Use slice attr types in otlptrace * Use slice attr types in zipkin exporter * Remove array attr test from deprectated oteltest func * Use StringSlice for cmd arg resource attr * Add changes to the changelog * Remove use of deprecated Array func

joe-elliott requested a review from a team November 16, 2020 21:14

jpkrohling approved these changes Nov 30, 2020

View reviewed changes

joe-elliott force-pushed the fix-set branch from 4311d42 to e84f3f7 Compare November 30, 2020 14:30

jpkrohling approved these changes Nov 30, 2020

View reviewed changes

tigrannajaryan reviewed Nov 30, 2020

View reviewed changes

service/service.go Outdated Show resolved Hide resolved

joe-elliott and others added 6 commits November 30, 2020 12:43

Revert "Revert "Support set flag for component configs (open-telemetr…

21713f2

…y#1640)" (open-telemetry#1905)" This reverts commit 2efdb28.

Set root keys only to fix --set param

45889d2

Signed-off-by: Joe Elliott <[email protected]>

Added {} to break old tests

82baad2

Signed-off-by: Joe Elliott <[email protected]>

Apply suggestions from code review

097fe80

Co-authored-by: Juraci Paixão Kröhling <[email protected]> Signed-off-by: Joe Elliott <[email protected]>

Removed unnecessary params

d9f50ef

Signed-off-by: Joe Elliott <[email protected]>

Improved explanation of viper shenanigans

2b52216

Signed-off-by: Joe Elliott <[email protected]>

joe-elliott force-pushed the fix-set branch from bd9ad67 to 2b52216 Compare November 30, 2020 17:43

typo

9720e4c

Signed-off-by: Joe Elliott <[email protected]>

tigrannajaryan approved these changes Dec 1, 2020

View reviewed changes

tigrannajaryan merged commit cb608a9 into open-telemetry:master Dec 1, 2020

Troels51 pushed a commit to Troels51/opentelemetry-collector that referenced this pull request Jul 5, 2024

Change OTLP Json field name to camelCase (open-telemetry#2162)

8b7f9b4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix --set flag #2162

Fix --set flag #2162

joe-elliott commented Nov 16, 2020 •

edited by tigrannajaryan

Loading

codecov bot commented Nov 16, 2020 •

edited

Loading

tigrannajaryan commented Nov 18, 2020

pavolloffay commented Nov 19, 2020

pavolloffay commented Nov 19, 2020

joe-elliott commented Nov 19, 2020

joe-elliott commented Nov 24, 2020

joe-elliott commented Nov 27, 2020

tigrannajaryan commented Nov 27, 2020

jpkrohling left a comment

jpkrohling Nov 30, 2020

joe-elliott Nov 30, 2020 •

edited

Loading

joe-elliott commented Nov 30, 2020

tigrannajaryan commented Nov 30, 2020

tigrannajaryan commented Nov 30, 2020

joe-elliott commented Dec 1, 2020

tigrannajaryan commented Dec 1, 2020

Fix --set flag #2162

Fix --set flag #2162

Conversation

joe-elliott commented Nov 16, 2020 • edited by tigrannajaryan Loading

codecov bot commented Nov 16, 2020 • edited Loading

Codecov Report

tigrannajaryan commented Nov 18, 2020

pavolloffay commented Nov 19, 2020

pavolloffay commented Nov 19, 2020

joe-elliott commented Nov 19, 2020

joe-elliott commented Nov 24, 2020

joe-elliott commented Nov 27, 2020

tigrannajaryan commented Nov 27, 2020

jpkrohling left a comment

Choose a reason for hiding this comment

jpkrohling Nov 30, 2020

Choose a reason for hiding this comment

joe-elliott Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

joe-elliott commented Nov 30, 2020

tigrannajaryan commented Nov 30, 2020

tigrannajaryan commented Nov 30, 2020

joe-elliott commented Dec 1, 2020

tigrannajaryan commented Dec 1, 2020

joe-elliott commented Nov 16, 2020 •

edited by tigrannajaryan

Loading

codecov bot commented Nov 16, 2020 •

edited

Loading

joe-elliott Nov 30, 2020 •

edited

Loading