out_stackdriver: support stackdriver trace span (#4223)

[0Delta]

logentry will be able to bind to span in the Google Cloud Web UI.

Specifically Cloud Trace (Stackdriver),
the logging.googleapis.com/traceId field is
moved to the top of the log entry and renames it to traceId.

Signed-off-by: 0Delta [email protected]

#4223

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[0Delta]

leak check is here.

$ valgrind ./bin/fluent-bit -i stdin -o stackdriver -f kubernetes -f 1
==13447== Memcheck, a memory error detector
==13447== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==13447== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==13447== Command: ./bin/fluent-bit -i stdin -o stackdriver -f kubernetes -f 1
==13447==
Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/10/25 14:03:49] [ info] [engine] started (pid=13447)
[2021/10/25 14:03:49] [ info] [storage] version=1.1.4, initializing...
[2021/10/25 14:03:49] [ info] [storage] in-memory
[2021/10/25 14:03:49] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/25 14:03:49] [ info] [cmetrics] version=0.2.2
{"caller":"build/req_sender_generated.go:38","message":"","severity":"ERROR","time":"2021-10-25T01:07:30.1052899Z","logging.googleapis.com/labels":{"k8s-pod/app":"dummy","k8s-pod/pod-template-hash":"dummyhashhashhash","userId":"test-user","sessionId":"test-session","type":"req","subtype":""},"logging.googleapis.com/local_resource_id":"k8s_container.default..main","logging.googleapis.com/trace":"00000000000000000000000000000000","logging.googleapis.com/spanId":"0000000000000000","logging.googleapis.com/http_request":{"requestMethod":"GET","requestUrl":"/","requestSize":0,"status":200,"responseSize":2,"userAgent":"curl/7.68.0","remoteIp":"172.17.0.1:52836","serverIp":"<nil>","referer":"","latency":"1.380293600s","protocol":"HTTP/1.1"},"stacktrace":"mylog.requestLog.Log\n\t/build/req_sender_generated.go:38\nmylog.requestLog.LogAuto\n\t/build/req_sender_generated.go:71\nmylog/addon/echo.(*requestLog).Middleware.func1\n\t/build/addon/echo/req_echo.go:49\ngithub.com/labstack/echo.(*Echo).ServeHTTP\n\t/go/pkg/mod/github.com/labstack/[email protected]+incompatible/echo.go:593\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2878\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1929"}
{"caller":"build/req_sender_generated.go:38","message":"","severity":"ERROR","time":"2021-10-25T01:07:30.1052899Z","logging.googleapis.com/labels":{"k8s-pod/app":"dummy","k8s-pod/pod-template-hash":"dummyhashhashhash","userId":"test-user","sessionId":"test-session","type":"req","subtype":""},"logging.googleapis.com/local_resource_id":"k8s_container.default..main","logging.googleapis.com/trace":"00000000000000000000000000000000","logging.googleapis.com/http_request":{"requestMethod":"GET","requestUrl":"/","requestSize":0,"status":200,"responseSize":2,"userAgent":"curl/7.68.0","remoteIp":"172.17.0.1:52836","serverIp":"<nil>","referer":"","latency":"1.380293600s","protocol":"HTTP/1.1"},"stacktrace":"mylog.requestLog.Log\n\t/build/req_sender_generated.go:38\nmylog.requestLog.LogAuto\n\t/build/req_sender_generated.go:71\nmylog/addon/echo.(*requestLog).Middleware.func1\n\t/build/addon/echo/req_echo.go:49\ngithub.com/labstack/echo.(*Echo).ServeHTTP\n\t/go/pkg/mod/github.com/labstack/[email protected]+incompatible/echo.go:593\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2878\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1929"}
^C[2021/10/25 14:04:06] [engine] caught signal (SIGINT)
==13447==
==13447== HEAP SUMMARY:
==13447==     in use at exit: 577,687 bytes in 1,312 blocks
==13447==   total heap usage: 2,459 allocs, 1,147 frees, 1,034,072 bytes allocated
==13447==
==13447== LEAK SUMMARY:
==13447==    definitely lost: 0 bytes in 0 blocks
==13447==    indirectly lost: 0 bytes in 0 blocks
==13447==      possibly lost: 288 bytes in 1 blocks
==13447==    still reachable: 577,399 bytes in 1,311 blocks
==13447==         suppressed: 0 bytes in 0 blocks
==13447== Rerun with --leak-check=full to see details of leaked memory
==13447==
==13447== For lists of detected and suppressed errors, rerun with: -s
==13447== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[0Delta]

debug log is here.

$ ./bin/fluent-bit -i stdin -o stackdriver -f kubernetes -f 1
Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/10/25 14:55:55] [ info] [engine] started (pid=14638)
[2021/10/25 14:55:55] [ info] [storage] version=1.1.4, initializing...
[2021/10/25 14:55:55] [ info] [storage] in-memory
[2021/10/25 14:55:55] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/25 14:55:55] [ info] [cmetrics] version=0.2.2
[2021/10/25 14:55:58] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2021/10/25 14:55:58] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2021/10/25 14:55:58] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2021/10/25 14:56:00] [ info] [sp] stream processor started
{"caller":"build/req_sender_generated.go:38","message":"","severity":"ERROR","time":"2021-10-25T01:07:30.1052899Z","logging.googleapis.com/labels":{"k8s-pod/app":"dummy","k8s-pod/pod-template-hash":"dummyhashhashhash","userId":"test-user","sessionId":"test-session","type":"req","subtype":""},"logging.googleapis.com/local_resource_id":"k8s_container.default..main","logging.googleapis.com/trace":"00000000000000000000000000000000","logging.googleapis.com/spanId":"0000000000000000","logging.googleapis.com/http_request":{"requestMethod":"GET","requestUrl":"/","requestSize":0,"status":200,"responseSize":2,"userAgent":"curl/7.68.0","remoteIp":"172.17.0.1:52836","serverIp":"<nil>","referer":"","latency":"1.380293600s","protocol":"HTTP/1.1"},"stacktrace":"mylog.requestLog.Log\n\t/build/req_sender_generated.go:38\nmylog.requestLog.LogAuto\n\t/build/req_sender_generated.go:71\nmylog/addon/echo.(*requestLog).Middleware.func1\n\t/build/addon/echo/req_echo.go:49\ngithub.com/labstack/echo.(*Echo).ServeHTTP\n\t/go/pkg/mod/github.com/labstack/[email protected]+incompatible/echo.go:593\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2878\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1929"}
^C[2021/10/25 14:56:46] [engine] caught signal (SIGINT)
[2021/10/25 14:56:46] [ warn] [engine] service will stop in 5 seconds
[2021/10/25 14:56:50] [ info] [engine] service stopped

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[edsiper]

@qingling128 can you review this one from usability perspective ?

[qingling128]

LGTM

[edsiper]

CI is failing due to a potential memory leak in the changes:

https://github.com/fluent/fluent-bit/runs/4790455332?check_suite_focus=true#step:5:3982

[0Delta]

I missed cleanup function.
fixed 60bd539 .

[0Delta]

Could you approve for run CI ?
test is fine in my local.

[patrick-stephens]

I can't see anything to approve here, do you want to give it a poke @0Delta - it may have been some error we've fixed or just some weird Github action issue. If you can push a new commit to refresh things it may spring to life.

[edsiper]

@patrick-stephens can you check pls why CI is stuck ?

[patrick-stephens]

It looks like the problem we had a while back with missing skipped options but that was fixed a long time ago. I notice the fork is a bit behind so it might be worth rebasing to give it a poke @0Delta

[patrick-stephens]

Ah, it appears to be an issue with pull_request_target for the int tests again: https://github.com/fluent/fluent-bit/actions/runs/1952674244
Startup failure as the reusable workflow parameters have changed but this PR cannot change them.

Trying to figure out why as the error itself is incorrect.
Typo introduced on #5000 should be resolved by #5022 now.

[edsiper]

@0Delta can you please rebase your PR on top of master so the CI can pick the changes ?

[patrick-stephens]

The int test ones use pull_request_target so I think they should run anyway now: https://github.com/fluent/fluent-bit/actions/runs/1953228119
However it would be good to rebase to make sure none of the other workflows are outdated, particularly the skipped issue.

[0Delta]

Apparently, the code that this PR is trying to change has been significantly changed by #5044.
I will try to deal with it, but I think it will take some time. Can you give me a little more time?

[0Delta]

I rebase and the test feels good.
Could you check it?

[l2cup]

Could we revive this? Wanted to open this myself, came across this PR.

[braydonk]

This is now possible today. The fields were renamed since this PR was made, but trace, spanId, and traceSampled are all supported by this plugin now.