docs: improvements to the user guide (#249)

Added: - Feature activation documentation (Tracer, Profiler, ASM) - Trace customization (`//dd:span`, `//dd:ignore`) Removed: - Details from `README.md`, replaced with pointers to the user guide --------- Co-authored-by: Nick Ripley <[email protected]>
DataDog · Aug 28, 2024 · aee181b · aee181b
1 parent 981f2c7
commit aee181b
Show file tree

Hide file tree

Showing 6 changed files with 268 additions and 60 deletions.
diff --git a/README.md b/README.md
@@ -113,63 +113,6 @@ In addition to this, Orchestrion only supports projects using [Go modules][go-mo
 > $ go get github.com/datadog/orchestrion@latest
 > ```
 
-## How it works
-
-The go toolchain's `-toolexec` feature invokes `orchestrion toolexec` with the complete list of arguments for each
-toolchain command invocation, allowing `orchestrion` to inspect and modify those before executing the actual command.
-This allows `orchestrion` to inspect all the go source files that contribute to the complete application, and to modify
-these to include instrumentation code where appropriate. Orchestrion uses [`dave/dst`][dave-dst] to parse and modify the
-go source code. Orchestrion adds `//line` directive comments in modified source files to make sure the stack trace
-information produced by the final application are not affected by additional code added during instrumentation.
-
-Since the instrumentation may use packages not present in the original code, `orchestrion` also intercepts the standard
-go linker command invocations to make the relevant packages available to the linker.
-
-[dave-dst]: https://github.com/dave/dst
-
-### Directive comments
-
-Directive comments are special single-line comments with no space between then `//` and the directive name. These allow
-influencing the behavior of Orchestrion in a declarative manner.
-
-#### `//dd:ignore`
-
-The `//dd:ignore` directive instructs Orchestrion not to perform any code changes in Go code nested in the decorated
-scope: when applied to a statement, it prevents instrumentations from being added to any component of this statement,
-and when applied to a block, or function, it prevents instrumentation from being added anywhere in that block or
-function.
-
-This is useful when you specifically want to opt out of instrumenting certain parts of your code, either because it has
-already been instrumented manually, or because the tracing is undesirable (not useful, adds too much overhead in a
-performance-critical section, etc...).
-
-#### `//dd:span`
-
-Use a `//dd:span` comment before any function or method to create specific spans from your automatically instrumented
-code. Spans will include tags described as arguments in the `//dd:span`. In order for the directive to be recognized,
-the line-comment must be spelled out with no white space after the `//` comment start.
-
-A function or method annotated with `//dd:span` must receive an argument of type `context.Context` or `*http.Request`.
-The context or request is required for trace information to be passed through function calls in a Go program. If this
-condition is met, the `//dd:span` comment is scanned and code is inserted in the function preamble (before any other
-code).
-
-Span tags are specified as a space-delimited series of `name:value` pairs, or as simple expressions referring to
-argument names (or access to fields thereof). All `name:value` pairs are provided as strings, and expressions are
-expected to evaluate to strings as well.
-
-```go
-//dd:span my:tag type:request name req.Method
-func HandleRequest(name string, req *http.Request) {
-	// ↓↓↓↓ Instrumentation added by Orchestrion ↓↓↓↓
-	req = req.WithContext(instrument.Report(req.Context(), event.EventStart, "function-name", "HandleRequest", "my", "tag", "type", "request", "name", name, "req.Method", req.Method))
-	defer instrument.Report(req.Context(), event.EventEnd, "function-name", "HandleRequest", "my", "tag", "type", "request", "name", name, "req.Method", req.Method)
-	// ↑↑↑↑ End of added instrumentation ↑↑↑↑
-
-	// your code here
-}
-```
-
 ## Supported libraries
 
 Orchestrion supports automatic tracing of the following libraries:
@@ -235,3 +178,12 @@ You can inspect everything Orchestrion is doing by adding the `-work` argument t
 the build will emit a `WORK=` line pointing to a working directory that is retained after the build is finished. The
 contents of this directory contains all updated source code Orchestrion produced and additional metadata that can help
 diagnosing issues.
+
+## More information
+
+Orchestrion's documentation can be found at [datadoghq.dev/orchestrion](https://datadoghq.dev/orchestrion); in
+particular:
+- the [user guide](https://datadoghq.dev/orchestrion/docs/) provides information about available configuration, and how
+  to customize the traces produced by your application;
+- the [contributor guide](https://datadoghq.dev/orchestrion/contributing/) provides more detailed information about how
+  orchestrion works and how to contribute new instrumentation to it.
diff --git a/docs/config/_default/hugo.toml b/docs/config/_default/hugo.toml
@@ -13,7 +13,7 @@ unsafe = true
 noClasses = false
 
 [[menu.main]]
-name = 'User Documentation'
+name = 'User Guide'
 pageRef = '/docs'
 weight = 2
 

diff --git a/docs/content/docs/built-in/_index.md b/docs/content/docs/built-in/_index.md
@@ -1,6 +1,6 @@
 ---
 title: Built-in Configuration
-weight: 10
+weight: 80
 prev: /docs/getting-started
 next: /docs/troubleshooting
 ---

diff --git a/docs/content/docs/custom-trace.md b/docs/content/docs/custom-trace.md
@@ -0,0 +1,183 @@
+---
+title: Trace Customization
+weight: 20
+prev: /docs/features
+next: /docs/built-in
+---
+
+Orchestrion offers serveral ways to control the traces produced by instrumented
+applications.
+
+## Prevent instrumentation of a section of code
+
+By default, `orchestrion` injects instrumentation _everywhere_ possible. This
+ensures users get the maximum possible coverage from their applications, as it
+removes the possibility of someone forgetting to instrument a particular call.
+
+There are however cases where you may want specific sections of your application
+to not be instrumented, either because they result in excessively verbose
+traces, or because those trace spans would be duplicated (for example, when
+using a custom-made `net/http` middleware stack).
+
+The `//dd:ignore` directive can be added anywhere in your application's code,
+and will disable all `orchestrion` instrumentation in the annotated syntax tree.
+
+For example:
+
+```go
+package demo
+
+import "net/http"
+
+//dd:ignore I don't want any of this to be instrumented, ever.
+func noInstrumentationThere() {
+  // Orchestrion will never add or modify any code in this function
+  // ... etc ...
+}
+
+func definitelyInstrumented() {
+  // Orchestrion may add or modify code in this function
+  // ... etc ...
+
+  //dd:ignore This particular server will NOT be instrumented
+  server := &http.Server {
+    Addr:    "127.0.0.1:8080",
+    Handler: internalServerHandler,
+  }
+
+  // Orchestrion may add or modify code further down in this function
+  // ... etc ...
+}
+```
+
+{{<callout emoji="⚠️">}}
+In certain cases, `orchestrion` adds instrumentation on the library side
+(sometimes referred to as _callee_ instrumentation; as opposed to _call site_
+instrumentation).
+
+In such cases, it is currently not possible to opt-out of instrumentation. This
+is the case for:
+- `net/http` client instrumentation
+- `github.com/gorilla/mux` middleware instrumentation
+{{</callout>}}
+
+## Creating custom trace spans
+
+Any function annotated with the `//dd:span` directive will result in a trace
+span being created around the function's execution. The directive can optionally
+provide custom span tags as `key:value` pairs (all parsed as literal strings):
+
+```go
+//dd:span tag-name:for other-tag:bar
+func tracedFunction() {
+  // This function will be represented as a span named "tracedFunction"
+}
+```
+
+### Result Capture
+
+Functions annotated with `//dd:span` which return an `error` value will
+automatically annotate spans with the returned `error` information if that is
+non-`nil`.
+
+```go
+package demo
+
+import "errors"
+
+//dd:span
+func failableFunction() (any, error) {
+  // This span will have error information attached automatically.
+  return nil, errors.ErrUnsupported
+}
+```
+
+### Operation Name
+
+The name of the operation (span name) is determined using the following
+precedence list (first non-empty is selected):
+
+- The `span.name` tag specified as a directive argument
+  ```go
+  //dd:span span.name:operationName
+  func tracedFunction() {
+    // This function will be represented as a span named "operationName"
+  }
+  ```
+- The name of the function (closures do not have a name)
+  ```go
+  //dd:span tag-name:for other-tag:bar
+  func tracedFunction() {
+    // This function will be represented as a span named "tracedFunction"
+  }
+  ```
+- The value of the very first tag from the directive arguments list
+  ```go
+  //dd:span tag-name:spanName other-tag:bar
+  tracedFunction := func() {
+    // This function will be represented as a span named "spanName"
+  }
+  ```
+
+### Trace Context Propagation
+
+If the annotated function accepts a {{<godoc "context" "Context" >}} argument,
+that context will be used for trace propagation. Otherwise, if the function
+accepts a {{<godoc "net/http" "Request" "*">}} argument, the request's context
+will be used for trace propagation.
+
+Functions that accept neither solely rely on _goroutine local storage_ for trace
+propagation. This means that traces may be split on _goroutine_ boundaries
+unless a {{<godoc "context" "Context" >}} or {{<godoc "net/http" "Request" "*">}}
+value carying trace context is passed across.
+
+Trace context carrying {{<godoc "context" "Context" >}} values are those that:
+
+- have been received by a `//dd:span` annotated function, as instrumentation
+  will create a new trace root span if if did not already carry trace context
+- are returned by:
+  - {{<godoc "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer" "StartSpanFromContext" >}}
+  - {{<godoc "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer" "ContextWithSpan" >}}
+
+```go
+package demo
+
+//dd:span
+func caller(ctx context.Context) {
+  wait := make(chan struct{}, 1)
+  defer close(wait)
+
+  // Weaving the span context into the child goroutine
+  go callee(ctx, wait)
+  <-wait
+}
+
+//dd:span
+func callee(ctx context.Context, done chan<- struct{}) {
+  done <- struct{}{}
+}
+```
+
+### Manual Instrumentation
+
+The {{<godoc "gopkg.in/DataDog/dd-trace-go.v1">}} library can be used to
+manually instrument sections of your code even when building with `orchestrion`.
+
+You can use APIs such as {{<godoc "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer" "StartSpanFromContext" >}}
+to create spans in any section of your code. This can be useful to delimit a
+specific section of your code with a span without having to refactor it in a
+separate function (which would allow the use of the `//dd:span` directive), or
+when you need to customize the span more than the `//dd:span` directive allows.
+
+{{<callout emoji="⚠️">}}
+You may also use integrations from the packages within
+{{<godoc "gopkg.in/DataDog/dd-trace-go.v1/contrib">}}, although this may result
+in duplicated trace spans if `orchestrion` supports automatic instrumentation of
+the same integration.
+
+This can be useful to instrument calls that `orchestrion` does not yet support.
+If you directly use integrations, we encourage you carefully review the
+[release notes](https://github.com/DataDog/orchestrion/releases) before
+upgrading to a new `orchestrion` release, so you can remove manual
+instrumentation that was made redundant as necessary.
+{{</callout>}}
diff --git a/docs/content/docs/features.md b/docs/content/docs/features.md
@@ -0,0 +1,73 @@
+---
+title: Feature Activation
+weight: 10
+
+prev: /getting-started
+next: /custom-trace
+---
+
+## Custom Go Tracer start-up
+
+All applications built using `orchestrion` automatically start the Datadog
+tracer at the beginning of the `main` function using the tracer library's
+default configuration. The recommended way to configure the tracer is by using
+the designated environment variables, such as `DD_ENV`, `DD_SERVICE`,
+`DD_VERSION`, etc... You can get more information on what environment variables
+are available in the [documentation][env-var-doc].
+
+If the `main` function is annotated with the `//dd:ignore` directive, the tracer
+will not be started automatically, and you are responsible for calling
+{{<godoc "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer" "Start" >}} with your
+preferred configuration options.
+
+[env-var-doc]: https://docs.datadoghq.com/tracing/trace_collection/library_config/go/#unified-service-tagging
+
+## Enabling the Go Profiler
+
+All applications built using `orchestrion` automatically start the Datadog
+continuous profiler if the `DD_PROFILING_ENABLED` environment variable is set
+to `1` or `true`. If profiling is enabled via the
+[Datadog Admission Controller][dd-adm-controller], `DD_PROFILING_ENABLED` can be
+set to `auto`.
+
+When enabled, the continuous profiler will activate the following profiles:
+- {{<godoc "gopkg.in/DataDog/dd-trace-go.v1/profiler" "CPUProfile" >}}
+- {{<godoc "gopkg.in/DataDog/dd-trace-go.v1/profiler" "HeapProfile" >}}
+- {{<godoc "gopkg.in/DataDog/dd-trace-go.v1/profiler" "GoroutineProfile" >}}
+- {{<godoc "gopkg.in/DataDog/dd-trace-go.v1/profiler" "MutexProfile" >}}
+
+[dd-adm-controller]: https://docs.datadoghq.com/containers/cluster_agent/admission_controller/?tab=datadogoperator
+
+## Enabling Application Security features
+
+Datadog Application Security (ASM) features are built into the tracer library,
+but need to be enabled at run-time. The [Enabling ASM for Go][asm-for-go]
+documentation explains how to enable Application Security for instrumented go
+applications.
+
+In the majority of cases, all that's needed is to set `DD_APPSEC_ENABLED` to `1`
+or `true`.
+
+{{<callout emoji="⚠️">}}
+Datadog's Application Security features are only supported on Linux (AMD64,
+ARM64) and macOS (AMD64, ARM64).
+
+On Linux platforms, the [Datadog in-app WAF][libddwaf] needs the `libc.so.6` and
+`libpthread.so.0` shared libraries to be available; even if `CGO_ENABLED=1`.
+
+If your are building your applications in environments where `CGO_ENABLED=0`,
+Application Security features are only available if you specify the `appsec`
+build tag (`orchestrion go build -tags=appsec .`).
+
+For more information, refer to the [Enabling ASM for Go][asm-for-go]
+documentation.
+
+[libddwaf]: https://github.com/DataDog/libddwaf
+[asm-for-go]: https://docs.datadoghq.com/security/application_security/threats/setup/threat_detection/go/
+{{</callout>}}
+
+Building applications with `orchestrion` allows you to maximize coverage for
+<abbr title="Runtime Application Self-Protection">RASP</abbr> features, such as
+automatic protection against SQL Injection attacks.
+
+[asm-for-go]: https://docs.datadoghq.com/security/application_security/threats/setup/threat_detection/go/
diff --git a/docs/content/docs/getting-started.md b/docs/content/docs/getting-started.md
@@ -3,7 +3,7 @@ title: "Getting Started"
 weight: 1
 
 prev: /docs
-next: /docs/built-in
+next: /features
 ---
 
 ## Requirements