Introduce Buffer Strategy agent config options

[powersj]

Use Case

Introduce a new agent config option which specifies a specific buffer strategy. It would have three options:

• memory: current behavior, in memory, limited, overflows drop metrics
• write-through: writes metrics to disk, all to disk in line protocol
• mem+disk: only writes to disk on overflow, an entire buffer at a time.

The key items to consider are:

• tracking metrics: when to acknowledge them with the write-through.
• ensuring we write the oldest to disk first
• write batch size at a time to disk
• ensure the rate-limiting PR lands as well

The config would take two parameters:

• the method as defined above
• the directory to write metrics to when writing to disk

These additional options could cause the disk to fill up and that is on the user to figure out and avoid.

Expected behavior

Allow the user to determine which strategy to use for the buffer.

Actual behavior

Only use in-memory buffer and when it overflows, metrics are discarded.

Additional info

No response

[powersj]

next steps: get draft PR up to start testing. Can do this one option at a time.

[srebhan]

@powersj now that we do have the spec thing, should we first get a PR for the spec and then put up the actual PR referencing this spec? Doesn't exclude a draft-PR before the spec though...

[powersj]

Yes, I agree, it would be good to at least have a PR for the draft.

[knollet]

We have this overflow problem, too.
+1 for implementation.

[powersj]

#15564 has landed and is available in nighties. We will keep the older issue open while we collect feedback as we do consider this experimental.