[exporter/file] Add buffer flush interval

[loomis-relativity]

Component(s)

exporter/file

Is your feature request related to a problem? Please describe.

The observability team at my company maintains a custom version of the collector. The custom collector includes custom receivers, processors, and exporters for all OpenTelemetry data types. The file exporter is used to verify the full data flow by comparing known inputs to their expected output payloads.
In v0.70.0, the file exporter moved to a buffered writer. This breaks the tests because we cannot reliably ensure that the payloads have been written to disk when the check happens. Having the buffered output as the default makes sense; however, we'd also like a mechanism for turning off the buffering for these types of use cases.

Describe the solution you'd like

Add a parameter to turn off the buffering within the file exporter.

Describe alternatives you've considered

We've discussed using the OpenTelemetry testbed or using a real backend for collecting the data. However, both of these solutions would significantly complicate the testing environment.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

This breaks the tests because we cannot reliably ensure that the payloads have been written to disk when the check happens.

Can you elaborate on this?

cc: @MovieStoreGuy, I believe #17543 is the relevant change here so maybe you have an opinion too.

[loomis-relativity]

@djaglowski To test the full data flow with our custom collector, we deploy it via k8s and compare known payloads to expected outputs. The actual outputs are written to the local disk in a k8s pod via the file exporter. These tests fail with the buffering turned on because the output isn't large enough to fill the buffers, so the actual files on disk remain empty. To be clear, these are our internal, custom tests, not the standard OpenTelemetry collector tests.

[loomis-relativity]

When using the file rotation, a writer from lumberjack is used. This implementation writes the payloads to disk with a reasonable delay for testing. For others that might need this, just change the file exporter configuration to:

file:
  path: "./foo"
  rotation:

[djaglowski]

I'm reopening this because I think it is a valid issue that should be addressed in some way. I'm glad there is a workaround for now but I'm still hoping @MovieStoreGuy will weigh in on whether this should be a boolean, or timer, or something else.

[damemi]

Just want to +1 that I hit this issue too. I thought something was wrong with my file exporter config, because I could see output from the logging exporter but nothing written to the file. It would also be helpful if the file exporter had more logging to show that it's processing work.

#18251 (comment) fixed it for me.

As a side note, I wonder why this hasn't hit the e2e tests yet (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/.github/workflows/e2e-tests.yml), or what is different in their setup that they don't hit this issue. I'm working on a similar test that follows basically the same process based on that.

[MovieStoreGuy]

I'm reopening this because I think it is a valid issue that should be addressed in some way. I'm glad there is a workaround for now but I'm still hoping @MovieStoreGuy will weigh in on whether this should be a boolean, or timer, or something else.

Sorry I am late to the party. periodic flushing making sense to me since it is the same as the batch exporter.