onBackpressureBufferToFile review

[davidmoten]

The ability to buffer streams to disk has been something that I've wondered about for a while. It was mentioned in #3751 and I thought I'd run with an implementation in rxjava-extras documented here.

Example usage relying on java.io serialization:

Observable<String> o = 
  Observable
    .just("a", "b", "c")
    .compose(Transformers.<String>onBackpressureBufferToFile());

A bit of a kitchen sink example:

Observable
  .just("a", "b", "c")
  .compose(
    Transformers.onBackpressureBufferToFile(
        serializer, 
        Schedulers.computation(), 
        Options
          .fileFactory(fileFactory)
          .bufferSizeBytes(1024)
          .rolloverEvery(10000)
          .rolloverSizeBytes(10000000)
          .delayError(false)
          .build()))

I'd love to get review of this new operator in terms of

• what use cases have you got? (volume, serialization method, serialized size, rate, platform, constraints)
• overall approach (would you throw this implementation out and start again!)
• API
• correctness of code (a big one because of the numerous sections of code subject to concurrency)
• testing on different OS (currently just tested on linux)
• performance (I've favoured correctness initially)
• where should this operator live?
• anything else you think of

The code is in the master branch and runtime jar is on Maven Central as described in rxjava-extras README.

The implementation has decent performance (~ 40MB/sec performance with 1K messages) and has a bunch of tests including concurrency tests that are passing some long runs.

I've opened an issue davidmoten/rxjava-extras#9 for comments though feel free to open up new ones.

Thanks!

[akarnokd]

Thanks for the interesting operator. Let's continue the discussion in rxjava-extras.