Log service

[superfell]

There should be a server implementation of the logspec service. Beam was moving from using Kafka to using this abstract log service definition for its log. The Kafka client needs some work for production usage.

[ghost]

I am very interested in the abstraction because I would like to support NATS instead of Kafka.
Just raising this in case others might be thinking to do it.

[superfell]

There's 2 abstraction points for the log that can be used.

• The gRPC log spec interface, a log service can implement this, and Beam can use this directly.
• Or you can implement the BeamLog go interface (which is how Kafka is currently integrated)

[superfell]

NATS looks interesting, the NATS streaming service looks it could work. However one thing i've seen is that Publishing a message doesn't return its sequence number in the Log. This is problematic as the writers to the log in the API tier need to know the sequence / log index number writes ended up at, for example, transactions are identified by their log seq #.
The guid's that the AsyncPublish call expose aren't included in the messages the subscriber sees, so you couldn't even have it use a private subscriber to correlate on the guid to then get the seq#.
Its possible i missed something, but looks like it wouldn't be possible to use the NATS streaming service with its current client API as a log for Beam.

[superfell]

Facebook's LogDevice looks like it'd be a good fit for the log service.

[joeblew99]

@superfell
NATS Streaming has those things and i agree they are needed for your use case.
Its explained here:
https://nats.io/documentation/streaming/nats-streaming-protocol/#PUBACK

Also the GO team are making a Enterprise framework and PUBSUB is in it.
https://github.com/google/go-cloud/tree/master/pubsub

[joeblew99]

AH its does not support ACK and NACK.
https://github.com/google/go-cloud/blob/master/pubsub/natspubsub/nats.go#L395
SO basically they have NATS but not NATS Streaming support.

[superfell]

Its good its in the protocol, its unfortunate that its not reflected in the go api i looked at
https://github.com/nats-io/go-nats-streaming