fix: message.max.bytes should default to 1048576 not 1 MB

[puellanivis]

https://kafka.apache.org/documentation/#brokerconfigs_message.max.bytes

Says this default value should be 1048588, not 1 megabyte (and also weirdly not 1 Mebibyte 🤷‍♀️ ) matching these expected values is important, as one language could publish an event that another language would then refuse to republish.

The produced error message ConfigurationError(fmt.Sprintf("Attempt to produce message larger than configured Producer.MaxMessageBytes: %d > %d", size, p.conf.Producer.MaxMessageBytes) is also poorly designed to allow for event handling of that message. It would require an errors.As followed by a strings.HasPrefix(…) which is really not what Go recommends to handle errors (that is, direct string manipulation, which is inherently fragile as the error string should be easily changeable.) This change doesn’t really address this issue, but I wanted to mention it.

[dnwe]

@puellanivis thanks for pointing out this discrepancy with current versions of Kafka. However, it is slightly more nuanced as this value has had different default overs the years.

The Java producer since v0.8.2 has always had max.request.size of 1048576 (the base 2 version of 1 MB (MiB))

Up to Kafka v0.8.2 message.max.bytes was 1000000 server side
From Kafka v0.9.0 up to Kafka v2.4 it was then 1000012 (adding 12 bytes for log overhead from KAFKA-506)
From Kafka v2.5 onwards it has been a default 1048588 (the base 2 value of 1 MB (MiB) + 12 bytes for overhead again) based on KAFKA-4203 updating the broker config default to match the Java producer default

[puellanivis]

Yeah, I’m totally sure that it’s been through a lot of different values. 😰 Just thinking matching whatever is current is useful.

I suppose in our specific use case, we should probably set it to some value greater than whatever anyone would pick as a default. That way we can avoid “chasing a default value”.

[dnwe]

I think it probably makes sense for us to match the Java producer, which uses 1048576

[puellanivis]

I suppose I need to rebase and add sign offs into the commit anyways… actually, I’d probably be faster to just open a new PR with a signed commit message. 😂

[dnwe]

No worries, I've done the rebase and updated the value to 1024*1024 as discussed

[puellanivis]

Is there anything blocking now?

[dnwe]

No further holdups, I was just waiting for you to reply with a thumbsup before I went ahead and merged 😄

[puellanivis]

😆 I guess in this case, I could have thumbs upped in a full comment rather than just as a reaction.