KAFKA-18026: KIP-1112, document new config and update the Streams upgrade guide #17906
Conversation
docs/streams/upgrade-guide.html
Outdated
| </tr> | ||
| <tr> | ||
| <td>2.2.1 and<br>2.3.x and<br>2.4.x and<br>2.5.x and<br>2.6.x and<br>2.7.x and<br>2.8.x and<br>3.0.x and<br>3.1.x and<br>3.2.x and<br>3.3.x and<br>3.4.x and<br>3.5.x and<br>3.6.x and<br>3.7.x</td> | ||
| <td>2.2.1 and<br>2.3.x and<br>2.4.x and<br>2.5.x and<br>2.6.x and<br>2.7.x and<br>2.8.x and<br>3.0.x and<br>3.1.x and<br>3.2.x and<br>3.3.x and<br>3.4.x and<br>3.5.x and<br>3.6.x and<br>3.7.x and<br>3.8.x and<br>3.9.x</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td>compatible; requires message format 0.11 or higher;<br>enabling exactly-once v2 requires 2.4.x or higher</td> |
There was a problem hiding this comment.
@mjsax is this right? I always thought eosv2 needed brokers to be 2.5 or higher, not 2.4. Am I misremembering this or is the matrix wrong?
This error message could also be wrong I suppose but this log in StreamThread at least seems to agree it's 2.5:
log.error("Shutting down because the Kafka cluster seems to be on a too old version. " +
"Setting {}=\"{}\" requires broker version 2.5 or higher.",
There was a problem hiding this comment.
ok according to the KafkaProducer javadocs for #sendOffsetsToTransaction it's 2.5 so I think that's right. I'll fix it here but it might be good to go back and fix this for the previous javadocs
There was a problem hiding this comment.
Yes, EOSv2 does require 2.5 brokers.
| </tr> | ||
| <tr> | ||
| <td>0.10.0.x</td> | ||
| <td>compatible</td> | ||
| <td>compatible</td> | ||
| <td>compatible</td> | ||
| <td>compatible</td> |
There was a problem hiding this comment.
@ijuma just to confirm my reading of KIP-896, the change only affects older clients compatibility with newer brokers, not the other way around, right? In other words a 0.11 client could still run against a 4.0 broker?
There was a problem hiding this comment.
In other words a 0.11 client could still run against a 4.0 broker?
I don't think so. According to KIP-896:
Users relying on clients not included in this document will have to ensure they are using a version that does not rely on protocol API versions that are being removed as part of this proposal.
This means that broker version 4.x can only communicate with clients version 2.1 and above.
There was a problem hiding this comment.
Correct @chia7712. This raises the minimum version for everything to 2.1.
There was a problem hiding this comment.
thanks for confirming -- glad I checked 🙂
|
@ableegoldman Could you please take a look at the screenshot above? It seems we are missing some cells in the table. |
|
@ableegoldman, I merged an update to the compatibility matrix with the missing 3.8.x and 3.9.x entries. I did that to conclude all release steps for 3.9 for Streams. Sorry, if that created the inconvenience to rebase your PR with conflicts. |
|
dammit Bruno 😠 (@cadonna just kidding, no worries) |
@chia7712 do you mean that some cells are literally non-existent, or are you referring to the ones that are just empty? I'll assume it's the latter, because I was confused about the empty cells at first too when I was working on this, but I realized they are intentional. If it doesn't explicitly say "compatible" (ie is empty), then it's not compatible. Personally I think just writing "incompatible" for these cells would convey the information much better. I can include that in this PR if others agree /cc @cadonna @mjsax |
This PR covers all the docs for KIP-1112, including the new config and a note about the new APIs in the 4.0 section of the upgrade guide.
This also fixes/updates some unrelated parts of the upgrade guide that were out-of-date, such as the broker compatibility matrix