Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][broker] forbid uploading BYTES schema with admin API #18995

Merged
merged 5 commits into from
Dec 28, 2022

Conversation

labuladong
Copy link
Contributor

@labuladong labuladong commented Dec 20, 2022

Fixes #18825

Motivation

Admin API can upload BYTES schema and store a NONE schema in the schema registry, which is inconsistent with the client side. After discussion, we decided to forbid users from uploading BYTES schema.

Mail list: https://lists.apache.org/thread/672zmptfblwjmrf9z8336mk12r7csngf

Modifications

Add a check to reject BYTES schema upload.

Verifying this change

  • Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: labuladong#17

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Dec 20, 2022
@Technoboy- Technoboy- added this to the 2.12.0 milestone Dec 21, 2022
@Technoboy- Technoboy- closed this Dec 21, 2022
@Technoboy- Technoboy- reopened this Dec 21, 2022
@codecov-commenter
Copy link

codecov-commenter commented Dec 21, 2022

Codecov Report

Merging #18995 (0adbbce) into master (feb3cb4) will decrease coverage by 0.47%.
The diff coverage is 6.25%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #18995      +/-   ##
============================================
- Coverage     46.35%   45.88%   -0.48%     
- Complexity     8939    10369    +1430     
============================================
  Files           597      711     +114     
  Lines         56858    69461   +12603     
  Branches       5905     7453    +1548     
============================================
+ Hits          26357    31871    +5514     
- Misses        27616    33919    +6303     
- Partials       2885     3671     +786     
Flag Coverage Δ
unittests 45.88% <6.25%> (-0.48%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...elayed/bucket/BookkeeperBucketSnapshotStorage.java 0.00% <0.00%> (ø)
...rg/apache/pulsar/broker/delayed/bucket/Bucket.java 0.00% <0.00%> (ø)
...yed/bucket/BucketSnapshotPersistenceException.java 0.00% <0.00%> (ø)
...d/bucket/BucketSnapshotSerializationException.java 0.00% <0.00%> (ø)
.../pulsar/broker/service/BrokerServiceException.java 45.45% <0.00%> (-0.85%) ⬇️
...va/org/apache/pulsar/client/impl/ConsumerImpl.java 15.06% <0.00%> (-0.03%) ⬇️
...he/pulsar/client/impl/MultiTopicsConsumerImpl.java 22.80% <0.00%> (+0.01%) ⬆️
...ache/pulsar/client/impl/ZeroQueueConsumerImpl.java 0.00% <0.00%> (ø)
...va/org/apache/pulsar/client/impl/ConsumerBase.java 21.51% <6.66%> (-0.42%) ⬇️
...va/org/apache/pulsar/broker/service/ServerCnx.java 42.84% <21.05%> (-6.14%) ⬇️
... and 208 more

@labuladong
Copy link
Contributor Author

Oh there is another test case fail... Let me fix it.

@labuladong
Copy link
Contributor Author

PTAL @codelipenghui @congbobo184

@@ -114,6 +114,10 @@ public CompletableFuture<SchemaVersion> deleteSchemaAsync(boolean authoritative,
}

public CompletableFuture<SchemaVersion> postSchemaAsync(PostSchemaPayload payload, boolean authoritative) {
if (SchemaType.BYTES.name().equals(payload.getType())) {
throw new RestException(Response.Status.NOT_ACCEPTABLE,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please return failed future to avoid additional try-catch when invoke this method.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Good catch!

+1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, understand, fixed.

@labuladong
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

Comment on lines +117 to +120
if (SchemaType.BYTES.name().equals(payload.getType())) {
return CompletableFuture.failedFuture(new RestException(Response.Status.NOT_ACCEPTABLE,
"Do not upload a BYTES schema, because it's the default schema type"));
}
Copy link
Member

@michaeljmarshall michaeljmarshall Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work if the schema is not currently bytes and someone wants to change it to bytes?

cc @mattisonchao @codelipenghui @congbobo184 @labuladong

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljmarshall I think you can delete all the current schemas of the topic. At this time, it's schema defaults to bytes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BYTES schema doesn't need to upload, when to create producer or consumer with bytes schema, the broker will not check the schema compatibility check

@cbornet
Copy link
Contributor

cbornet commented Mar 21, 2023

This is a breaking change.
It broke the Flink SQL Connector : streamnative/flink#270

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker doc-not-needed Your PR changes do not impact docs ready-to-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Upload BYTES schema but got NULL schema type
9 participants