-
Notifications
You must be signed in to change notification settings - Fork 956
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
0x00 never supported in Postgres, automatic stripping #1216
Comments
is anyone interested in this change? We could do this in
that replaces \u0000 chars with nothing… |
I would like to work on this one. I will try to come with PR during the weekend. |
I created a PR: #1264 |
@oleksandrkovalenko any news on the PR #1264 being accepted? |
I don't know anything more than PR itself, which is silence. Maybe we could tag somebody, but I don't know who it could be. Our workaround if you are interested: We had only one such case and have manually moved the offset in Kafka to unblock processing. Also, we requested changes in the source MySQL database to get new records in Kafka without \0x00. As we are running Kafka connect using our custom image, we are considering replacing the original connector from maven-central with our custom-built connector if the case occurs again. By the way, I just realised that I didn't try to build a jar from the sources. All in all not a blocker for us, just learning about \0x00 differences between Postgres and MySQL. |
Thank you, @oleksandrkovalenko, Your solution resolved the issue for me. It's unclear why this problem occurred in our case and not for others postgres users. |
im considering just writing an SMT which removes the character, what do you'll think? p.s. even if someone was to review and approve, there appears to be a jenkins CI failure. |
I know, but there are no errors logged, so I am not sure what should be fixed. Any thought? @cyberjar09 |
sorry @oleksandrkovalenko have not checked on whats the failure but FWIW, I wrote the SMT which fixes my problem, if you are interested here is the link: https://github.com/cyberjar09/StripUnicodeNullTransform |
I've written an SMT that can replace the |
Hi, @cyberjar09 This https://github.com/cyberjar09/StripUnicodeNullTransform using in your SMT sink? @roadSurfer your have the code and example? |
I was using it, but we moved away from debezium as a solution so ive decommissioned it since |
Yes, I have an updated example here: https://github.com/roadSurfer/StripUnicodeNullTransform It's little more than a clone of @cyberjar09's with some extra flexibility and changes based on SonarLint reports. We actually ditched the SMT in favour of have the Processors strip the |
When copying over data from MySQL to Postgres I hit the exception:
According to: https://stackoverflow.com/a/1348551, 0x00 is never a supported character
Question
Since we cannot change the source data (being a downstream only service), what could be the best strategy to remove the 0x00 characters?
How could/Should this be baked into JDBCSinkConnector?
The text was updated successfully, but these errors were encountered: