Further flink kafka optimizations #408
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wagmarcel
force-pushed
the
Further-Flink-Kafka-optimizations
branch
3 times, most recently
from
9a79cf9
to
74d6798
Compare
… checkpoint settings to leave it to local job configuration - Upgraded from Flink 1.16.1 to 1.16.2 testGateway - Rocksdb metrics are inserted into flink-config to allow better debugging and optimization - Removed local Checkpoint configurations. This will be done by the jobs in future - Add debug mode to to Gateway: Do not delete the deployment directories in /tmp when DEBUG is set to 'true'
…ms, improve udf handling and flink job configuration - A problem with the Strimzi TopicOperator: It creates topic CRDs also from automatically created topics which could conflict with helmfile objects. To avoid that the flini-deploy must disable topicOperator remove all kafka objects, deploy the new flink-version and enable the topicOperator afterwards. - SHACL.ttl now contains the statetime udf for the OEE time calculation - Create_udf is now using regexp to parse filename to allow filenames with underscore - SQLStatementset now contains the rocksdb configuration and disables the upsert Sink Materializer (as it creates high load)
- Configure Kafka in two profiles "default" and production - Restrict Production retention time to 8h and default to 1h - Production allows compression with gzip, default is without compression
- Problem with Flink: left joins sometimes give early results wich are retracted few milliseconds later. This creates short term alerts on Alerta which creates sometimes a lot of unneccessary alerts flood and confuses the user. - This is solved by adding Flink later arrival filtering: Alerts which are 200ms late are grouped with the previous window. - This creates on the other hand a more complicated E2E testing case.
wagmarcel
force-pushed
the
Further-Flink-Kafka-optimizations
branch
from
74d6798
to
367152e
Compare
oguzcankirmemis
approved these changes
Jul 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.