Code

feat: add KafkaBatchPublisher#19

Code

Merged

farbodahm merged 12 commits intomainDataChefHQ/sparkle:mainfrom feature/14-add-kafka-batch-publisherDataChefHQ/sparkle:feature/14-add-kafka-batch-publisherCopy head branch name to clipboardOct 3, 2024

+355-88

Commits

Commits on Oct 3, 2024

• feat: add kafka-ui service so I can see what goes on

  Show description for 2da9db5
  

  Federico Zambelli

  committed
  2da9db5

  Copy full SHA for 2da9db5

• test: separate batch from stream writer

  

  Federico Zambelli

  committed
  aeaed3d

  Copy full SHA for aeaed3d

• feat: add chispa pkg for easy spark DF comparisons

  Show description for c558638
  

  Federico Zambelli

  committed
  c558638

  Copy full SHA for c558638

• feat(kafkaWriter): add batch publisher

  

  Federico Zambelli

  committed
  03457fe

  Copy full SHA for 03457fe

• test(kafkaWriter): add tests for batch publisher

  

  Federico Zambelli

  committed
  bf8c9e8

  Copy full SHA for bf8c9e8

• chore: ignore mypy warning for dev dependency "chispa"

  Show description for a96f6a2
  

  Federico Zambelli

  committed
  a96f6a2

  Copy full SHA for a96f6a2

• fix: check if Kafka columns in writer are *exactly* {key, value}

  Show description for 33ce26e
  

  2 peopleauthored and

  Federico Zambelli

  committed
  33ce26e

  Copy full SHA for 33ce26e

• Update tests/conftest.py

  Show description for d148db7
  

  2 peopleauthored and

  Federico Zambelli

  committed
  d148db7

  Copy full SHA for d148db7

• chore: add explanation to pytest flags

  

  Federico Zambelli

  committed
  63a2851

  Copy full SHA for 63a2851

• chore: add logging to some tests

  

  Federico Zambelli

  committed
  c108e70

  Copy full SHA for c108e70

• fix: add missing colon

  

  Federico Zambelli

  committed
  7791ae0

  Copy full SHA for 7791ae0

• chore: add script to remove unused containers

  

  Federico Zambelli

  committed
  f992d87

  Copy full SHA for f992d87