feat: Adding support for Cassandra and Scylla

[fruch]

This add the support for those cassandra based dbs
and their drivers, cassandra-driver, scylla-driver

Ref: https://cassandra.apache.org/
Ref: https://www.scylladb.com/
Ref: https://pypi.org/project/cassandra-driver/
Ref: https://pypi.org/project/scylla-driver/

[fruch]

Hi @tillahoffmann and @SergeyPirogov, pretty please for approving the workflow to start ?

FYI, it can be eased a bit, the approval part:
https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#configuring-required-approval-for-workflows-from-public-forks

[yakimka]

@fruch hello. Can you update this PR?

[tillahoffmann]

Thanks for the additions and apologies for the slow reply. A few things to get this integrated:

• Rebase on master.
• Regenerate requirement files.
• Add cassandra and scylla to the test-components in .github/workflows/main.yml.

[fruch]

Thanks for the additions and apologies for the slow reply. A few things to get this integrated:

• Rebase on master.
• Regenerate requirement files.
• Add cassandra and scylla to the test-components in .github/workflows/main.yml.

I've rebased and added the test to CI

but the building of requirements is failing:

There are incompatible versions in the resolved dependencies:
  protobuf<4.0.0dev (from google-cloud-pubsub==1.7.2->testcontainers===dev->-r requirements.in (line 1))
  protobuf<5.0.0dev,>=3.20.1 (from google-api-core[grpc]==2.10.1->google-cloud-pubsub==1.7.2->testcontainers===dev->-r requirements.in (line 1))
  protobuf<5.0.0dev,>=3.15.0 (from googleapis-common-protos[grpc]==1.56.4->google-api-core[grpc]==2.10.1->google-cloud-pubsub==1.7.2->testcontainers===dev->-r requirements.in (line 1))
  protobuf>=4.21.3 (from grpcio-status==1.49.0->google-api-core[grpc]==2.10.1->google-cloud-pubsub==1.7.2->testcontainers===dev->-r requirements.in (line 1))

I think pinning of google-cloud-pubsub is problematic, and google packages are conflicting each other.
I'm trying to unpin it, and regenerate (but it quite a long process, meanwhile it worked for python3.7)

[fruch]

@yakimka @tillahoffmann waiting for approval for running the CI

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (main@ca65a91). Click here to learn what that means.

❗ Current head dd6520d differs from pull request most recent head 1114f4d. Consider uploading reports for the commit 1114f4d to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #167   +/-   ##
=======================================
  Coverage        ?   87.81%           
=======================================
  Files           ?       32           
  Lines           ?      886           
  Branches        ?       58           
=======================================
  Hits            ?      778           
  Misses          ?       78           
  Partials        ?       30           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tillahoffmann]

We may have to get #209 merged first to get around the google dependency issue.

[tillahoffmann]

I have a relatively strong preference to remove the cache for two reasons. First, it's more code to maintain. Second, I don't think the caching will help in most real-world scenarios.

I appreciate that runtime is different across different machines, but getting a single image takes about 4ms on my machine. Using the cache, getting a single image takes about 300ms. It'll of course be quicker the next time images are fetched, but we'd have to use at least 75 containers per test suite to make the cache worthwhile. To have a delay of one second over the cached version, we'd have to run about 325 containers. I'd argue that for a test suite that runs more than 300 docker containers a delay of one second is negligible. But let me know if I misunderstood/miscalculated.

[fruch]

We may have to get #209 merged first to get around the google dependency issue.

Seems that just removing the pinning helped

[fruch]

I have a relatively strong preference to remove the cache for two reasons. First, it's more code to maintain. Second, I don't think the caching will help in most real-world scenarios.

I appreciate that runtime is different across different machines, but getting a single image takes about 4ms on my machine. Using the cache, getting a single image takes about 300ms. It'll of course be quicker the next time images are fetched, but we'd have to use at least 75 containers per test suite to make the cache worthwhile. To have a delay of one second over the cached version, we'd have to run about 325 containers. I'd argue that for a test suite that runs more than 300 docker containers a delay of one second is negligible. But let me know if I misunderstood/miscalculated.

Didn't understand where you are going with the calculation here, for me getting a docker image out of dockerhub can take a minute or two. I was more talking about pip-compile that in case of conflicts can download all the versions of a package to try solving it, (I know it, cause I'm using it in in one of my projects in my day job),

And as you said I don't think it's worth while for a library to maintain such a lock file. even that tool like poetry for example are doing just that. (also can be slow and annoying)

[tillahoffmann]

Sorry, I posted on the wrong thread. 🤦‍♂️

[fruch]

We may have to get #209 merged first to get around the google dependency issue.

@tillahoffmann would it be o.k. to take the code changes from #209, and test it in tandam ?

[spicy-sauce]

Hi,
What's the status of this PR? Do you plan to merge it to the master anytime soon?

@fruch I see there are conflicts that should be resolved, is it something that you should handle?

Thanks,
Yossi

[fruch]

Hi,
What's the status of this PR? Do you plan to merge it to the master anytime soon?

@fruch I see there are conflicts that should be resolved, is it something that you should handle?

I guess I could, I've almost forgotten about this one, it's almost a whole year since I've opened this one.

Thanks,
Yossi

[spicy-sauce]

I guess I could, I've almost forgotten about this one, it's almost a whole year since I've opened this one.

Thanks!

@tillahoffmann can we please push it once @fruch resolves the conflicts? This seems like a great addition to testcontainers!

[fruch]

I guess I could, I've almost forgotten about this one, it's almost a whole year since I've opened this one.

Thanks!

@tillahoffmann can we please push it once @fruch resolves the conflicts? This seems like a great addition to testcontainers!

BTW @spicy-sauce can you share a bit what do you need it for ? Cassandra or Scylla ? And which type of testing it's gonna be part of ?

[spicy-sauce]

BTW @spicy-sauce can you share a bit what do you need it for ? Cassandra or Scylla ? And which type of testing it's gonna be part of ?

At this point I need it for Cassandra as I'm going to implement such a block in DataYoga and would love to use it for the testing part.

Read more here:
https://datayoga-io.github.io/datayoga/

[fruch]

BTW @spicy-sauce can you share a bit what do you need it for ? Cassandra or Scylla ? And which type of testing it's gonna be part of ?

At this point I need it for Cassandra as I'm going to implement such a block in DataYoga and would love to use it for the testing part.

Read more here: https://datayoga-io.github.io/datayoga/

looks interesting, ping me when you have it running with Cassandra or Scylla, maybe adding it in our https://university.scylladb.com/

anyhow I'm the maintainer of python driver fork
https://github.com/scylladb/python-driver

if you have question you can reach me at [email protected]

[fruch]

@tillahoffmann #209 didn't seemed to be merged

Anyhow I've rebased again, and rebuild the requirements files again.

Can now it be run in CI and merged ?

[fruch]

@tillahoffmann looks like I've missed updating the requriements files for 3.7, now it should run o.k.

[spicy-sauce]

Guys, can we please push this one? It's really a great addition and we would like to avoid another conflicts.
Thanks a lot!!!

[fruch]

@tillahoffmann @yakimka it's now passes all the tests...

[spicy-sauce]

Is there anything preventing this from being merged?

In the meanwhile, I'd solve the current conflicts.

[SergeyPirogov]

There is a issue with relases because I live in Kyiv and we are fighting with Russsia, so there is no time for the project

[alexanderankin]

@SergeyPirogov that very important and I wish for victory for Ukraine of course.

can we help with releases in the meantime? My username is daveankin on Pypi and if you add me as owner I will work with currently available maintainers to update this library and Pypi, without removing you from the project at all. I hope soon in the future you will have some time for the project and return but until then, please consider adding myself or someone from testcontainers/atomicjar for the meantime.

Thank you!

edit: I have accepted the invite to join on pypi

[guy9]

Also interested in this and happy to help

[alexanderankin]

Latest is on this branch: * Merged testcontainers-core back with testcontainers like the other modules * Fixed tests on Ubuntu * Fixed docs/doc tests after moving modules/on Ubuntu * Fixed npipe bug on windows (still wondering about the right answer there in #415) * Wasn't able to check the whole windows test suite yet, where I left off main...alexanderankin:testcontainers-python:poetry-refactor-changes-to-publish

…

On Tue, Jan 23, 2024, 6:53 AM Guy Shtub ***@***.***> wrote: Also interested in this and happy to help — Reply to this email directly, view it on GitHub <#167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACECGJBGE7SUCW5L5D67OKLYP6QDNAVCNFSM5LAIYIL2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJQGU4DOOJXGAYA> . You are receiving this because you commented.Message ID: ***@***.***>

[alexanderankin]

Maybe I can try to merge or rebase this branch locally or test just these changes locally since technically this is just maintenance we have to do to release after merging

…

On Tue, Jan 23, 2024, 8:17 AM David Ankin ***@***.***> wrote: Latest is on this branch: * Merged testcontainers-core back with testcontainers like the other modules * Fixed tests on Ubuntu * Fixed docs/doc tests after moving modules/on Ubuntu * Fixed npipe bug on windows (still wondering about the right answer there in #415) * Wasn't able to check the whole windows test suite yet, where I left off main...alexanderankin:testcontainers-python:poetry-refactor-changes-to-publish On Tue, Jan 23, 2024, 6:53 AM Guy Shtub ***@***.***> wrote: > Also interested in this and happy to help > > — > Reply to this email directly, view it on GitHub > <#167 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACECGJBGE7SUCW5L5D67OKLYP6QDNAVCNFSM5LAIYIL2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJQGU4DOOJXGAYA> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

[alexanderankin]

hi @fruch,

i have backed up your commit as it exists today in here: dd6520d - please confirm its okay to rebase and push some commits to your branch to adjust to the new structure (#408)?

i see that you have enabled Maintainers are allowed to edit this pull request., so thank you for that, but i am in a habit of asking before i force push, so I appreciate your timely response (or we will figure out something to do for a cassandra implementation based off of this one, i suppose)

thanks,
David

[fruch]

hi @fruch,

i have backed up your commit as it exists today in here: dd6520d - please confirm its okay to rebase and push some commits to your branch to adjust to the new structure (#408)?

i see that you have enabled Maintainers are allowed to edit this pull request., so thank you for that, but i am in a habit of asking before i force push, so I appreciate your timely response (or we will figure out something to do for a cassandra implementation based off of this one, i suppose)

thanks,
David

Go right ahead, ping me you are done, I would give it a look (can do that also after it merged)

[alexanderankin]

i dont think we actually want to merge this when it references an abandoned docker image for cassandra, so ive made some adjustments here - fruch/testcontainers-python@cassandra_scylla_support...testcontainers:testcontainers-python:cassandra_scylla_support_image_improvement

with these additional changes i think this is ready to merge

[alexanderankin]

rebased so this is ready to be merged

linking related - https://github.com/testcontainers/testcontainers-java/pull/8002/files