feat: scylladb database provider implementation #320

DanielHe4rt · 2024-11-04T23:17:53Z

By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license.

Feature

This pull request introduces support for ScyllaDB as a new database option in the project. It includes modifications to various parts of the codebase to integrate ScyllaDB, including database drivers, templates, and Docker configurations.

Description of Changes:

Added ScyllaDB GoCQL Fork to the project
Added ScyllaDB Docker to the Test Container
Added Dockerfile for the ScyllaDB Environment
Added Environment Files to the ScyllaDB Implementation
Added go mod edit helper to get the updated driver

Checklist

I have self-reviewed the changes being requested
I have updated the documentation (check issue ticket 📋Update documentation if the PR introduces a new feature!📋 #218)

cmd/template/dbdriver/files/env/scylla.tmpl

mykaul · 2024-11-05T10:39:19Z

cmd/program/program.go

@@ -103,6 +103,7 @@ var (
 	sqliteDriver   = []string{"github.com/mattn/go-sqlite3"}
 	redisDriver    = []string{"github.com/redis/go-redis/v9"}
 	mongoDriver    = []string{"go.mongodb.org/mongo-driver"}
+	scyllaDriver   = []string{"github.com/gocql/gocql"}


Suggested change

scyllaDriver = []string{"github.com/gocql/gocql"}

scyllaDriver = []string{"github.com/scylladb/gocql"}

Since we have to replace the module, I managed to add a GoCQL driver and then a a replaceable right after it.

Should be clean enough for now.

cmd/program/program.go

cmd/steps/steps.go

cmd/template/dbdriver/files/service/scylla.tmpl

cmd/template/dbdriver/files/tests/scylla.tmpl

cmd/template/docker/files/docker-compose/scylla.tmpl

docs/docs/endpoints-test/scylladb.md

cmd/template/docker/files/docker-compose/scylla.tmpl

cmd/template/dbdriver/files/service/scylla.tmpl

Co-authored-by: Yaniv Kaul <[email protected]>

Co-authored-by: Dmitry Kropachev <[email protected]>

DanielHe4rt · 2024-11-05T15:46:50Z

Ready to review @Melkeydev

Ujstor

We had a new release, rebased, and checked the updated Docker configuration. I also noticed that a test is failing.

make itest
Running integration tests...
2024/11/17 22:17:22 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 27.3.1
  API Version: 1.46
  Operating System: Debian GNU/Linux 12 (bookworm)
  Total Memory: 64206 MB
  Testcontainers for Go Version: v0.34.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 3edd946ba504e85fbb20c0ce8590efde82e51b2ba5456422a77900931f8d8910
  Test ProcessID: 95e3c3a0-f2c2-49c7-b01c-1ef279c3d674
2024/11/17 22:17:22 🐳 Creating container for image testcontainers/ryuk:0.10.2
2024/11/17 22:17:22 ✅ Container created: a1b7597d722b
2024/11/17 22:17:22 🐳 Starting container: a1b7597d722b
2024/11/17 22:17:23 ✅ Container started: a1b7597d722b
2024/11/17 22:17:23 ⏳ Waiting for container id a1b7597d722b image: testcontainers/ryuk:0.10.2. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms skipInternalCheck:false}
2024/11/17 22:17:23 🔔 Container is ready: a1b7597d722b
2024/11/17 22:17:23 🐳 Creating container for image scylladb/scylla:6.2
2024/11/17 22:17:23 ✅ Container created: d8ce0a90298b
2024/11/17 22:17:23 🐳 Starting container: d8ce0a90298b
2024/11/17 22:17:23 ✅ Container started: d8ce0a90298b
2024/11/17 22:17:23 ⏳ Waiting for container id d8ce0a90298b image: scylladb/scylla:6.2. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0xc00036ccc0 0xc00036ccf0 0xc000358900]}
2024/11/17 22:17:27 🔔 Container is ready: d8ce0a90298b
=== RUN   TestNew
2024/11/17 22:17:28 Failed to connect to ScyllaDB cluster: no connections were made when creating the session
FAIL    test1/internal/database 5.312s
FAIL
make: *** [Makefile:40: itest] Error 1

DanielHe4rt · 2024-11-18T17:26:30Z

I just ran it locally and seems to work flawlessly. Also the pipeline didn't accused anything... Maybe could be related to fs.aio-max-nr. I don't remeber if it can turn into a problem with a single node tbh. Here's some instructions (idk if you're running locally but):

cc: @gvieira18

Configure ScyllaDB settings

Before starting the cluster, ensure the fs.aio-max-nr value is sufficient (e.g. 1048576 or 2097152 or more).

You can use the Makefile setup command to configure the fs.aio-max-nr value. It will set the value to 1048576, which is the minimum recommended for clusters.

make setup

If you prefer to configure it manually, run one of the following commands to check the current value:

sysctl --all | grep --word-regexp -- 'aio-max-nr'

sysctl fs.aio-max-nr

cat /proc/sys/fs/aio-max-nr

If the value is lower than required, you can use one of these commands:

# Update config non-persistent
sysctl --write fs.aio-max-nr=1048576

Some logs from my local test:

make itest           
Running integration tests...
2024/11/18 14:20:19 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 27.3.1
  API Version: 1.46
  Operating System: EndeavourOS
  Total Memory: 63499 MB
  Testcontainers for Go Version: v0.34.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 9237e0a86ea19e6aadd6b3c99fce9cbcf8a8e55c6f4d099bf20474c1428b5f70
  Test ProcessID: 8974501f-cbd9-42eb-8333-0f00c486e231
2024/11/18 14:20:19 Failed to get image auth for https://index.docker.io/v1/. Setting empty credentials for the image: testcontainers/ryuk:0.10.2. Error is: credentials not found in native keychain
2024/11/18 14:20:22 🐳 Creating container for image testcontainers/ryuk:0.10.2
2024/11/18 14:20:22 ✅ Container created: a55e8340cff5
2024/11/18 14:20:22 🐳 Starting container: a55e8340cff5
2024/11/18 14:20:22 ✅ Container started: a55e8340cff5
2024/11/18 14:20:22 ⏳ Waiting for container id a55e8340cff5 image: testcontainers/ryuk:0.10.2. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms skipInternalCheck:false}
2024/11/18 14:20:22 🔔 Container is ready: a55e8340cff5
2024/11/18 14:20:22 🐳 Creating container for image scylladb/scylla:6.2
2024/11/18 14:20:22 ✅ Container created: 1e262c1e3948
2024/11/18 14:20:22 🐳 Starting container: 1e262c1e3948
2024/11/18 14:20:22 ✅ Container started: 1e262c1e3948
2024/11/18 14:20:22 ⏳ Waiting for container id 1e262c1e3948 image: scylladb/scylla:6.2. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0xc0002840c0 0xc000284150 0xc000288b40]}
2024/11/18 14:20:25 🔔 Container is ready: 1e262c1e3948
=== RUN   TestNew
--- PASS: TestNew (0.00s)
=== RUN   TestHealth
--- PASS: TestHealth (0.01s)
=== RUN   TestClose
--- PASS: TestClose (0.00s)
PASS
2024/11/18 14:20:25 🐳 Stopping container: 1e262c1e3948
2024/11/18 14:20:30 ✅ Container stopped: 1e262c1e3948
2024/11/18 14:20:30 🐳 Terminating container: 1e262c1e3948
2024/11/18 14:20:30 🚫 Container terminated: 1e262c1e3948
ok      fodase/internal/database        11.226s

gvieira18 · 2024-11-18T18:43:01Z

@DanielHe4rt I get the same behavior here, my fs.aio-max-nr is 65536.

# go clean -testcache && make itest

Running integration tests...
2024/11/18 15:26:03 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 27.3.1
  API Version: 1.46
  Operating System: Debian GNU/Linux 12 (bookworm)
  Total Memory: 15996 MB
  Testcontainers for Go Version: v0.34.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 7926d623964bdc80c86869a0e083128a04c8b911a4aae9d79b89bead644b26e3
  Test ProcessID: 9e9eee67-f1b5-4d4f-89fb-43cc1069178d
2024/11/18 15:26:03 🐳 Creating container for image testcontainers/ryuk:0.10.2
2024/11/18 15:26:03 ✅ Container created: 20e7ad71202d
2024/11/18 15:26:03 🐳 Starting container: 20e7ad71202d
2024/11/18 15:26:03 ✅ Container started: 20e7ad71202d
2024/11/18 15:26:03 ⏳ Waiting for container id 20e7ad71202d image: testcontainers/ryuk:0.10.2. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms skipInternalCheck:false}
2024/11/18 15:26:04 🔔 Container is ready: 20e7ad71202d
2024/11/18 15:26:04 🐳 Creating container for image scylladb/scylla:6.2
2024/11/18 15:26:04 ✅ Container created: 503de84e8751
2024/11/18 15:26:04 🐳 Starting container: 503de84e8751
2024/11/18 15:26:04 ✅ Container started: 503de84e8751
2024/11/18 15:26:04 ⏳ Waiting for container id 503de84e8751 image: scylladb/scylla:6.2. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0xc000275b60 0xc000275b90 0xc000269140]}
2024/11/18 15:26:43 🔔 Container is ready: 503de84e8751
=== RUN   TestNew
--- PASS: TestNew (0.00s)
=== RUN   TestHealth
--- PASS: TestHealth (0.01s)
=== RUN   TestClose
--- PASS: TestClose (0.00s)
PASS
2024/11/18 15:26:43 🐳 Stopping container: 503de84e8751
2024/11/18 15:26:49 ✅ Container stopped: 503de84e8751
2024/11/18 15:26:49 🐳 Terminating container: 503de84e8751
2024/11/18 15:26:49 🚫 Container terminated: 503de84e8751
ok      simple-test/internal/database   46.535s

Ujstor · 2024-11-18T19:18:12Z

@DanielHe4rt
this is new db template, we added react in latest release:

services:
{{- if .AdvancedOptions.docker }}
  app:
    build:
      context: .
      dockerfile: Dockerfile
      target: prod
    restart: unless-stopped
    ports:
      - ${PORT}:${PORT}
    environment:
      APP_ENV: ${APP_ENV}
      PORT: ${PORT}
      BLUEPRINT_DB_HOST: ${BLUEPRINT_DB_HOST}
      BLUEPRINT_DB_PORT: ${BLUEPRINT_DB_PORT}
      BLUEPRINT_DB_DATABASE: ${BLUEPRINT_DB_DATABASE}
      BLUEPRINT_DB_USERNAME: ${BLUEPRINT_DB_USERNAME}
      BLUEPRINT_DB_PASSWORD: ${BLUEPRINT_DB_PASSWORD}
      BLUEPRINT_DB_SCHEMA: ${BLUEPRINT_DB_SCHEMA}
    depends_on:
      psql_bp:
        condition: service_healthy
    networks:
      - blueprint
{{- end }}
{{- if and .AdvancedOptions.react .AdvancedOptions.docker }}
  frontend:
    build:
      context: .
      dockerfile: Dockerfile
      target: frontend
    restart: unless-stopped
    depends_on:
      - app
    ports:
      - 5173:5173
    networks:
      - blueprint
{{- end }}
  psql_bp:
    image: postgres:latest
    restart: unless-stopped
    environment:
      POSTGRES_DB: ${BLUEPRINT_DB_DATABASE}
      POSTGRES_USER: ${BLUEPRINT_DB_USERNAME}
      POSTGRES_PASSWORD: ${BLUEPRINT_DB_PASSWORD}
    ports:
      - "${BLUEPRINT_DB_PORT}:5432"
    volumes:
      - psql_volume_bp:/var/lib/postgresql/data
    {{- if .AdvancedOptions.docker }}
    healthcheck:
      test: ["CMD-SHELL", "sh -c 'pg_isready -U ${BLUEPRINT_DB_USERNAME} -d ${BLUEPRINT_DB_DATABASE}'"]
      interval: 5s
      timeout: 5s
      retries: 3
      start_period: 15s
    networks:
      - blueprint
    {{- end }}

volumes:
  psql_volume_bp:
{{- if .AdvancedOptions.docker }}
networks:
  blueprint:
{{- end }}

Regarding the testcontainerd case, on my machine, the tests are failing. I'm not sure why, as I configured it as you suggested:

sudo sysctl --all | grep --word-regexp -- 'aio-max-nr'
fs.aio-max-nr = 1048576

I thought maybe my local environment was broken in some way, so I tested other databases. In this case, the test passed. If I am having this issue, other users might experience the same. We need to investigate this further and update the documentation accordingly.

Running the database in a container and testing the health endpoint works as expected:

curl localhost:8080/health
{"message":"It's healthy","scylla_cluster_nodes_down":"0","scylla_cluster_nodes_up":"1","scylla_cluster_size":"1","scylla_current_datacenter":"datacenter1","scylla_current_time":"2024-11-18 18:58:32.016 +0000 UTC","scylla_health_check_duration":"8.134206ms","scylla_keyspaces":"6","status":"up"}

I am sending my logs:

 make itest
Running integration tests...
2024/11/18 20:12:38 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 27.3.1
  API Version: 1.46
  Operating System: Debian GNU/Linux 12 (bookworm)
  Total Memory: 64206 MB
  Testcontainers for Go Version: v0.34.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 698c302e2118be82e763b4dd71b363ce392a72516252e6d06b4574d0842d8fb2
  Test ProcessID: 01047aa4-6e3f-4ee9-843f-11867b3548e3
2024/11/18 20:12:38 🐳 Creating container for image testcontainers/ryuk:0.10.2
2024/11/18 20:12:38 ✅ Container created: 5cb859125e16
2024/11/18 20:12:38 🐳 Starting container: 5cb859125e16
2024/11/18 20:12:38 ✅ Container started: 5cb859125e16
2024/11/18 20:12:38 ⏳ Waiting for container id 5cb859125e16 image: testcontainers/ryuk:0.10.2. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms skipInternalCheck:false}
2024/11/18 20:12:38 🔔 Container is ready: 5cb859125e16
2024/11/18 20:12:38 🐳 Creating container for image scylladb/scylla:6.2
2024/11/18 20:12:38 ✅ Container created: 79a6c52d563a
2024/11/18 20:12:38 🐳 Starting container: 79a6c52d563a
2024/11/18 20:12:39 ✅ Container started: 79a6c52d563a
2024/11/18 20:12:39 ⏳ Waiting for container id 79a6c52d563a image: scylladb/scylla:6.2. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0xc0002f5cb0 0xc0002f5ce0 0xc0002e0ac0]}
2024/11/18 20:12:43 🔔 Container is ready: 79a6c52d563a
=== RUN   TestNew
--- PASS: TestNew (0.00s)
=== RUN   TestHealth
2024/11/18 20:12:43 Failed to connect to ScyllaDB cluster: no connections were made when creating the session
FAIL    test3/internal/database 4.995s
FAIL
make: *** [Makefile:40: itest] Error 1

Ujstor · 2024-11-18T19:34:08Z

Please add scylladb to workflows:

name: Linting Generated Blueprints Core

jobs:
  framework_matrix:
    strategy:
      matrix:
        framework: [chi, gin, fiber, gorilla/mux, httprouter, standard-library, echo]
        driver: [mysql, postgres, sqlite, mongo, redis, none]
        git: [commit, stage, skip]

and

name: Integrations Test for the Generated Blueprints

jobs:
  itests_matrix:
    strategy:
      matrix:
        driver:
          [mysql, postgres, mongo, redis]

Signed-off-by: Gabriel do Carmo Vieira <[email protected]>

DanielHe4rt · 2024-11-19T18:14:48Z

Seems that @gvieira18 made the changes. @Ujstor can you review it?

Ujstor

The PR is really good. There are two more things to address:

Update the documentation with information on how a local machine/server needs to be configured to run ScyllaDB in case of issues similar to mine.

Second one, I need to handle this myself. I'll try testing the container-test part that is failing for me on another machine. In the pipeline, the tests are passing, so I think the issue is on my side.

mdelapenya · 2024-11-20T11:10:06Z

cmd/template/dbdriver/files/tests/scylla.tmpl

+				return strings.Contains(string(data), "COMPLETED")
+			}),
+		),
+		Name: name,


I'd not pin the container name, as it would mean that it won't be possible to run multiple tests in parallel. Testcontainers (Docker) creates random names for each container without name.

mdelapenya · 2024-11-20T11:11:56Z

cmd/template/dbdriver/files/tests/scylla.tmpl

+		return nil, err
+	}
+
+	hosts = fmt.Sprintf("localhost:%v", mappedPort.Port())


bug: we'd need to initialise the rest of the fields (username, password, consistencyLevel) in order to bootstrap the cluster correctly. Please check how to define the credentials in the docker image, so that the container request sets them.

@DanielHe4rt would you be interested in contributing a ScyllaDB module for testcontainers-go? This way here we'd use the module instead of the GenericContainer approach. In the end is the same, but easier for the end user to interact with a built-in module.

I can help you out with the creation of the module. Thoughts?

I'm actually with a task to work on it under the Testcontainers with Go :p

Quite amazing to have a core-contributor looking for new contributions around! We can chat and make this a thing, even start some Livestream while we work on it.

I just sent you a LinkedIn connection, we can chat there if you want!

bug: we'd need to initialise the rest of the fields (username, password, consistencyLevel) in order to bootstrap the cluster correctly. Please check how to define the credentials in the docker image, so that the container request sets them.

@DanielHe4rt Was there a particular reason for not including this in the fix?

Scylla doesn't need these extra fields. Initially it just need the Host and Port since we're using only one instance instead a 3 Node Cluster.

There's no bug there since by default all these options has a fallback. It's fully up to the user, the boilerplate will only fulfill the basics.

Sorry, I'm not a Scylla user, so I wrongly understood the credentials needed to be set. LGTM then

Ujstor

@mdelapenya, thanks for the review. @DanielHe4rt, make the changes that were pointed out in the review. Tests need to work OOB. After that, we are good to merge.

Ujstor

LGTM

DanielHe4rt added 4 commits November 4, 2024 19:14

feat: add scylladb driver

9fd7322

feat: scylladb db boilerplate with right driver replacement

0d8f3e9

feat: removing prints from scylladb implementation

9250c20

docs: add scylladb driver implementation

307bffe

DanielHe4rt marked this pull request as ready for review November 5, 2024 00:01

gvieira18 added 2 commits November 4, 2024 21:09

refactor: rename ScyllaDB nomenclature

0169891

feat: update ScyllaDB templates for improved docker support

f34ee04

DanielHe4rt mentioned this pull request Nov 5, 2024

[Feature Request] Implementing ScyllaDB as a Provider #319

Closed

1 task

Daniel-Boll reviewed Nov 5, 2024

View reviewed changes

cmd/template/dbdriver/files/env/scylla.tmpl Outdated Show resolved Hide resolved

docs: adding scylladb provider to readme

f65fa44

mykaul reviewed Nov 5, 2024

View reviewed changes

cmd/program/program.go Outdated Show resolved Hide resolved

mykaul reviewed Nov 5, 2024

View reviewed changes

cmd/steps/steps.go Outdated Show resolved Hide resolved

mykaul reviewed Nov 5, 2024

View reviewed changes

cmd/template/dbdriver/files/service/scylla.tmpl Show resolved Hide resolved

dkropachev reviewed Nov 5, 2024

View reviewed changes

DanielHe4rt and others added 6 commits November 5, 2024 11:03

typo: driver related

3dd4028

Co-authored-by: Yaniv Kaul <[email protected]>

fix: min shards for default dockerfile

5e9a992

Co-authored-by: Dmitry Kropachev <[email protected]>

fix: scylladb service implementation

2de0a33

chore: scylladb go module replacement

f14dcdd

fix: scylladb consistency default env

eeb1daf

fix: escape quotes for healthcheck command

be6f3ce

docs(scylladb): updating the Health function

227deaa

Ujstor requested changes Nov 17, 2024

View reviewed changes

Merge branch 'Melkeydev:main' into main

7eb2127

DanielHe4rt requested a review from Ujstor November 18, 2024 17:29

gvieira18 added 2 commits November 18, 2024 18:43

chore: add scylla to CI workflow

427e05a

Signed-off-by: Gabriel do Carmo Vieira <[email protected]>

add react tmpl service

ed9c9a9

Signed-off-by: Gabriel do Carmo Vieira <[email protected]>

Ujstor requested changes Nov 19, 2024

View reviewed changes

mdelapenya reviewed Nov 20, 2024

View reviewed changes

Ujstor requested changes Nov 20, 2024

View reviewed changes

feat: scylladb testcontainers fix with docs

1a6c598

DanielHe4rt requested a review from Ujstor November 20, 2024 16:21

Ujstor approved these changes Nov 20, 2024

View reviewed changes

Ujstor merged commit ff91dce into Melkeydev:main Nov 20, 2024
195 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: scylladb database provider implementation #320

feat: scylladb database provider implementation #320

DanielHe4rt commented Nov 4, 2024 •

edited

Loading

mykaul Nov 5, 2024

DanielHe4rt Nov 5, 2024

DanielHe4rt commented Nov 5, 2024

Ujstor left a comment

DanielHe4rt commented Nov 18, 2024

gvieira18 commented Nov 18, 2024

Ujstor commented Nov 18, 2024

Ujstor commented Nov 18, 2024

DanielHe4rt commented Nov 19, 2024

Ujstor left a comment •

edited

Loading

mdelapenya Nov 20, 2024

mdelapenya Nov 20, 2024

mdelapenya Nov 20, 2024

DanielHe4rt Nov 20, 2024

Ujstor Nov 20, 2024

DanielHe4rt Nov 20, 2024

mdelapenya Nov 20, 2024

Ujstor left a comment

Ujstor left a comment

	scyllaDriver = []string{"github.com/gocql/gocql"}
	scyllaDriver = []string{"github.com/scylladb/gocql"}

feat: scylladb database provider implementation #320

feat: scylladb database provider implementation #320

Conversation

DanielHe4rt commented Nov 4, 2024 • edited Loading

Feature

Description of Changes:

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanielHe4rt commented Nov 5, 2024

Ujstor left a comment

Choose a reason for hiding this comment

DanielHe4rt commented Nov 18, 2024

Configure ScyllaDB settings

gvieira18 commented Nov 18, 2024

Ujstor commented Nov 18, 2024

Ujstor commented Nov 18, 2024

DanielHe4rt commented Nov 19, 2024

Ujstor left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ujstor left a comment

Choose a reason for hiding this comment

Ujstor left a comment

Choose a reason for hiding this comment

DanielHe4rt commented Nov 4, 2024 •

edited

Loading

Ujstor left a comment •

edited

Loading