Create Cassandra db schema on session initialization #5922

akstron · 2024-09-02T18:04:32Z

Create Schema (if not present) on Session Initialization

Once a session is established with cassandra db, the added code parses the template file containing queries for creating schema and create queries out of it. Post which it executes those queries to create the required types and tables.

Which problem is this PR solving?

Resolves #5797

Description of the changes

The PR includes the following changes:
1. Embedding template files into binary
1. Creation of database schema in initialization steps once session to database is established.

How was this change tested?

Schema rendering is being tested with unit test.

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: yarn lint and yarn test

plugin/storage/cassandra/factory.go

yurishkuro · 2024-09-02T21:48:46Z

plugin/storage/cassandra/factory.go

+	datacenter             string
+	trace_ttl              int
+	dependencies_ttl       int
+	keyspace               string


this should already be defined in the main config

Are you talking about this config: jaeger/cmd/jaeger/config-cassandra.yaml ? How would I know what all config key-value pairs are already defined?

Hi @yurishkuro , I needed some help here. Is this: jaeger/cmd/jaeger/internal/all-in-one.yaml the example for the config file which would be used in v2? If yes, under what section should I expect the cassandra specific configs to be put? I couldn't find any example for this in the current code base.

Just to get the complete picture, would the schema configs I am adding look something like this:

schema: datacenter: <datacenterName> keyspace: <keyspace> .......

cmd/jaeger/config-cassandra.yaml

Let me know if the current place of defining it works?

plugin/storage/cassandra/factory.go

yurishkuro · 2024-09-02T21:54:36Z

plugin/storage/cassandra/factory.go

+	return result
+}
+
+func constructQueriesFromTemplateFiles(session cassandra.Session, params *StorageConfigParams) ([]cassandra.Query, error) {


Is cassandra.Session not able to execute multiple queries at once?

Are you talking about running individual queries in parallel or a batch query option? I didn't try executing parallel queries.

plugin/storage/cassandra/factory.go

yurishkuro · 2024-10-26T17:35:17Z

cmd/jaeger/config-cassandra.yaml

@@ -24,6 +24,12 @@ extensions:
        cassandra:
          schema: 
            keyspace: "jaeger_v1_dc1"
+            datacenter: "test"


We need to add these values somewhere in create default config function.

Added to func DefaultConfiguration. Thanks

In that case do you need the values to still be defined here?

I see. I was thinking this acts as a quick reference to how the config file should look. I will remove this since this may not be required.

yurishkuro · 2024-10-26T17:37:14Z

pkg/cassandra/config/config.go

@@ -51,13 +51,26 @@ type Connection struct {
 	ProtoVersion int `mapstructure:"proto_version"`
 }

+type SchemaConfig struct {


Please add comments to each field with sample values/units.

pkg/cassandra/config/config.go

yurishkuro · 2024-10-26T18:57:07Z

plugin/storage/cassandra/schema/schema.go

+	}
+
+	for _, query := range casQueries {
+		err := query.Exec()


If a corresponding table already exists, will this be a no-op or will it update the settings in the table? We need to allow users to be able to customize the settings.

All the table creation queries use: CREATE TABLE IF NOT EXISTS, so this will be a no-op.

…ution for initialize database Signed-off-by: Alok Kumar Singh <[email protected]>

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro · 2024-10-28T16:07:39Z

pkg/cassandra/config/config.go

+	//Datacenter is the name for network topology
+	Datacenter string `mapstructure:"datacenter" valid:"optional"`
+	// TraceTTL is Time To Live (TTL) for the trace data in seconds
+	TraceTTL int `mapstructure:"trace_ttl" valid:"optional"`


can the type here be time.Duration so that the user could specify 72h?

I am assuming that there should be a validation to make sure that user can't specify something like "ms"?

that's the point of using strong type - the business logic does not need to worry about validations, they should happen separately during parsing. I believe if you simply change int to time.Duration and use 10ms as a value in YAML, the parser will do the right thing.

yurishkuro · 2024-10-28T16:08:33Z

pkg/cassandra/config/config.go

+	// CasVersion is version of cassandra used
+	CasVersion int `mapstructure:"cas_version" valid:"optional"`
+	// CompactionWindow of format "^[0-9]+[mhd]$" tells the compaction window of the db
+	CompactionWindow string `mapstructure:"compaction_window" valid:"optional"`


so this is defining a time interval? Can we then also use time.Duration type?

yurishkuro · 2024-10-28T16:09:27Z

pkg/cassandra/config/config.go

+	// CompactionWindowSize is the numberical part of CompactionWindow. Extracted from CompactionWindow
+	CompactionWindowSize int `mapstructure:"compaction_window_size" valid:"optional"`
+	// CompactionWindowUnit is the time unit of the CompactionWindow. Extracted from CompactionWindow
+	CompactionWindowUnit string `mapstructure:"compaction_window_unit" valid:"optional"`


if we use time.Duration type we will not need the unit separately

Also, Size and Unit seems to be derived fields, if so they should not be declared in the Config struct. You can create a different struct for use with Go templates, e.g.

type TemplateParams struct { config.Schema // embed config // add other derived fields }

Added TemplateParams with embedded config.Schema and other derived fields.

yurishkuro · 2024-10-28T16:11:03Z

plugin/storage/cassandra/schema/v004-go-tmpl-test.cql.tmpl

@@ -0,0 +1,84 @@
+-- There are total 9 queries here


what is "test"?

[Improved test and changed name in recent commit] The test is to check if the extracted query strings are same as the expected query strings.

yurishkuro · 2024-10-28T16:12:52Z

plugin/storage/cassandra/schema/schema.go

+		traceTTLMinutes := cfg.TraceTTL / 60
+
+		cfg.CompactionWindowSize = (traceTTLMinutes + 30 - 1) / 30
+		cfg.CompactionWindowUnit = "m"


below you use "MINUTES", is "m" also valid?

Yes, m, h and d are valid.

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro · 2024-10-31T02:09:50Z

plugin/storage/cassandra/schema/v004-go-tmpl-test.cql.tmpl

@@ -0,0 +1,43 @@
+-- There are total 4 queries here


what's the point of test schema?

Instead of testing on all the queries and manually creating expected result for all of them, I reduced the number of queries to ease the creation of expected result.

that doesn't really make sense to me. We can either test that the template is correct or that the code using the template is correct. You're not doing the former by having another template, and the latter you can do with the primary template. We don't need to validate that the output of running template is "as expected", that's like testing the Go template package.

We don't need to validate that the output of running template is "as expected", that's like testing the Go template package.
We are also removing comments and constructing individual query strings out of it by iterating over lines in the template output. The test checks this "individual query string" construction, which not just involves using the template package.

and the latter you can do with the primary template.
Sure, I think I can write an integration test for this.

yurishkuro reviewed Sep 2, 2024

View reviewed changes

yurishkuro mentioned this pull request Sep 7, 2024

[jaeger-v2] Align Cassandra Storage Config With OTEL #5928

Closed

6 tasks

mahadzaryab1 reviewed Sep 7, 2024

View reviewed changes

plugin/storage/cassandra/factory.go Outdated Show resolved Hide resolved

akstron force-pushed the create-database-scheme-cassandra branch 2 times, most recently from 69275fb to bcad4c0 Compare October 25, 2024 13:34

akstron marked this pull request as ready for review October 25, 2024 13:52

akstron requested a review from a team as a code owner October 25, 2024 13:52

akstron requested a review from albertteoh October 25, 2024 13:52

dosubot bot added area/storage storage/cassandra labels Oct 25, 2024

akstron requested a review from yurishkuro October 26, 2024 16:18

yurishkuro reviewed Oct 26, 2024

View reviewed changes

akstron force-pushed the create-database-scheme-cassandra branch from f9dd90e to 90368b1 Compare October 28, 2024 10:46

Embeded template file in binary and added query construction and exec…

afc786d

…ution for initialize database Signed-off-by: Alok Kumar Singh <[email protected]>

akstron force-pushed the create-database-scheme-cassandra branch from 90368b1 to afc786d Compare October 28, 2024 10:50

akstron added 3 commits October 28, 2024 16:45

Removed unnecessary SchemaConfig struct

3a82683

Signed-off-by: Alok Kumar Singh <[email protected]>

Added new schema configs in default config generator

b3e0370

Signed-off-by: Alok Kumar Singh <[email protected]>

Revert Keyspace removal

c35ef2d

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro reviewed Oct 28, 2024

View reviewed changes

akstron added 4 commits October 29, 2024 12:31

Bug fix while creating queries

18dcf1e

Signed-off-by: Alok Kumar Singh <[email protected]>

Improving test

b230412

Signed-off-by: Alok Kumar Singh <[email protected]>

Created new struct for derived params

9642e38

Signed-off-by: Alok Kumar Singh <[email protected]>

Remove fields from yaml file

32485af

Signed-off-by: Alok Kumar Singh <[email protected]>

yurishkuro reviewed Oct 31, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Cassandra db schema on session initialization #5922

Create Cassandra db schema on session initialization #5922

akstron commented Sep 2, 2024 •

edited

Loading

yurishkuro Sep 2, 2024

akstron Sep 7, 2024

akstron Sep 11, 2024 •

edited

Loading

yurishkuro Sep 11, 2024

akstron Oct 25, 2024

yurishkuro Sep 2, 2024

akstron Oct 25, 2024

yurishkuro Oct 26, 2024

akstron Oct 28, 2024

yurishkuro Oct 28, 2024

akstron Oct 28, 2024

yurishkuro Oct 26, 2024

akstron Oct 28, 2024

yurishkuro Oct 26, 2024

akstron Oct 28, 2024

yurishkuro Oct 28, 2024

akstron Oct 29, 2024 •

edited

Loading

yurishkuro Oct 31, 2024

yurishkuro Oct 28, 2024

yurishkuro Oct 28, 2024

yurishkuro Oct 28, 2024

akstron Oct 29, 2024 •

edited

Loading

yurishkuro Oct 28, 2024

akstron Oct 29, 2024

yurishkuro Oct 28, 2024

akstron Oct 29, 2024 •

edited

Loading

yurishkuro Oct 31, 2024

akstron Oct 31, 2024 •

edited

Loading

yurishkuro Oct 31, 2024

akstron Nov 1, 2024

Create Cassandra db schema on session initialization #5922

Are you sure you want to change the base?

Create Cassandra db schema on session initialization #5922

Conversation

akstron commented Sep 2, 2024 • edited Loading

Which problem is this PR solving?

Description of the changes

How was this change tested?

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Sep 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akstron commented Sep 2, 2024 •

edited

Loading

akstron Sep 11, 2024 •

edited

Loading

akstron Oct 29, 2024 •

edited

Loading

akstron Oct 29, 2024 •

edited

Loading

akstron Oct 29, 2024 •

edited

Loading

akstron Oct 31, 2024 •

edited

Loading