test: Add integration tests for `Indexer` #627

morgsmccauley · 2024-03-29T00:52:40Z

This PR adds a very basic integration test for Indexer. It uses testcontainers to stand up both postgres and hasura so that Indexer can talk to real components rather than mocks. The test uses Indexer directly, which means S3/Redis are still somewhat mocked/ignored. We can add those in later if need be.

This is essentially just the scaffolding for integration testing which can be expanded over time. The suite includes only 1 very basic test, which if successful should provide a fair amount of confidence that things are working as expected. The flow includes: provisioning, writing data to Postgres, and then asserting its existence via GraphQL. All errors bubble up from Indexer so this test should catch most problems.

This PR points to #625, as so I could test the pg_cron flow via this integration test :)

morgsmccauley · 2024-03-29T00:59:23Z

hasura/Dockerfile

@@ -0,0 +1,9 @@
+FROM hasura/graphql-engine:latest.cli-migrations-v3


Automatically applies locally stored metadata/migrations (i.e. logs/state tables, append role etc.) prior to standing up graphql server - probably only useful in tests right now

morgsmccauley · 2024-03-29T00:59:37Z

postgres.Dockerfile

@@ -0,0 +1,11 @@
+FROM postgres:14


postgres image which enables pg_cron

morgsmccauley · 2024-03-29T01:00:49Z

runner/tsconfig.build.json

@@ -0,0 +1,5 @@
+{


Separate tsconfig to use when building, the default tsconfig.json should only be used locally now since it includes tests files etc.

morgsmccauley · 2024-03-29T01:01:58Z

runner/src/provisioner/provisioner.ts

@@ -96,9 +108,9 @@ export default class Provisioner {
        const userCronPgClient = new this.PgClient({


the connection parameters returned from hasura will include the host/port used within the docker network, therefore we need an override

morgsmccauley · 2024-03-29T01:06:10Z

runner/src/pg-client.ts

      database: connectionParams.database,
      ...poolConfig,
    });
+
+    this.pgPool.on('error', onError);


pgPool will throw an error if the connection is closed from the server side - which will be the case during integration tests as the container will shut down first. Catching it here to prevent runtime errors.

morgsmccauley · 2024-03-29T01:06:32Z

runner/src/indexer/indexer.ts

@@ -49,6 +59,7 @@ export default class Indexer {
    deps?: Partial<Dependencies>,
    databaseConnectionParameters = undefined,
    dmlHandler = undefined,
+    private readonly config: Config = defaultConfig,


Need to parameterise these so they are configurable from test

morgsmccauley · 2024-03-29T01:07:14Z

runner/src/hasura-client/hasura-client.ts

 export default class HasuraClient {
  static DEFAULT_DATABASE = 'default';
  static DEFAULT_SCHEMA = 'public';

  private readonly deps: Dependencies;

-  constructor (deps?: Partial<Dependencies>) {
+  constructor (deps?: Partial<Dependencies>, private readonly config: Config = defaultConfig) {


parameterising so they are configurable from tests

gabehamilton

Cool!

darunrs

Neat work! This will be fun to build on top of to get a good quite of integ tests going.

Just some small comments.

darunrs · 2024-03-29T20:12:05Z

postgres.Dockerfile

+
+RUN echo "shared_preload_libraries = 'pg_cron'" >> /usr/share/postgresql/postgresql.conf.sample
+
+RUN echo "CREATE EXTENSION pg_cron;" > /docker-entrypoint-initdb.d/init-pg-cron.sql


This probably works. Though, I actually have more SQL that's necessary to set up pgBouncer. I created an sql file in queryApi which is copied into the docker entry point inside the docker compose. Since you're adding a dockerfile for postgres now, maybe we can merge the two approaches together?

Maybe we can create a top level postgres/ directory which houses both this Dockerfile and the init script. I'll look at that in a follow up :)

darunrs · 2024-03-29T20:13:45Z

runner/src/hasura-client/hasura-client.ts

@@ -10,17 +10,43 @@ interface SqlOptions {
  source?: string
 }

+interface DatabaseConnectionParameters {


I actually export this interface in Indexer. You can import it here.

Again related to #625, I'll fix in a follow up PR

darunrs · 2024-03-29T20:17:09Z

runner/src/provisioner/provisioner.ts

+
+        await userCronPgClient.query(
+          this.pgFormat(
+            "SELECT cron.schedule_in_database('%1$I_logs_create_partition', '0 1 * * *', $$SELECT fn_create_partition('%1$I.__logs', CURRENT_DATE, '1 day', '2 day')$$, %2$L);",


Perhaps we can convert these into constants and label them something like CreatePartitionCronSql or something.

This changes are from #625 which this PR originally pointed to but has since been merged 😄

darunrs · 2024-03-29T20:19:00Z

runner/tests/integration.test.ts

+    await indexer.runFunctions(
+      Block.fromStreamerMessage(block1 as any as StreamerMessage),
+      {
+        'morgs.near/test': {


This works fine for now. How are you thinking of organizing the integ tests once we have more test cases and test code? Maybe a new folder called integ/ and label the code according to the test? Same for sample block data. Maybe we can also leverage the block-server to download actual blocks too?

I hadn't put a lot of though in to that because I'm not entirely sure how these will evolve over time - we can tackle that when needed

morgsmccauley changed the base branch from main to feat/schedule-log-partition-jobs March 29, 2024 00:53

morgsmccauley changed the title ~~test/runner integration~~ test: Add integration tests for Indexer Mar 29, 2024

morgsmccauley linked an issue Mar 29, 2024 that may be closed by this pull request

Implement Runner integration testing #609

Closed

morgsmccauley commented Mar 29, 2024

View reviewed changes

postgres.Dockerfile

@@ -0,0 +1,11 @@

FROM postgres:14

Copy link

Collaborator Author

morgsmccauley Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

postgres image which enables pg_cron

morgsmccauley commented Mar 29, 2024

View reviewed changes

morgsmccauley marked this pull request as ready for review March 29, 2024 01:11

morgsmccauley requested a review from a team as a code owner March 29, 2024 01:11

gabehamilton approved these changes Mar 29, 2024

View reviewed changes

Base automatically changed from feat/schedule-log-partition-jobs to main March 29, 2024 20:04

refactor: Use shared pg format instance

7a5049c

darunrs approved these changes Mar 29, 2024

View reviewed changes

morgsmccauley added 14 commits March 30, 2024 09:21

feat: Schedule cron jobs under user role

d643db8

test: Exclude tests in dist/

1bb09e3

test: Suppress console output when running tests

879f447

refactor: Parameterise hasura env config

c041293

test: Create postgres/hasura testcontainers

051b759

build: Configure lint/tsc/jest to respect tests/

da647d9

test: Build hasura container from local Dockerfile

be0752b

fix: Use passed in port for pg client

46341d4

feat: Add default pg pool error handler

5a876f0

feat: Parameterise Indexer env config

db92cc2

test: Add integration tests for Indexer

a233293

refactor: Extract testcontainer logConsumer

ad4afa3

test: Fix unit tests after parameterisation of env

2e75e03

test: Ensure pg client points to correct location during integration

669fce5

test: Add pg_cron to integration tests

7a4862a

morgsmccauley force-pushed the test/runner-integration branch from 7332228 to 7a4862a Compare March 29, 2024 20:21

morgsmccauley merged commit b7d43e1 into main Mar 29, 2024
3 checks passed

morgsmccauley deleted the test/runner-integration branch March 29, 2024 20:36

This was referenced Apr 22, 2024

Prod Release 23/04/24 #686

Merged

test stable branch git fix up #687

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Add integration tests for `Indexer` #627

test: Add integration tests for `Indexer` #627

morgsmccauley commented Mar 29, 2024 •

edited

Loading

morgsmccauley Mar 29, 2024

morgsmccauley Mar 29, 2024

morgsmccauley Mar 29, 2024

morgsmccauley Mar 29, 2024

morgsmccauley Mar 29, 2024

morgsmccauley Mar 29, 2024

morgsmccauley Mar 29, 2024

gabehamilton left a comment

darunrs left a comment

darunrs Mar 29, 2024

morgsmccauley Mar 29, 2024

darunrs Mar 29, 2024

morgsmccauley Mar 29, 2024

darunrs Mar 29, 2024

morgsmccauley Mar 29, 2024

darunrs Mar 29, 2024

morgsmccauley Mar 29, 2024

		@@ -0,0 +1,9 @@
		FROM hasura/graphql-engine:latest.cli-migrations-v3

		@@ -96,9 +108,9 @@ export default class Provisioner {
		const userCronPgClient = new this.PgClient({


		RUN echo "shared_preload_libraries = 'pg_cron'" >> /usr/share/postgresql/postgresql.conf.sample

		RUN echo "CREATE EXTENSION pg_cron;" > /docker-entrypoint-initdb.d/init-pg-cron.sql

test: Add integration tests for Indexer #627

test: Add integration tests for Indexer #627

Conversation

morgsmccauley commented Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabehamilton left a comment

Choose a reason for hiding this comment

darunrs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

test: Add integration tests for `Indexer` #627

test: Add integration tests for `Indexer` #627

morgsmccauley commented Mar 29, 2024 •

edited

Loading