Add /files/contracts/partial/{chain} endpoint and explicit ordering, resolve SQL NULL comparison bug

[kuzdogan]

This PR was intended to add the /files/contracts/partial/{chain} endpoint that is missing to list all contracts under a certain chain.

However during the process, specifically when adding proper tests, I've noticed the partial and any endpoints not working as expected, and some other potential issues explained below for further investigation.

TLDR;

• Adding the /files/contracts/partial/{chain} endpoint as specified in Implementing /files/contracts/partial endpoint #1415 with an option to sort asc and desc
• Adding tests
• SQL NULL comparison bug
• Potential issues:
  • Deduplication not working in compiled_contracts
  • matchType not used in database-utils.ts

SQL NULL comparison bug

This was due to the queries being used in countSourcifyMatchAddresses and getSourcifyMatchAddressesByChainAndMatch.

Both SQL queries count the number of perfect matches with creation_match = 'perfect' OR runtime_match = 'perfect' and the partial matches with the logical inverse creation_match != 'perfect' AND runtime_match != 'perfect'. However this does not work as expected in SQL because the creation_match field in our case can be NULL and any comparison with NULL resolves to NULL, and not to true as expected.

This can also be seen by comparing the contract numbers in stats.json vs the API output:

/stats.json for Mainnet:

  "1": {
    ...
    "full_match": 131482,
    "partial_match": 529877,
  },
  // total: 661,359

/files/contracts/full/1/:

"pagination": {
    ...
    "totalResults": 132056
  }

/files/contracts/any/1/:

"pagination": {
    ...
    "totalResults": 273320
  }

The difference (i.e. partial count) gives 141,264 whereas from stats.json we've got 529,877.

The bug can be resolved by adding a COALESCE(creation_match, '') to make the NULL values '' for comparisons.

Potential issues

Deduplication not working in compiled_contracts

When debugging the tests I've noticed the identical test contracts are getting added to the compiled_contracts table multiple times. I'd have expected them to be deduplicated because the table has a unique key property in the schema definition:

sourcify/services/database/migrations/20231109160022-create-alliance-schema.js

Line 204 in 653952d

CONSTRAINT compiled_contracts_pseudo_pkey UNIQUE (compiler, language, creation_code_hash, runtime_code_hash)

However during testing the contracts get added multiple times:

SELECT compiler, "language", creation_code_hash, runtime_code_hash FROM public.compiled_contracts x

Am I missing something or is there something wrong?

To reproduce

1. add a breakpoint at
   

   sourcify/services/server/test/integration/repository-handlers/files.spec.ts

   Line 56 in 653952d

   const address = await deployAndVerifyContract(

2. Add an .only modifier to describe:
   

   sourcify/services/server/test/integration/repository-handlers/files.spec.ts

   Line 43 in 653952d

   describe.only(`Pagination in /files/contracts/{full|any|partial}/${chainFixture.chainId}`, async function () {

3. run the Mocha - Server Integration, deploy and verify multiple contracts, and connect to the test database at localhost:5431.

matchType in database_utils.ts

While looking at the database_utils.ts code I've noticed the matchType variable is not used in functions:

sourcify/services/server/src/server/services/utils/database-util.ts

Line 429 in eeec824

export async function insertSourcifySync(

sourcify/services/server/src/server/services/utils/database-util.ts

Line 452 in eeec824

export async function updateSourcifySync(

But it seems these functions themselves are not used at all. Can they be removed?

[marcocastignoli]

Deduplication not working in compiled_contracts

I cannot understand why this happens. Maybe the unique constraint doesn't work with a NULL column?

But it seems these functions themselves are not used at all. Can they be removed?

Right now they are not used. If we need them for the rebuilding the database we can implement them again, you can remove them

[marcocastignoli]

The bug can be resolved by adding a COALESCE(creation_match, '') to make the NULL values '' for comparisons.

I didn't know about it, good catch!

[marcocastignoli]

Just a doubt on why random is used, but other than that LGTM 👍

[kuzdogan]

Deduplication not working in compiled_contracts

I cannot understand why this happens. Maybe the unique constraint doesn't work with a NULL column?

Ok I'll spin out another issue for the deduplication issue in `compiled_contracts

But it seems these functions themselves are not used at all. Can they be removed?

Right now they are not used. If we need them for the rebuilding the database we can implement them again, you can remove them

Ok but I don't really understand the use of the matchType variable there. I can keep the functions by noting they are unused for now, but the matchType is not used inside that function at all.

sourcify/services/server/src/server/services/utils/database-util.ts

Lines 429 to 451 in 28c6827

	export async function insertSourcifySync(
	pool: Pool,
	{ chain_id, address, match_type }: Tables.SourcifySync
	) {
	// I'm doing this because before passing the match_type I'm calling getMatchStatus(match)
	// but then I need to convert the status to full_match | partial_match
	let matchType;
	switch (match_type) {
	case "perfect":
	matchType = "full_match";
	break;
	case "partial":
	matchType = "partial_match";
	break;
	}
	await pool.query(
	`INSERT INTO sourcify_sync
	(chain_id, address, match_type, synced)
	VALUES ($1, $2, $3, true)`,
	[chain_id, address, match_type]
	);
	}

[marcocastignoli]

Ok but I don't really understand the use of the matchType variable there. I can keep the functions by noting they are unused for now, but the matchType is not used inside that function at all.

sourcify/services/server/src/server/services/utils/database-util.ts

Lines 429 to 451 in 28c6827

	export async function insertSourcifySync(
	pool: Pool,
	{ chain_id, address, match_type }: Tables.SourcifySync
	) {
	// I'm doing this because before passing the match_type I'm calling getMatchStatus(match)
	// but then I need to convert the status to full_match | partial_match
	let matchType;
	switch (match_type) {
	case "perfect":
	matchType = "full_match";
	break;
	case "partial":
	matchType = "partial_match";
	break;
	}
	await pool.query(
	`INSERT INTO sourcify_sync
	(chain_id, address, match_type, synced)
	VALUES ($1, $2, $3, true)`,
	[chain_id, address, match_type]
	);
	}

Do you remember that inside sourcify_sync we had some values with full_match and some with perfect. The reason is that this function wasn't doing what it supposed to be doing, instead of inserting match_type, it should have inserted matchType. It's a mistake. As I told you that function is not used anymore, so we can remove it.