Sync 20230111 (#90)

* Bring back profiling fetchBlocksBatches, remove unnessary await (#1235)

* [SKIP CI] Prerelease

* schema migration (#1226)

* [SKIP CI] Prerelease

* Fix Queue exceeds max size error (#1239)

* [SKIP CI] Prerelease

* fix config db selection when subscription is enabled (#1241)

* [SKIP CI] Prerelease

* Fork OrderByAggregatesPlugin and modify to be compatible with historical feature (#1242)

* [SKIP CI] Prerelease

* Fix block height being applied to all queries (#1238)

* [SKIP CI] Prerelease

* Fix running custom ds processors in parallel on different data (#1243)

* [SKIP CI] Prerelease

* [release] 20220811 (#1245)

* [release] 20220811

* update changelog

* support bulkGet and add bulkUpdate  (#1246)

* add bulkUpdate

* tidy up

* support historical

* improve bulkUpdate

* fix

* [SKIP CI] Prerelease

* Fix issue with db sync and workers enabled (#1250)

* Fix issue with db sync and workers enabled. Improved worker error handling

* Remove cpu count workers limit

* [SKIP CI] Prerelease

* Improve Performance logging (#1244)

* profilewrap for handler

* wip

* tidying up

* remove console.log

* tidy up

* add transactions

* Subql init mono (#1236)

* draft

* Tidy up

* tidy up

* Fix tests

* [SKIP CI] Prerelease

* [SKIP CI] Prerelease

* Fix bigInt array entity being treated as a big int (#1252)

* [SKIP CI] Prerelease

* [release] 20220817 (#1254)

* [release] 20220817

* Fix typo

* Address comments

* Update packages/types/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

Co-authored-by: Marta Adamczyk <[email protected]>

* Update README.md

* Node core (#1222)

* both packages working

* working on node core

* avalanche with types avalanche

* adding provider

* provider extends ethersproject provider

* Fix build issues

* Remove avalanche packages

* Add noImplicitAny flag to node-core and fix errors

* Update build tooling and changelogs

* Move events to node-core

* Move store service to node-core

* Use logger/profiler from node-core

* Move poi service to node core

* Move mmr service to node-core

* Move over meta controllers/services to node-core

* Fix imports

* Fix build issues

* Fix linter warning

* Fix lint issues, move worker builder to node core

* Remove BlockedQueue, fix more lint errors

* Move some test files

* Fix rebase issues, remove avax deps from node core

* Fix build issue

* Fix some tests

* fix processBlockCount metric with workers

Co-authored-by: roduquen <[email protected]>
Co-authored-by: JQQQ <[email protected]>

* Add limit and offset options to store.getByField (#1259)

* Add limit and offset options to store.getByField

* Revert types change

* Add delay and don't break loop when process queue is full (#1261)

* Update cmd options in subql-node README (#1265)

* Update README.md

* Update packages/node/README.md

* Fix symbol type being possible in string interpolation (#1264)

* Fix symbol type being possible in string interpolation (#1266)

* [SKIP CI] Prerelease

* [release] 20220826 (#1263)

* Fix imports not being relative (#1268)

* [SKIP CI] Prerelease

* [release] node-core 20220826 (#1269)

* prerelease with @subql/node-core (#1270)

* [SKIP CI] Prerelease

* fix node-core import in node (#1272)

* [SKIP CI] Prerelease

* [release] 20220829 (#1273)

* Move auto queue to node core (#1278)

* [SKIP CI] Prerelease

* update polkadot and types, to support with wasm (#1280)

* update polkadot and types, to support with wasm

* move logic to ds processor, update change for interface

* remove dsAssets

* remove dsAssets

* update

* fix fetch due to type changed (#1282)

* fix fetch due to type changed

* prerelease for types

* [SKIP CI] Prerelease

* fix missing await for dictionary (#1283)

* [SKIP CI] Prerelease

* rollback changes (#1286)

* [SKIP CI] Prerelease

* Fix tests failing with  Cannot read properties of null (reading 'getProcessedBlockCount') (#1285)

* Fix tests failing with  Cannot read properties of null (reading 'getProcessedBlockCount')

* Fix same error in more tests

* Fix lint issue

* [SKIP CI] Prerelease

* fix remote gitpath (#1288)

* [SKIP CI] Prerelease

* Fix tests (#1297)

* fix tests

* tidy up

* removed v0.0.1 test

* [release] 20220913 (#1291)

* [release] 20220913

* Updated CHANGELOG.MD

Co-authored-by: Marta Adamczyk <[email protected]>

Co-authored-by: Marta Adamczyk <[email protected]>

* Set useragent headers on polkadot api (#1296)

* [SKIP CI] Prerelease

* change dictionary to use equalToInsensitive (#1301)

* change dictionary to use equalToInsensitive

* add comments

* [SKIP CI] Prerelease

* [release] 20220919 (#1302)

* [release] 20220919

* Update packages/node/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

Co-authored-by: Marta Adamczyk <[email protected]>

* update sequelize (#1311)

* update sequelize

* update sequelize query

* [SKIP CI] Prerelease

* Subcommands for force-clean and reindex (#1281)

* draft pr for documentation

* merge

* force-clean command working, refactoring needed

* refactor force-clean

* unsafe bug, in sandboxService, export package.json bug

* updated package.json node-core

* subql-node starting

* refactor

* fix up

* node test suites still broken

* Fix init logger with tests

* Use old logger defaults

* tests passes

* tidy up

* builds

* Fix exporting logger (#1300)

* tidy up, skip db.module tests

* fixed db.module

* tidy up getExistingProjectSchema

* Reindex command (#1307)

* init reindex command

* add services to module

* reindex functioning

* add force-clean for issue  #1230

* tidy up

* tidy up based on comments

* remove comments

* edit command des

* remove reindex from nodeconfig

* moved functions into node-core from utils

* update version

Co-authored-by: Scott Twiname <[email protected]>

* Fixes #1309 (#1314)

* fix publish failed due to file object is null (#1316)

* [SKIP CI] Prerelease

* [release] 20220924 (#1317)

* chnage from multi branch to main (#1318)

* fix logger undefined (#1315)

* [SKIP CI] Prerelease

* change max connection to 10 and query service 10000 (#1322)

Co-authored-by: Louise wang <[email protected]>

* [SKIP CI] Prerelease

* update polkadot api to 9.4.2 (#1325)

* functioning hot schema reload (#1321)

* [SKIP CI] Prerelease

* improve codegen error log (#1324)

Co-authored-by: Louise wang <[email protected]>

* change createproject URL , project create issue, regx for project nam… (#1323)

* change createproject URL , project create issue, regx for project name and project key

* project name can be cap case

* code improve by Jay comment

Co-authored-by: Louise wang <[email protected]>

* [SKIP CI] Prerelease

* manual prerelease (#1326)

* [SKIP CI] Prerelease

* enable historical by default (#1327)

* [SKIP CI] Prerelease

* [release] 20220929 (#1328)

* [release] 20220929

* update

* Update packages/node/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

* Update packages/query/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

* Update packages/utils/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

* Update packages/query/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

* Update packages/node/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

* Update packages/common/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

Co-authored-by: JQQQ <[email protected]>
Co-authored-by: Marta Adamczyk <[email protected]>

* Update ipfs endpoints, improve error handling (#1337)

* [SKIP CI] Prerelease

* Fix nodeConfig not being provided to benchmark service (#1338)

* [SKIP CI] Prerelease

* [release] 20221006 (#1339)

* Remove Subqueries table (#1340)

* Remove Subqueries table

* Fix NOT_SUPPORT function in project

* Remove tests relating to old subqueries table

* [SKIP CI] Prerelease

* [release] 20221010 (#1342)

* [release] 20221010

* Update packages/node/CHANGELOG.md

Co-authored-by: Marta Adamczyk <[email protected]>

Co-authored-by: Marta Adamczyk <[email protected]>

* Validate ds processor in worker threads (#1343)

* [SKIP CI] Prerelease

* Fix publish ipfs 503 (#1331)

* add comments

* publish ipfs issue

* publish ipfs issue

* Update packages/cli/src/controller/publish-controller.ts

Co-authored-by: bz888 <[email protected]>
Co-authored-by: Louise wang <[email protected]>
Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Move common dictionary service to node-core (#1345)

* Move common dictionary service to node-core

* Apply matcher from cosmos dictionary

* Move sandbox code to node-core

* Remove testing log

* [SKIP CI] Prerelease

* [release] 20221011

* Back port changes in algorand sdk to dictionary service (#1346)

* [SKIP CI] Prerelease

* temp disable hot schema reload (#1349)

* temp disable hot schema reload

* add CLI flag for hotschema reload

* fix

* print node version (#1348)

* print node version

* update logger info

* update with node name

Co-authored-by: Louise wang <[email protected]>

* [SKIP CI] Prerelease

* [SKIP CI] Prerelease

* fix output-format flag on node (#1351)

* fix typo

* update workers and yargs outputFmt

* [SKIP CI] Prerelease

* Timestamp block filters (#1310)

* add block timestamp filter

* bug fixes

* bug fixes

* clean code and write test for timestamp filter

* move cron generation to SubqueryProject

* fix next timestamp log bug

* remove modifiedDataSources

* [SKIP CI] Prerelease

* Update to latest polkadot api (#1356)

* [SKIP CI] Prerelease

* add ad flags (#1352)

* add ad flags

* add interface for all AD flags and add prefix for indexer/query flags

* change string of to for

* some variable name change and update interface

Co-authored-by: Louise wang <[email protected]>

* support version without v (#1354)

* support version without v

* improvement addv FUNC and solved undefined input

Co-authored-by: Louise wang <[email protected]>

* fix deploy-test (#1357)

* [SKIP CI] Prerelease

* add dedicate data base flag for create project (#1355)

* add dedicate data base flag for create project

* test with latest createproject api host service and update wording

Co-authored-by: Louise wang <[email protected]>

* Optimise modulo filter (#1358)

* scale batch size if modulo filter used

* add comments

* add comments

* fix logic runModuloblock

* remove comments

* remove more comments

* add check datasources, check height logic

* removed comments

* refactor

* flatten handler array

* Fix template issues (#1363)

* Fix the problem of creating multiple templates in the same block with the same template address

* refactor: cache ds in same block

Co-authored-by: Orion <[email protected]>

* fix deployment and project tests (#1364)

* fix deployment and project tests

* fix linting error

* [SKIP CI] Prerelease

* [release] 20221027 (#1365)

* fix: incomplete ds data in same block (#1370)

Co-authored-by: Orion <[email protected]>

* [SKIP CI] Prerelease

* Best Block (#1308)

* Draft

* update changes

* rebase changes

* fix

* fix (#1329)

* disable best blocks for workers until we support it

* Rename bestBlock to unfinalizedBlocks, other clean up

* Clean up

* Further clean up

* Dedupe reindex function

* Fix tests

* Clean up

* Update checking finalization to use parent hash

* Rename logger

* Refactor unfinalized blocks

* Use header rather than full block, improve detecting forks

* Verify unfinalized blocks when disabled.

Use sorted array for storing unfinalized blocks

* Clean up logs

* Fix not indexing unfinalized blocks right away, exit if historical not enabled

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Fix bugs with unfinalized blocks (#1374)

* Fix bugs with unfinalized blocks

- Fix using wrong hash for unfinalized blocks
- Fix not removing unfinalized blocks when the latest unfinalized block < finalized

* Fix tests getting stuck

* Fix issue finding where fork occurred

* [SKIP CI] Prerelease

* [release] 20221028 (#1372)

* hot fix tests (#1360)

* update tests: api.service.spec, init-controller.test, publish-controller.spec

* update tests following comments

* [SKIP CI] Prerelease

* add ethereum to CLI and Validator (#1378)

* Imporve dictionary query (#1371)

* wip

* wip

* wip

* broken wip

* wip refactoring

* seperate dictionaryQueryEntries

* fix logic for setDicitonaryQueryEntries

* cleaning up

* move dictionaryEntry.ts to node-core

* remove comments

* create class for dictionaryqueryentries

* update tests

* move dictionaryQueryService into dicitonaryService

* refactor

* update logic for useDicitonary

* relocated dictionaryQuery funcs

* add test for dictquerymap

* tests failing on windows?

* test getDictionaryQueryEntries

* test fixed for getDictioanryQueryEntries

* added test for sorting

* conflict fixed ?

* fix conflict_2

* fix conflict_3

* add delete temp ds records back

* clean up with new logic

* clean up, add comments for test

* fix

* update logic

* add generic type

* [SKIP CI] Prerelease

* Enable for better inheritance of generated entity modeld (#1377)

* refactor: enable for entity inheritance

* refactor: include also filed getters

* [SKIP CI] Prerelease

* fix comments issue with new package (#1380)

* fix comments issue with new package

* moved yaml package

* [SKIP CI] Prerelease

* fix logic with reindex and unfinalized height and dynamic ds (#1382)

* fix logic with reindex and unfinalized height

* fix

* include fix for #1379

* update polkadot to 9.7.1 (#1384)

* [release] 20221107

* Fix remove alter table (#1387)

* remove migrate alter table

* remove

* [SKIP CI] Prerelease

* [release] 20221108 (#1388)

* fix missing sequelize sync (#1389)

* [SKIP CI] Prerelease

* [release] 20221108 patch (#1390)

* reindex bind (#1391)

* [SKIP CI] Prerelease

* [release] 20221109 (#1393)

* Handle fetch errors, then retry (#1386)

* add retryOnFail function

* add retryOnFail

* add test, fix logic

* [SKIP CI] Prerelease

* fix (#1395)

* [SKIP CI] Prerelease

* [release] 20221109 node-core (#1394)

* Fix tests hanging (#1396)

* Fix tests hanging

* Update base docker image with newer git version

* [SKIP CI] Prerelease

* Add distinct query plugin (#1274)

* Add distinct query plugin

* Clean up log

* Fix distinct not being provided to query

* Uppercase enum to be consistent with other enums

* Update dictionary queries to try distinct argument

* [SKIP CI] Prerelease

* Add query distinct dependencies (#1398)

* fix missing update forked graphile dependencies

* tidy up

* tidy up

* [SKIP CI] Prerelease

* Break block dispatcher file up and move common code to base class (#1397)

* [SKIP CI] Prerelease

* Hot schema trigger (#1401)

* implement trigger with notification

* working prior clean up

* refactor and clean up on async and await

* clean up

* clear comments

* add filter

* fix

* fix err

* [SKIP CI] Prerelease

* [release] 20221115 (#1402)

* [release] 20221115

* [release] 20221115

* [release] 20221115

* fix hot schema (#1404)

* fix and refactor

* refactor getTriggers

* [SKIP CI] Prerelease

* [release] 20221115 (#1408)

* [release] 20221115

* [release] 20221115

* fix fetchblock for works (#1410)

* [SKIP CI] Prerelease

* fix row estimate (#1417)

* fix row estimate

* tidy up

* [SKIP CI] Prerelease

* Add support to cli/validator for flare (#1416)

* [SKIP CI] Prerelease

* missing validator missing ipfs chainTypes (#1419)

* [SKIP CI] Prerelease

* query explain include graphql query (#1426)

* [SKIP CI] Prerelease

* dictionary auth link integration (#1411)

* add required dependancies and add flag for feature

* update modules and add logic to dictionary

* update dictionary parameters, update tests, remove hardcoded values

* simplify auth dictionary handling

* add apollo link type, remove un-needed from wrapper

* remove dictionary from indexer module

* update flag name, add error check

* update yargs flag description

* Update appolo-links

* update @subql/apollo-links version, remove apollo/client version change

* fix yarn.lock

* add await init to dictionary tests

* fix fetch test dictionary provider, update yarn.lock

Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: HZ <[email protected]>

* Fix missing speChanged logic with getRuntime (#1421)

* init

* need fix worker

* fix test

* tidy up

* fix test, fix missing dictionary

* inject service to runtime

* remove unused code

* [SKIP CI] Prerelease

* Feat/multi chain indexing (#1375)

* [SKIP CI] Prerelease

* remove enums (#1427)

* [SKIP CI] Prerelease

* change to print graphql sql on request (#1428)

* [SKIP CI] Prerelease

* fix missing util-crptyo in util package (#1429)

* fix missing util-crptyo in util package

* need prerelease in query service

* [SKIP CI] Prerelease

* fix import issue in query service (#1430)

* [SKIP CI] Prerelease

* fix missing query (#1432)

* [SKIP CI] Prerelease

* change fetch module SubqueryProject => ISubqueryProject (#1433)

Co-authored-by: Lachlan McCrae <[email protected]>

* [SKIP CI] Prerelease

* Update polkadot 9.9.1 (#1434)

* update polkadot api, also fix force-clean dependencies issue

* tidy up

* [SKIP CI] Prerelease

* [release] 20221123 (#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* Add a way to pass settings to the graphql playground (#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <[email protected]>

* [SKIP CI] Prerelease

* add try and catch for hsr (#1449)

* [SKIP CI] Prerelease

* Bypass blocks (#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (#1463)

* [SKIP CI] Prerelease

* update deploy cli (#1460)

* [SKIP CI] Prerelease

* exit on fail (#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* update

Co-authored-by: Scott Twiname <[email protected]>

* Fix templates with other sdks, make -f flag work like other commands (#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (#1471)

* [SKIP CI] Prerelease

* Fix trigger function (#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (#1474)

* validate dictionary with start height (#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* add count for entity (#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* add dictionary validation expections

Co-authored-by: Scott Twiname <[email protected]>
Co-authored-by: Jay Ji <[email protected]>
Co-authored-by: Marta Adamczyk <[email protected]>
Co-authored-by: James Bayly <[email protected]>
Co-authored-by: roduquen <[email protected]>
Co-authored-by: HAO ZHAI <[email protected]>
Co-authored-by: bighammerw <[email protected]>
Co-authored-by: Louise wang <[email protected]>
Co-authored-by: Naveen V <[email protected]>
Co-authored-by: hariu-starfish <[email protected]>
Co-authored-by: Orion <[email protected]>
Co-authored-by: Filippo <[email protected]>
Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: MOZGIII <[email protected]>

package.json

@@ -33,8 +33,8 @@
     "typescript": "^4.4.4"
   },
   "resolutions": {
-    "@polkadot/api": "9.5.2",
-    "@polkadot/util": "10.2.1",
+    "@polkadot/api": "9.11.1",
+    "@polkadot/util": "10.2.3",
     "@cosmjs/cosmwasm-stargate": "0.28.9",
     "node-fetch": "2.6.7"
   },

packages/node/package.json

@@ -30,7 +30,7 @@
     "@polkadot/api": "9.5.2",
     "@subql/common": "latest",
     "@subql/common-cosmos": "workspace:*",
-    "@subql/node-core": "1.6.1-1",
+    "@subql/node-core": "1.7.2-2",
     "@subql/types-cosmos": "workspace:*",
     "@subql/utils": "latest",
     "@subql/x-merkle-mountain-range": "2.0.0-0.1.2",

packages/node/src/indexer/fetch.service.ts

@@ -52,6 +52,7 @@ const DICTIONARY_MAX_QUERY_SIZE = 10000;
 const CHECK_MEMORY_INTERVAL = 60000;
 const MINIMUM_BATCH_SIZE = 5;
 const INTERVAL_PERCENT = 0.9;
+const DICTIONARY_VALIDATION_EXCEPTIONS = ['juno-1'];
 
 export function eventFilterToQueryEntry(
   filter: SubqlCosmosEventFilter,
@@ -115,14 +116,12 @@ export function messageFilterToQueryEntry(
 
 @Injectable()
 export class FetchService implements OnApplicationShutdown {
-  private latestBestHeight: number;
   private latestFinalizedHeight: number;
   private isShutdown = false;
   private batchSizeScale: number;
   private templateDynamicDatasouces: SubqlProjectDs[];
   private dictionaryGenesisMatches = true;
   private bypassBlocks: number[] = [];
-  private bypassBufferHeight: number;
 
   constructor(
     private apiService: ApiService,
@@ -285,9 +284,27 @@ export class FetchService implements OnApplicationShutdown {
 
     await this.blockDispatcher.init(this.resetForNewDs.bind(this));
 
+    //  Call metadata here, other network should align with this
+    //  For substrate, we might use the specVersion metadata in future if we have same error handling as in node-core
+    const metadata = await this.dictionaryService.getMetadata();
+
+    const validChecker = await this.dictionaryValidation(metadata);
+
+    if (validChecker) {
+      this.dictionaryService.setDictionaryStartHeight(
+        metadata._metadata.startHeight,
+      );
+    }
+
+    await this.blockDispatcher.init(this.resetForNewDs.bind(this));
     void this.startLoop(startHeight);
   }
 
+  // Substrate exclusive for getRuntime
+  // getUseDictionary(): boolean {
+  //   return this.useDictionary;
+  // }
+
   @Interval(CHECK_MEMORY_INTERVAL)
   checkBatchScale(): void {
     if (this.nodeConfig['scale-batch-size']) {
@@ -374,6 +391,14 @@ export class FetchService implements OnApplicationShutdown {
         : initBlockHeight;
     };
 
+    if (this.dictionaryService.startHeight > getStartBlockHeight()) {
+      logger.warn(
+        `Dictionary start height ${
+          this.dictionaryService.startHeight
+        } is beyond indexing height ${getStartBlockHeight()}, skipping dictionary for now`,
+      );
+    }
+
     while (!this.isShutdown) {
       startBlockHeight = getStartBlockHeight();
 
@@ -390,7 +415,10 @@ export class FetchService implements OnApplicationShutdown {
         continue;
       }
 
-      if (this.useDictionary) {
+      if (
+        this.useDictionary &&
+        startBlockHeight >= this.dictionaryService.startHeight
+      ) {
         const queryEndBlock = startBlockHeight + DICTIONARY_MAX_QUERY_SIZE;
         const moduloBlocks = this.getModuloBlocks(
           startBlockHeight,
@@ -526,8 +554,13 @@ export class FetchService implements OnApplicationShutdown {
     if (dictionary !== undefined) {
       const { _metadata: metaData } = dictionary;
 
-      const chain = await this.api.getChainId();
-      if (metaData.chain !== chain) {
+      const chain = await this.api.getChainId().catch();
+
+      // Exception for juno as juno does not have a genesisHash
+      if (
+        metaData.chain !== chain &&
+        !DICTIONARY_VALIDATION_EXCEPTIONS.find((ele) => ele === chain)
+      ) {
         logger.error(
           'The dictionary that you have specified does not match the chain you are indexing, it will be ignored. Please update your project manifest to reference the correct dictionary',
         );
@@ -548,17 +581,12 @@ export class FetchService implements OnApplicationShutdown {
           return false;
         }
       }
+
       return true;
     }
     return false;
   }
 
-  async resetForIncorrectBestBlock(blockHeight: number): Promise<void> {
-    await this.syncDynamicDatascourcesFromMeta();
-    this.updateDictionary();
-    this.blockDispatcher.flushQueue(blockHeight);
-  }
-
   private getBaseHandlerKind(
     ds: SubqlCosmosDataSource,
     handler: SubqlCosmosHandler,

packages/types/src/interfaces.ts

@@ -15,6 +15,7 @@ export type FunctionPropertyNames<T> = {
 }[keyof T];
 
 export interface Store {
+  count(entity: string, field?: string, value?: any, options?: {distinct?: boolean; col?: string}): Promise<number>;
   get(entity: string, id: string): Promise<Entity | null>;
   getByField(entity: string, field: string, value: any, options?: {offset?: number; limit?: number}): Promise<Entity[]>;
   getOneByField(entity: string, field: string, value: any): Promise<Entity | null>;