Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the issue that may occur when using a dictionary to crawl blocks. #2648

Merged
merged 3 commits into from
Jan 25, 2025
Merged

Fix the issue that may occur when using a dictionary to crawl blocks. #2648

merged 3 commits into from
Jan 25, 2025

Conversation

yoozo
Copy link
Collaborator

@yoozo yoozo commented Jan 21, 2025 •
edited
Loading

Description

When setting a smaller batch size and the current processing height reaches latestFinalizedHeight, it causes the program to exit unexpectedly. Will cause the following error.

2025-01-21T07:09:18.745Z <BlockDispatcherService> INFO Enqueueing blocks 21671197...21671197, total 1 blocks 
2025-01-21T07:09:19.661Z <BlockDispatcherService> INFO Enqueueing blocks 21671197...21671197, total 1 blocks 
2025-01-21T07:09:19.932Z <BlockDispatcherService> ERROR Failed to index block at height 21671197  AssertionError: Block processed out of order. Height: 21671197. Latest: 21671197
2025-01-21T07:09:19.933Z <BlockDispatcherService> ERROR undefined Error: Failed to enqueue fetched block to process
Cause: AssertionError: Block processed out of order. Height: 21671197. Latest: 21671197

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • I have tested locally
  • I have performed a self review of my changes
  • Updated any relevant documentation
  • Linked to any relevant issues
  • I have added tests relevant to my changes
  • Any dependent changes have been merged and published in downstream modules
  • My code is up to date with the base branch
  • I have updated relevant changelogs. We suggest using chan

@yoozo yoozo requested a review from stwiname January 21, 2025 07:11
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 21, 2025 •
edited
Loading

Coverage report for .

Caution

Test run failed

St.❔
 Category Percentage Covered / Total
🟡 Statements
68.12% (-1.99% 🔻)
 20877/30648
🟡 Branches
79.86% (+1.56% 🔼)
 2874/3599
🟡 Functions
71.52% (+9.02% 🔼)
 1175/1643
🟡 Lines
68.12% (-1.99% 🔻)
 20877/30648
Show new covered files 🐣
St.❔
 File Statements Branches Functions Lines
🟢
... / migrate.ts
 85.34% 16.67% 100% 85.34%
🟢
... / migrate-abis.controller.ts
 95% 87.5% 100% 95%
🟢
... / constants.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / ethereum.ts
 82.28% 83.33% 100% 82.28%
🟢
... / migrate-manifest.controller.ts
 88% 73.33% 88.89% 88%
🟢
... / migrate-mapping.controller.ts
 100% 100% 100% 100%
🟢
... / migrate-controller.ts
 92.41% 85.19% 100% 92.41%
🟢
... / migrate.fixtures.ts
 100% 100% 100% 100%
🟢
... / migrate-schema.controller.ts
 97.23% 79.49% 100% 97.23%
🔴
... / types.ts
 0% 0% 0% 0%
🟢
... / createProject.fixtures.ts
 94.92% 77.78% 100% 94.92%
🟢
... / config.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / moduleLoader.ts
 56% 40% 100% 56%
🟢
... / types.ts
 100% 100% 100% 100%
🔴
... / utils.ts
 0% 0% 0% 0%
🟢
... / networkFamily.ts
 92.59% 88.89% 100% 92.59%
🟡
... / IPFSHTTPClientLite.ts
 75.74% 80% 88.89% 75.74%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / utils.ts
 92% 66.67% 100% 92%
🟡
... / admin.controller.ts
 75.56% 96% 78.57% 75.56%
🟡
... / blockRange.ts
 68.52% 100% 80% 68.52%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / SubqueryProject.ts
 85.52% 83.33% 71.43% 85.52%
🟢
... / core.module.ts
 100% 100% 100% 100%
🟢
... / dictionary.fixtures.ts
 100% 100% 100% 100%
🟡
... / monitor.service.ts
 69.61% 85.88% 94.12% 69.61%
🔴
... / sandbox.service.ts
 33.33% 50% 25% 33.33%
🟢
... / baseCache.service.ts
 94.59% 92.31% 83.33% 94.59%
🟢
... / baseStoreModel.service.ts
 93.22% 83.33% 100% 93.22%
🟢
... / cacheable.ts
 100% 100% 100% 100%
🟡
... / csvStore.ts
 73.96% 73.68% 85.71% 73.96%
🟡
... / exporter.ts
 70% 71.43% 71.43% 70%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / cacheMetadata.ts
 82.82% 76.74% 93.33% 82.82%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / metadata.ts
 53.01% 71.43% 55.56% 53.01%
🟢
... / utils.ts
 100% 100% 100% 100%
🟢
... / cacheModel.ts
 84.77% 78.75% 90% 84.77%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / model.ts
 88.89% 72% 100% 88.89%
🟢
... / utils.ts
 100% 100% 100% 100%
🟡
... / cachePoi.ts
 76.84% 89.47% 85.71% 76.84%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / poi.ts
 41.07% 71.43% 44.44% 41.07%
🟢
... / setValueModel.ts
 85.26% 83.78% 85.71% 85.26%
🟢
... / storeCache.service.ts
 80% 74.19% 80% 80%
🟡
... / storeModel.service.ts
 64.84% 61.54% 71.43% 64.84%
🟢
... / types.ts
 100% 100% 100% 100%
🟢
... / utils.ts
 93.75% 66.67% 100% 93.75%
🟢
... / worker.core.module.ts
 100% 100% 0% 100%
🔴
... / worker.monitor.service.ts
 39.53% 100% 0% 39.53%
🔴
... / meta.controller.ts
 53.33% 100% 0% 53.33%
🟡
... / meta.module.ts
 60% 100% 0% 60%
🟡
... / process.ts
 61.29% 66.67% 40% 61.29%
🔴
... / foreceClean.init.ts
 47.62% 100% 0% 47.62%
🔴
... / reindex.init.ts
 33.33% 100% 0% 33.33%
🟡
... / autoQueue.ts
 64.88% 87.18% 73.33% 64.88%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / queue.ts
 45.83% 80% 66.67% 45.83%
🟡
... / rampQueue.ts
 77.66% 91.3% 87.5% 77.66%
🟢
... / string.ts
 100% 95.45% 100% 100%
🔴
... / datasourceProcessors.ts
 0% 0% 0% 0%
🔴
... / index.ts
 0% 0% 0% 0%
🔴
... / types.ts
 0% 0% 0% 0%
🔴
... / networking.ts
 31.58% 100% 0% 31.58%
Show files with reduced coverage 🔻
St.❔
 File Statements Branches Functions Lines
🔴
... / delete.ts
48.72% (-5.57% 🔻)
 100%
50% (+50% 🔼)
48.72% (-5.57% 🔻)
🔴
... / promote.ts
44.19% (-4.53% 🔻)
 100%
50% (+50% 🔼)
44.19% (-4.53% 🔻)
🟡
... / add.ts
61.29% (-0.78% 🔻)
 100%
50% (+50% 🔼)
61.29% (-0.78% 🔻)
🟢
... / publish.ts
83.72% (-5.61% 🔻)
60% (+2.86% 🔼)
 100%
83.72% (-5.61% 🔻)
🔴
... / add-chain-controller.ts
39.85% (-0.23% 🔻)
 66.67% 30%
39.85% (-0.23% 🔻)
🟢
... / build-controller.ts
90.44% (-0.86% 🔻)
71.43% (+14.29% 🔼)
 100%
90.44% (-0.86% 🔻)
🟢
... / codegen-controller.ts
88.68% (-0.71% 🔻)
83.95% (+8.37% 🔼)
 100%
88.68% (-0.71% 🔻)
🟡
... / deploy-controller.ts
64.94% (+27.48% 🔼)
72.73% (-5.05% 🔻)
64.29% (+39.29% 🔼)
64.94% (+27.48% 🔼)
🟢
... / generate-controller.ts
97.75% (+0.1% 🔼)
93.24% (-0.79% 🔻)
 100%
97.75% (+0.1% 🔼)
🟡
... / init-controller.ts
76.7% (+6.21% 🔼)
67.65% (-0.35% 🔻)
61.11% (+7.78% 🔼)
76.7% (+6.21% 🔼)
🟢
... / project-controller.ts
83.13% (-14.13% 🔻)
42.86% (-17.14% 🔻)
75% (-25% 🔻)
83.13% (-14.13% 🔻)
🟢
... / publish-controller.ts
82.63% (+2.38% 🔼)
70.27% (-0.84% 🔻)
 100%
82.63% (+2.38% 🔼)
🔴
... / models.ts
50% (-49.44% 🔻)
94.12% (+1.81% 🔼)
30.77% (-69.23% 🔻)
50% (-49.44% 🔻)
🟢
... / ProjectManifestVersioned.ts
82.19% (-2.74% 🔻)
88.89% (-11.11% 🔻)
61.54% (+3.21% 🔼)
82.19% (-2.74% 🔻)
🟢
... / model.ts
93.14% (-4.57% 🔻)
 100%
73.33% (+1.9% 🔼)
93.14% (-4.57% 🔻)
🔴
... / models.ts
47.06% (-52.94% 🔻)
 100%
0% (-100% 🔻)
47.06% (-52.94% 🔻)
🟡
... / load.ts
64.29% (+5.1% 🔼)
38.89% (-16.67% 🔻)
83.33% (+16.67% 🔼)
64.29% (+5.1% 🔼)
🟡
... / github-reader.ts
65.52% (-23.77% 🔻)
85.71% (+10.71% 🔼)
71.43% (-11.9% 🔻)
65.52% (-23.77% 🔻)
🟡
... / ipfs-reader.ts
61.9% (-23.81% 🔻)
72.73% (+9.09% 🔼)
71.43% (-11.9% 🔻)
61.9% (-23.81% 🔻)
🟡
... / local-reader.ts
67.5% (-18.55% 🔻)
100% (+37.5% 🔼)
83.33% (-16.67% 🔻)
67.5% (-18.55% 🔻)
🟢
... / reader.ts
81.58% (-7.89% 🔻)
57.14% (-17.86% 🔻)
 100%
81.58% (-7.89% 🔻)
🟢
... / base.ts
90.65% (-7.35% 🔻)
92.31% (-7.69% 🔻)
 80%
90.65% (-7.35% 🔻)
🔴
... / models.ts
49.1% (-50.9% 🔻)
 100%
33.33% (-66.67% 🔻)
49.1% (-50.9% 🔻)
🟡
... / ProjectUpgrade.service.ts
78.72% (-8.35% 🔻)
83.72% (-7.64% 🔻)
92.31% (+1% 🔼)
78.72% (-8.35% 🔻)
🟢
... / configure.module.ts
82.67% (-1.9% 🔻)
73.08% (+9.08% 🔼)
 100%
82.67% (-1.9% 🔻)
🟢
... / db.module.ts
89.63% (+1.35% 🔼)
62.5% (-20.83% 🔻)
 80%
89.63% (+1.35% 🔼)
🟢
... / SchemaMigration.service.ts
84.04% (-3.52% 🔻)
81.82% (+0.57% 🔼)
83.33% (-16.67% 🔻)
84.04% (-3.52% 🔻)
🟡
... / migration-helpers.ts
75.28% (-0.74% 🔻)
88.24% (+7.64% 🔼)
 90.91%
75.28% (-0.74% 🔻)
🟡
... / migration.ts
73.17% (-9.12% 🔻)
83.33% (+3.6% 🔼)
85% (-5.48% 🔻)
73.17% (-9.12% 🔻)
🟡
... / StoreOperations.ts
72.94% (-19.37% 🔻)
86.36% (+1.36% 🔼)
87.5% (-12.5% 🔻)
72.94% (-19.37% 🔻)
🔴
... / benchmark.service.ts
43.36% (-11.89% 🔻)
 66.67%
36.36% (-3.64% 🔻)
43.36% (-11.89% 🔻)
🔴
... / base-block-dispatcher.ts
17.61% (-29.74% 🔻)
 100%
4.55% (-5.45% 🔻)
17.61% (-29.74% 🔻)
🔴
... / block-dispatcher.ts
29.72% (-11.27% 🔻)
 66.67% 28.57%
29.72% (-11.27% 🔻)
🔴
... / worker-block-dispatcher.ts
15.02% (-35.19% 🔻)
 100%
0% (-27.27% 🔻)
15.02% (-35.19% 🔻)
🔴
... / coreDictionary.ts
45.33% (-54.67% 🔻)
87.5% (-6.94% 🔻)
77.78% (-22.22% 🔻)
45.33% (-54.67% 🔻)
🟡
... / dictionary.service.ts
68.47% (-16.67% 🔻)
83.78% (-1.93% 🔻)
83.33% (-7.58% 🔻)
68.47% (-16.67% 🔻)
🟡
... / dictionaryV1.ts
73.06% (-20.17% 🔻)
75% (-3.26% 🔻)
91.67% (-8.33% 🔻)
73.06% (-20.17% 🔻)
🟡
... / dictionaryV2.ts
78.42% (-3.25% 🔻)
71.88% (+6.66% 🔼)
88.89% (-11.11% 🔻)
78.42% (-3.25% 🔻)
🔴
... / ds-processor.service.ts
32.29% (-29.8% 🔻)
73.33% (-1.67% 🔻)
58.33% (-29.17% 🔻)
32.29% (-29.8% 🔻)
🟢
... / inMemoryCache.service.ts
86.21% (-13.79% 🔻)
 100%
75% (-25% 🔻)
86.21% (-13.79% 🔻)
🔴
... / indexer.manager.ts
16.55% (-16.66% 🔻)
 100%
10% (-2.5% 🔻)
16.55% (-16.66% 🔻)
🟢
... / PoiBlock.ts
83.87% (-14.52% 🔻)
 96.55%
75% (-25% 🔻)
83.87% (-14.52% 🔻)
🔴
... / poi.service.ts
55.7% (+31.78% 🔼)
57.69% (-42.31% 🔻)
80% (+51.43% 🔼)
55.7% (+31.78% 🔼)
🔴
... / poiSync.service.ts
55.88% (-31% 🔻)
77.61% (+1.05% 🔼)
88.89% (-3.42% 🔻)
55.88% (-31% 🔻)
🟡
... / project.service.ts
73.54% (-6.46% 🔻)
77.78% (+7.6% 🔼)
85.19% (+1.19% 🔼)
73.54% (-6.46% 🔻)
🔴
... / sandbox.ts
59.09% (-8.89% 🔻)
56.25% (-0.89% 🔻)
57.14% (-4.4% 🔻)
59.09% (-8.89% 🔻)
🟡
... / store.service.ts
60.52% (-5.52% 🔻)
75.41% (+6.99% 🔼)
80.65% (+23.5% 🔼)
60.52% (-5.52% 🔻)
🔴
... / entity.ts
50% (-11.54% 🔻)
 100%
50% (+50% 🔼)
50% (-11.54% 🔻)
🔴
... / store.ts
30% (-2.2% 🔻)
62.5% (-37.5% 🔻)
33.33% (+33.33% 🔼)
30% (-2.2% 🔻)
🟢
... / test.runner.ts
95.06% (-4.27% 🔻)
77.78% (+4.44% 🔼)
75% (-25% 🔻)
95.06% (-4.27% 🔻)
🔴
... / testing.service.ts
11.54% (-17.03% 🔻)
 100% 0%
11.54% (-17.03% 🔻)
🔴
... / unfinalizedBlocks.service.ts
56.21% (-27.78% 🔻)
75% (-21.72% 🔻)
92% (+0.33% 🔼)
56.21% (-27.78% 🔻)
🟡
... / worker.builder.ts
60.68% (-18.5% 🔻)
 65.38% 80%
60.68% (-18.5% 🔻)
🔴
... / worker.cache.service.ts
48.78% (-9.76% 🔻)
 100% 0%
48.78% (-9.76% 🔻)
🔴
... / worker.connectionPoolState.manager.ts
38.21% (-22.76% 🔻)
 100% 0%
38.21% (-22.76% 🔻)
🔴
... / worker.dynamic-ds.service.ts
44.44% (-17.78% 🔻)
 100% 0%
44.44% (-17.78% 🔻)
🔴
... / worker.service.ts
20.35% (-24.29% 🔻)
 100% 0%
20.35% (-24.29% 🔻)
🔴
... / worker.ts
41.18% (-1.28% 🔻)
 100% 0%
41.18% (-1.28% 🔻)
🔴
... / worker.unfinalizedBlocks.service.ts
26.79% (-73.21% 🔻)
 100%
0% (-100% 🔻)
26.79% (-73.21% 🔻)
🔴
... / logger.ts
47.89% (-16.59% 🔻)
 33.33%
40% (+20% 🔼)
47.89% (-16.59% 🔻)
🔴
... / event.listener.ts
15.15% (-65.66% 🔻)
 100%
0% (-25% 🔻)
15.15% (-65.66% 🔻)
🔴
... / health.controller.ts
37.93% (-17.24% 🔻)
 100%
0% (-50% 🔻)
37.93% (-17.24% 🔻)
🔴
... / health.service.ts
21.05% (-36.48% 🔻)
 100%
0% (-20% 🔻)
21.05% (-36.48% 🔻)
🔴
... / meta.service.ts
34.26% (-35.26% 🔻)
100% (+20% 🔼)
0% (-40% 🔻)
34.26% (-35.26% 🔻)
🔴
... / ready.controller.ts
34.78% (-21.74% 🔻)
 100%
0% (-50% 🔻)
34.78% (-21.74% 🔻)
🔴
... / ready.service.ts
37.5% (-54.17% 🔻)
 100%
0% (-66.67% 🔻)
37.5% (-54.17% 🔻)
🔴
... / forceClean.service.ts
21.05% (-5.26% 🔻)
 100% 0%
21.05% (-5.26% 🔻)
🔴
... / reindex.service.ts
18.62% (-14.71% 🔻)
 100% 0%
18.62% (-14.71% 🔻)
🟡
... / blocks.ts
71.21% (-28.79% 🔻)
91.67% (+2.78% 🔼)
75% (-25% 🔻)
71.21% (-28.79% 🔻)
🟡
... / project.ts
71.88% (-1.49% 🔻)
75% (+0.42% 🔼)
 77.78%
71.88% (-1.49% 🔻)
🟢
... / promise.ts
86.44% (-5.32% 🔻)
100% (+13.64% 🔼)
71.43% (-14.29% 🔻)
86.44% (-5.32% 🔻)
🟢
... / reindex.ts
82.05% (+48.01% 🔼)
14.29% (-85.71% 🔻)
100% (+100% 🔼)
82.05% (+48.01% 🔼)
🔴
... / yargs.ts
44.18% (-1.3% 🔻)
 100% 14.29%
44.18% (-1.3% 🔻)
🟢
... / apiPromise.connection.ts
80% (+10.92% 🔼)
61.54% (-18.46% 🔻)
66.67% (+23.81% 🔼)
80% (+10.92% 🔼)
🟡
... / block-dispatcher.service.ts
62.83% (-6.04% 🔻)
 100%
28.57% (+3.57% 🔼)
62.83% (-6.04% 🔻)
🔴
... / worker-block-dispatcher.service.ts
24.16% (-12.8% 🔻)
 100% 0%
24.16% (-12.8% 🔻)
🟢
... / substrateDictionary.service.ts
98.11% (+7.55% 🔼)
66.67% (-6.06% 🔻)
100% (+20% 🔼)
98.11% (+7.55% 🔼)
🟢
... / substrateDictionaryV1.ts
85.19% (-0.3% 🔻)
78.72% (+8.96% 🔼)
84.62% (+1.28% 🔼)
85.19% (-0.3% 🔻)
🟡
... / substrateDictionaryV2.ts
73.03% (-2.48% 🔻)
 75% 33.33%
73.03% (-2.48% 🔻)
🟢
... / fetch.module.ts
89.6% (-2.27% 🔻)
50% (-25% 🔻)
 100%
89.6% (-2.27% 🔻)
🔴
... / indexer.manager.ts
50% (-7.55% 🔻)
 100%
20% (+10.91% 🔼)
50% (-7.55% 🔻)
🟡
... / base-runtime.service.ts
72.17% (-0.15% 🔻)
76.92% (+1.92% 🔼)
77.78% (+2.78% 🔼)
72.17% (-0.15% 🔻)
🟢
... / runtimeService.ts
94.52% (+2.31% 🔼)
81.82% (-7.07% 🔻)
100% (+20% 🔼)
94.52% (+2.31% 🔼)
🟡
... / types.ts
70.45% (-14.84% 🔻)
 100% 0%
70.45% (-14.84% 🔻)
🟢
... / unfinalizedBlocks.service.ts
85.45% (-0.82% 🔻)
 100%
60% (+10% 🔼)
85.45% (-0.82% 🔻)
🟢
... / project.ts
88.73% (-0.16% 🔻)
 68.18% 87.5%
88.73% (-0.16% 🔻)
🟢
... / buffer.ts
90.24% (-9.76% 🔻)
91.3% (-8.7% 🔻)
90.48% (+6.27% 🔼)
90.24% (-9.76% 🔻)
🟢
... / builder.ts
 92.77%
82.14% (-0.62% 🔻)
 100% 92.77%
🟡
... / logger.ts
70.35% (+0.75% 🔼)
51.85% (-3.15% 🔻)
78.57% (+3.57% 🔼)
70.35% (+0.75% 🔼)

Test suite run failed

Failed tests: 1/611. Failed suites: 1/108. 
  ● Codegen can generate schema › Codegen should be able to generate ABIs from customName datasources

    expect(received).resolves.toBeTruthy()

    Received promise rejected instead of resolved
    Rejected to value: [Error: ENOENT: no such file or directory, open '/home/runner/work/subql/subql/packages/cli/test/schemaTest/src/types/abi-interfaces/Erc721.ts']

      47 |     const projectPath = path.join(__dirname, '../../test/schemaTest');
      48 |     await codegen(projectPath);
    > 49 |     await expect(
         |           ^
      50 |       fs.promises.readFile(`${projectPath}/src/types/abi-interfaces/Erc721.ts`, 'utf8')
      51 |     ).resolves.toBeTruthy();
      52 |   });

      at expect (node_modules/expect/build/index.js:105:15)
      at Object.<anonymous> (packages/cli/src/controller/codegen-controller.test.ts:49:11)

Report generated by 🧪jest coverage report action from efc7a81

stwiname
stwiname reviewed Jan 22, 2025
View reviewed changes
packages/node-core/src/indexer/fetch.service.ts
batchBlocks
);
if (mergedBlocks.length === 0) {
// There we're no blocks in this query range, we can set a new height we're up to
await this.enqueueBlocks([], dictionary.lastBufferedHeight);
if (startBlockHeight <= lastBufferedHeight) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a test for this so we don't have a regression.

@stwiname stwiname merged commit efdd476 into main Jan 25, 2025
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stwiname stwiname stwiname left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yoozo @stwiname