Add files indexers #977

marc-aurele-besner · 2024-11-20T19:23:35Z

User description

Add files indexers

Add an indexer that indexes cids, files, folders, metadata and their relations

This does not rebuild the file, store the data in plain text in the chunk table

PR Type

enhancement, documentation

Description

Added files indexers for both Mainnet and Taurus Testnet, including project configurations, handlers, and database functions.
Defined GraphQL schemas and TypeScript types for file-related entities.
Updated Docker and database configurations to include new services and tables.
Added documentation files, including README and LICENSE, for both Mainnet and Taurus Testnet projects.

Changes walkthrough 📝

Relevant files

Enhancement

12 files

project.ts `Add project configuration for Mainnet files indexer` indexers/mainnet/files/project.ts Added project configuration for Autonomys Mainnet files indexer. Defined network endpoints and data sources. Configured handlers for specific blockchain events.	+95/-0
index.ts `Export handler functions and import API augmentations` indexers/mainnet/files/src/index.ts Exported all handler functions. Imported necessary API augmentations.	+3/-0
db.ts `Implement database functions for entity creation and saving` indexers/mainnet/files/src/mappings/db.ts Implemented functions to create and save Cid, Chunk, Metadata, Folder, and File. Established relationships between entities.	+122/-0
mappingHandlers.ts `Add handler for processing blockchain extrinsics` indexers/mainnet/files/src/mappings/mappingHandlers.ts Added handler for processing blockchain extrinsics. Implemented logic to decode and save node data.	+102/-0
types.ts `Define types for extrinsic data structures` indexers/mainnet/files/src/mappings/types.ts Defined types for extrinsic primitives and human-readable formats.	+9/-0
project.ts `Add project configuration for Taurus Testnet files indexer` indexers/taurus/files/project.ts Added project configuration for Autonomys Taurus Testnet files indexer. Defined network endpoints and data sources. Configured handlers for specific blockchain events.	+95/-0
index.ts `Export handler functions and import API augmentations` indexers/taurus/files/src/index.ts Exported all handler functions. Imported necessary API augmentations.	+3/-0
db.ts `Implement database functions for entity creation and saving` indexers/taurus/files/src/mappings/db.ts Implemented functions to create and save Cid, Chunk, Metadata, Folder, and File. Established relationships between entities.	+122/-0
mappingHandlers.ts `Add handler for processing blockchain extrinsics` indexers/taurus/files/src/mappings/mappingHandlers.ts Added handler for processing blockchain extrinsics. Implemented logic to decode and save node data.	+102/-0
types.ts `Define types for extrinsic data structures` indexers/taurus/files/src/mappings/types.ts Defined types for extrinsic primitives and human-readable formats.	+9/-0
schema.graphql `Define GraphQL schema for file-related entities` indexers/mainnet/files/schema.graphql Defined GraphQL schema for Cid, Chunk, Metadata, Folder, and File entities.	+56/-0
schema.graphql `Define GraphQL schema for file-related entities` indexers/taurus/files/schema.graphql Defined GraphQL schema for Cid, Chunk, Metadata, Folder, and File entities.	+56/-0

Configuration changes

8 files

docker-compose.yml `Add Docker service configuration for files indexer` docker-compose.yml Added service configuration for files_subquery_node. Defined environment variables and health checks.	+35/-0
files_chunks.yaml `Define schema and permissions for chunks table` indexers/db/metadata/databases/default/tables/files_chunks.yaml Defined table schema for chunks in files database. Set select permissions for user role.	+28/-0
files_cids.yaml `Define schema and permissions for cids table` indexers/db/metadata/databases/default/tables/files_cids.yaml Defined table schema for cids in files database. Set select permissions for user role.	+30/-0
tables.yaml `Update metadata configuration to include files tables` indexers/db/metadata/databases/default/tables/tables.yaml Included new files-related tables in the metadata configuration.	+8/-0
package.json `Add package.json with project metadata` indexers/mainnet/files/package.json Added package.json with project metadata and dependencies.	+57/-0
tsconfig.json `Add TypeScript configuration file` indexers/mainnet/files/tsconfig.json Added TypeScript configuration file.	+20/-0
package.json `Add package.json with project metadata` indexers/taurus/files/package.json Added package.json with project metadata and dependencies.	+57/-0
tsconfig.json `Add TypeScript configuration file` indexers/taurus/files/tsconfig.json Added TypeScript configuration file.	+20/-0

Documentation

4 files

LICENSE `Add MIT License file` indexers/mainnet/files/LICENSE Added MIT License file for the project.	+21/-0
README.md `Add README with project overview and instructions` indexers/mainnet/files/README.md Added README with project overview and instructions.	+40/-0
LICENSE `Add MIT License file` indexers/taurus/files/LICENSE Added MIT License file for the project.	+21/-0
README.md `Add README with project overview and instructions` indexers/taurus/files/README.md Added README with project overview and instructions.	+40/-0

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

socket-security · 2024-11-20T19:25:08Z

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@subql/[email protected]	Transitive: environment, eval, filesystem, network, shell, unsafe	`+386`	51.3 MB	onfinality-admin
npm/@subql/[email protected]	Transitive: environment, filesystem, network, shell	`+92`	4.18 MB	onfinality-admin
npm/@subql/[email protected]	Transitive: environment, eval, filesystem, network, shell, unsafe	`+313`	47 MB	onfinality-admin
npm/@subql/[email protected]	None	`0`	258 kB	onfinality-admin
npm/[email protected]	None	`0`	776 kB	typestack-release-bot
npm/[email protected]	None	`+3`	14.3 MB	typestack-release-bot
npm/[email protected]	environment	`0`	1.36 MB	benjie
npm/[email protected]	filesystem, network, unsafe	`+2`	2.54 MB	simenb

View full report↗︎

socket-security · 2024-11-20T19:25:10Z

🚨 Potential security issues detected. Learn more about Socket for GitHub ↗︎

To accept the risk, merge this PR and you will not be notified again.

Alert	Package	Note	Source	CI
Critical CVE	npm/[email protected]	CVE: GHSA-cchq-frgv-rjh5 vm2 Sandbox Escape vulnerability (CRITICAL) Affected versions: <= 3.9.19 Patched version: No patched versions	`indexers/taurus/consensus/package.json`	⚠︎
Critical CVE	npm/[email protected]	CVE: GHSA-g644-9gfx-q4q4 vm2 Sandbox Escape vulnerability (CRITICAL) Affected versions: <= 3.9.19 Patched version: No patched versions	`indexers/taurus/consensus/package.json`	⚠︎
Protestware or potentially unwanted behavior	npm/[email protected]	Note: The script attempts to run a local post-install script, which could potentially contain malicious code. The error handling suggests that it is designed to fail silently, which is a common tactic in malicious scripts.	`indexers/taurus/consensus/package.json`	⚠︎
Protestware or potentially unwanted behavior	npm/[email protected]	Note: This package prints a protestware console message on install regarding Ukraine for users with Russian language locale	`indexers/taurus/consensus/package.json`	⚠︎

View full report↗︎

Next steps

What is a critical CVE?

Contains a Critical Common Vulnerability and Exposure (CVE).

Remove or replace dependencies that include known critical CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies.

What is protestware?

This package is a joke, parody, or includes undocumented or hidden behavior unrelated to its primary function.

Consider that consuming this package may come along with functionality unrelated to its primary purpose.

Take a deeper look at the dependency

Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev.

Remove the package

If you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency.

Mark a package as acceptable risk

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of ecosystem/package-name@version specifiers. e.g. @SocketSecurity ignore npm/[email protected] or ignore all packages with @SocketSecurity ignore-all

@SocketSecurity ignore npm/[email protected]
@SocketSecurity ignore npm/[email protected]

github-actions · 2024-11-20T19:25:16Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Error Handling The error handling in the `handleCall` function could be improved by providing more specific error messages related to the context of the failure, especially in the decoding process. Data Validation There is a lack of validation for the `cid` and `links` data received from the blockchain before processing. This could lead to potential issues if invalid data is received. Possible Performance Issue The functions `createAndSaveMetadata`, `createAndSaveFolder`, and `createAndSaveFile` use `Promise.all` for saving multiple relations which might lead to performance bottlenecks with large arrays of links.

github-actions · 2024-11-20T19:25:33Z

PR Code Suggestions ✨

No code suggestions found for the PR.

…dd-files-indexers

netlify · 2024-11-21T16:14:57Z

✅ Deploy Preview for dev-astral canceled.

Name	Link
🔨 Latest commit	`4523508`
🔍 Latest deploy log	https://app.netlify.com/sites/dev-astral/deploys/673f5c700a2a120008c04e44

clostao · 2024-11-21T22:26:38Z

I think we should avoid indexing chunks in the long-term. Besides that, looks so nice!

marcusDesk added 8 commits November 20, 2024 13:12

add base for files indexers

b3b81e5

add files indexers schema, db helper and types

3ea45ff

add files mapping

7ccbe3f

add tabes relations between cids

b48666d

change relation field name

501462a

add table metadata in hasura

38e3698

add tables relations

42afcae

add tables permissions

cb4e854

marc-aurele-besner requested review from jfrank-summit and clostao November 20, 2024 19:23

github-actions bot added the Review effort [1-5]: 4 label Nov 20, 2024

marcusDesk added 3 commits November 20, 2024 14:25

Merge branch 'feat/consensus-indexer-convert-data-to-cid' into feat/a…

41c23b9

…dd-files-indexers

Merge branch 'feat/consensus-indexer-convert-data-to-cid' into feat/a…

f08b043

…dd-files-indexers

add support for node sent as Bytes, thanks @clostao

e21f908

Base automatically changed from feat/consensus-indexer-convert-data-to-cid to main November 21, 2024 16:14

Merge branch 'main' into feat/add-files-indexers

4523508

clostao approved these changes Nov 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add files indexers #977

Add files indexers #977

marc-aurele-besner commented Nov 20, 2024 •

edited by github-actions bot

Loading

socket-security bot commented Nov 20, 2024 •

edited

Loading

socket-security bot commented Nov 20, 2024 •

edited

Loading

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

netlify bot commented Nov 21, 2024 •

edited

Loading

clostao commented Nov 21, 2024 •

edited

Loading

Add files indexers #977

Are you sure you want to change the base?

Add files indexers #977

Conversation

marc-aurele-besner commented Nov 20, 2024 • edited by github-actions bot Loading

User description