Contract source filter

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,21 @@
+name: Feature request
+description: Request new features or improvements.
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for completing this feature request form! The more info you provide, the faster we can help you. Before you proceed, please confirm that there isn't [already an issue](https://github.com/ponder-sh/ponder/issues) for this feature request.
+
+  - type: textarea
+    attributes:
+      label: Problem / use case
+      description: A description of the problem that the new feature would solve. Please include a real-world example or use case.
+    validations:
+      required: true
+
+  - type: textarea
+    attributes:
+      label: Proposed solution
+      description: A description of the proposed solution.
+    validations:
+      required: false

docs/pages/docs/contracts-and-networks.mdx

@@ -80,7 +80,7 @@ export default createConfig({
 
 ### Multiple ABIs
 
-It's occasionally useful to provide multiple ABIs for one contract, like when defining a proxy/upgradable contract that has gone through multiple implementation contracts. The [`mergeAbis`](/docs/utilities/abi) utility function safely removes duplicate ABI items and maintains strict types.
+It's occasionally useful to provide multiple ABIs for one contract, like when defining a proxy/upgradable contract that has gone through multiple implementation contracts. The [`mergeAbis`](/docs/utilities/merge-abis) utility function safely removes duplicate ABI items and maintains strict types.
 
 ```ts filename="ponder.config.ts" {1,14}
 import { createConfig, mergeAbis } from "ponder";
@@ -466,7 +466,7 @@ event ChildCreated(ChildContract child);
 
 ### Proxy & upgradable contracts
 
-To index a proxy/upgradable contract, use the proxy contract address in the `address` field. Then, be sure to include the ABIs of all implementation contracts that the proxy has ever had. The implementation ABIs are required to properly identify and decode all historical event logs. To add multiple ABIs safely, use the [`mergeAbis`](/docs/utilities/abi) utility function.
+To index a proxy/upgradable contract, use the proxy contract address in the `address` field. Then, be sure to include the ABIs of all implementation contracts that the proxy has ever had. The implementation ABIs are required to properly identify and decode all historical event logs. To add multiple ABIs safely, use the [`mergeAbis`](/docs/utilities/merge-abis) utility function.
 
 <Callout>
 Tip: On Etherscan, there is a link to the current implementation contract on the **Contract → Read as Proxy** tab. You can copy all the implementation ABIs as text and paste them into `.ts` files.
@@ -477,49 +477,10 @@ Tip: On Etherscan, there is a link to the current implementation contract on the
 
 ## Event filter
 
-The `filter` option filters for events by signature and indexed argument values. This makes it possible to index events emitted by _any contract on the network_ that match the specified signature and arguments.
-
-### By event name or signature
-
-The `filter.event` option accepts an event name (or list of event names) present in the provided ABI.
-
-```ts filename="ponder.config.ts" {14}
-import { createConfig } from "ponder";
-import { http } from "viem";
-
-import { ERC20Abi } from "./abis/ERC20";
-
-export default createConfig({
-  networks: {
-    mainnet: { chainId: 1, transport: http(process.env.PONDER_RPC_URL_1) },
-  },
-  contracts: {
-    ERC20: {
-      abi: ERC20Abi,
-      network: "mainnet",
-      filter: { event: "Transfer" },
-      //        ^? "Transfer" | "Approval" | ("Transfer" | "Approval")[]
-      startBlock: 18500000,
-      endBlock: 18505000,
-    },
-  },
-});
-```
-
-The indexing functions you write will run for all events matching the filter, regardless of which contract emitted them.
-
-```ts filename="src/index.ts"
-import { ponder } from "ponder:registry";
-
-ponder.on("ERC20:Transfer", async ({ event }) => {
-  // This is the address of the contract that emitted the event.
-  // With this config, it could be any ERC20 contract on mainnet.
-  event.log.address;
-  //        ^? string
-});
-```
-
-### By indexed argument value
+<Callout type="info">
+  Ponder will automatically filter out event signatures that don't have a
+  registered indexing function.
+</Callout>
 
 You can use the `filter.event` and `filter.args` options together to filter for events that have specific indexed argument values. Each indexed argument field accepts a single value or a list of values to match.
 
@@ -569,6 +530,10 @@ ponder.on("ERC20:Transfer", async ({ event }) => {
 });
 ```
 
+<Callout type="info">
+  Multiple filters can be specified by passing an array of filter objects.
+</Callout>
+
 ## Call traces
 
 The `includeCallTraces{:ts}` option specifies whether to enable call trace indexing for a contract. By default, call traces are **disabled**.

docs/pages/docs/migration-guide.mdx

@@ -87,6 +87,10 @@ export default app;
 
 A new package for querying Ponder apps. [Read more](/docs/query/client).
 
+### Event filters
+
+The `contracts.filter` property now supports multiple filters, and requires argument values. [Read more](/docs/contracts-and-networks#event-filter).
+
 ## 0.8
 
 <Callout type="warning">

packages/core/CHANGELOG.md

@@ -1,5 +1,25 @@
 # ponder
 
+## 0.8.10
+
+### Patch Changes
+
+- [#1370](https://github.com/ponder-sh/ponder/pull/1370) [`61b0b04c3306929bf2ff1ef781be874f561d8e11`](https://github.com/ponder-sh/ponder/commit/61b0b04c3306929bf2ff1ef781be874f561d8e11) Thanks [@khaidarkairbek](https://github.com/khaidarkairbek)! - Fixed a bug causing future end blocks to error.
+
+## 0.8.9
+
+### Patch Changes
+
+- [#1381](https://github.com/ponder-sh/ponder/pull/1381) [`dae8801ea3ddf732d8284ff84bc7dc21ada22f0e`](https://github.com/ponder-sh/ponder/commit/dae8801ea3ddf732d8284ff84bc7dc21ada22f0e) Thanks [@typedarray](https://github.com/typedarray)! - Fixed a bug where data inserted using raw SQL near the beginning of historical indexing was not found by subsequent `find`, `update`, or `delete` operations using the store/in-memory API.
+
+- [#1375](https://github.com/ponder-sh/ponder/pull/1375) [`1687033a74fb8e7a7d843b4fe1f7f1cd4cf866a9`](https://github.com/ponder-sh/ponder/commit/1687033a74fb8e7a7d843b4fe1f7f1cd4cf866a9) Thanks [@typedarray](https://github.com/typedarray)! - Improved logs for Postgres pool errors.
+
+## 0.8.8
+
+### Patch Changes
+
+- [#1368](https://github.com/ponder-sh/ponder/pull/1368) [`492d7e7744dbddede2e72b0649cd7dffb96173cd`](https://github.com/ponder-sh/ponder/commit/492d7e7744dbddede2e72b0649cd7dffb96173cd) Thanks [@typedarray](https://github.com/typedarray)! - Improved debug-level logs for historical indexing observability.
+
 ## 0.8.7
 
 ### Patch Changes

packages/core/src/_test/e2e/erc20/ponder.config.ts

@@ -1,5 +1,5 @@
 import { http } from "viem";
-import { createConfig } from "../../../config/config.js";
+import { createConfig } from "../../../config/index.js";
 import { erc20ABI } from "../../generated.js";
 
 const poolId = Number(process.env.VITEST_POOL_ID ?? 1);

packages/core/src/_test/e2e/factory/ponder.config.ts

@@ -1,6 +1,6 @@
 import { factory } from "@/config/address.js";
 import { http, getAbiItem } from "viem";
-import { createConfig } from "../../../config/config.js";
+import { createConfig } from "../../../config/index.js";
 import { factoryABI, pairABI } from "../../generated.js";
 
 const poolId = Number(process.env.VITEST_POOL_ID ?? 1);

packages/core/src/_test/setup.ts

@@ -1,10 +1,10 @@
+import type { DatabaseConfig } from "@/build/index.js";
 import { buildSchema } from "@/build/schema.js";
 import type { Common } from "@/common/common.js";
 import { createLogger } from "@/common/logger.js";
 import { MetricsService } from "@/common/metrics.js";
 import { buildOptions } from "@/common/options.js";
 import { createTelemetry } from "@/common/telemetry.js";
-import type { DatabaseConfig } from "@/config/database.js";
 import { type Database, createDatabase } from "@/database/index.js";
 import type { Schema } from "@/drizzle/index.js";
 import type { IndexingStore } from "@/indexing-store/index.js";
@@ -72,6 +72,14 @@ export async function setupIsolatedDatabase(context: TestContext) {
 
     const client = new pg.Client({ connectionString });
     await client.connect();
+    await client.query(
+      `
+      SELECT pg_terminate_backend(pg_stat_activity.pid)
+      FROM pg_stat_activity
+      WHERE pg_stat_activity.datname = $1 AND pid <> pg_backend_pid()
+      `,
+      [databaseName],
+    );
     await client.query(`DROP DATABASE IF EXISTS "${databaseName}"`);
     await client.query(`CREATE DATABASE "${databaseName}"`);
     await client.end();

packages/core/src/_test/utils.ts

@@ -1,7 +1,7 @@
 import { type AddressInfo, createServer } from "node:net";
+import type { Network } from "@/build/index.js";
 import { factory } from "@/config/address.js";
-import { createConfig } from "@/config/config.js";
-import type { Network } from "@/config/networks.js";
+import { createConfig } from "@/config/index.js";
 import type { Status } from "@/sync/index.js";
 import type { Address, Chain } from "viem";
 import { http, createPublicClient, createTestClient, getAbiItem } from "viem";

packages/core/src/bin/utils/run.ts

@@ -1,6 +1,7 @@
 import type { IndexingBuild, SchemaBuild } from "@/build/index.js";
 import { runCodegen } from "@/common/codegen.js";
 import type { Common } from "@/common/common.js";
+import { getAppProgress } from "@/common/metrics.js";
 import type { Database } from "@/database/index.js";
 import { createHistoricalIndexingStore } from "@/indexing-store/historical.js";
 import { getMetadataStore } from "@/indexing-store/metadata.js";
@@ -15,6 +16,7 @@ import {
   encodeCheckpoint,
   zeroCheckpoint,
 } from "@/utils/checkpoint.js";
+import { formatEta, formatPercentage } from "@/utils/format.js";
 import { never } from "@/utils/never.js";
 import { createQueue } from "@ponder/common";
 
@@ -161,10 +163,25 @@ export async function run({
     for await (const { events, checkpoint } of sync.getEvents()) {
       end = checkpoint;
 
-      const result = await handleEvents(
-        decodeEvents(common, indexingBuild.sources, events),
-        checkpoint,
-      );
+      const decodedEvents = decodeEvents(common, indexingBuild.sources, events);
+      const result = await handleEvents(decodedEvents, checkpoint);
+
+      // underlying metrics collection is actually synchronous
+      // https://github.com/siimon/prom-client/blob/master/lib/histogram.js#L102-L125
+      const { eta, progress } = await getAppProgress(common.metrics);
+      if (events.length > 0) {
+        if (eta === undefined || progress === undefined) {
+          common.logger.info({
+            service: "app",
+            msg: `Indexed ${events.length} events`,
+          });
+        } else {
+          common.logger.info({
+            service: "app",
+            msg: `Indexed ${events.length} events with ${formatPercentage(progress)} complete and ${formatEta(eta)} remaining`,
+          });
+        }
+      }
 
       // Persist the indexing store to the db if it is too full. The `finalized`
       // checkpoint is used as a mutex. Any rows in the reorg table that may
@@ -176,6 +193,18 @@ export async function run({
           lastFlush + 5_000 < Date.now() &&
           events.length > 0)
       ) {
+        if (historicalIndexingStore.isCacheFull()) {
+          common.logger.debug({
+            service: "indexing",
+            msg: `Indexing cache has exceeded ${common.options.indexingCacheMaxBytes} MB limit, starting flush`,
+          });
+        } else {
+          common.logger.debug({
+            service: "indexing",
+            msg: "Dev server periodic flush triggered, starting flush",
+          });
+        }
+
         await database.finalize({
           checkpoint: encodeCheckpoint(zeroCheckpoint),
         });
@@ -187,6 +216,11 @@ export async function run({
           checkpoint: events[events.length - 1]!.checkpoint,
         });
         lastFlush = Date.now();
+
+        common.logger.debug({
+          service: "indexing",
+          msg: "Completed flush",
+        });
       }
 
       await metadataStore.setStatus(sync.getStatus());
@@ -205,6 +239,11 @@ export async function run({
     // have been written because of raw sql access are deleted. Also must truncate
     // the reorg tables that may have been written because of raw sql access.
 
+    common.logger.debug({
+      service: "indexing",
+      msg: "Completed all historical events, starting final flush",
+    });
+
     await database.finalize({ checkpoint: encodeCheckpoint(zeroCheckpoint) });
     await historicalIndexingStore.flush();
     await database.complete({ checkpoint: encodeCheckpoint(zeroCheckpoint) });