Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eth_subscriptionLogs with topics filter problem #4030

Closed
d-polikhranidi opened this issue Apr 29, 2022 · 20 comments
Closed

eth_subscriptionLogs with topics filter problem #4030

d-polikhranidi opened this issue Apr 29, 2022 · 20 comments
Assignees
Labels
gateway Issue assigned to gateway team

Comments

@d-polikhranidi
Copy link

System information

Erigon version: erigon version 2022.04.3-alpha-87421417

OS & Version: Linux (Alpine 3.15.4 in Docker)

Commit hash : 8742141

Expected behaviour

When I'm subscribe to erigon node with topics filter - I must see transactions that corresponds to topics.

Actual behaviour

I see no any transactions at all, but they are (according to ETH/BSC explorer). If I don't use any filters (no topics) - I see all transactions, but I need only transactions with topics I'm interested in.

Steps to reproduce the behaviour

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.WebsocketProvider("wss://NODE_IP:NODE_PORT"));

let options = {
    topics: [
        null,
        '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2',
        null
    ]
};

let subscription = web3.eth.subscribe('logs', options, (err, event) => {
    if (err) throw err;
});

subscription.on('data', event => console.log(event))
subscription.on('changed', changed => console.log(changed))
subscription.on('error', err => { throw err })
subscription.on('connected', nr => console.log(nr))

The minimal code above works well with gEth nodes and getblock.io node.

So, what I miss?

@AskAlexSharov
Copy link
Collaborator

I think subscription to logs not supported: https://github.com/ledgerwatch/erigon/tree/devel/cmd/rpcdaemon#rpc-implementation-status
eth_getLogs - works and fast

@d-polikhranidi
Copy link
Author

I think subscription to logs not supported: https://github.com/ledgerwatch/erigon/tree/devel/cmd/rpcdaemon#rpc-implementation-status
eth_getLogs - works and fast

But subscription to logs without topics - works great

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.WebsocketProvider("wss://NODE_IP:NODE_PORT"));

let subscription = web3.eth.subscribe('logs', {}, (err, event) => {
    if (err) throw err;
});

subscription.on('data', event => console.log(event))
subscription.on('changed', changed => console.log(changed))
subscription.on('error', err => { throw err })
subscription.on('connected', nr => console.log(nr))

This works. But when I add topics to options it's shows nothing((

@AskAlexSharov
Copy link
Collaborator

Don’t know, maybe bug. Need somebody to check.

@primalcs primalcs self-assigned this Jun 7, 2022
@primalcs
Copy link
Contributor

Could you please try
let options = { topics: [ '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2' ] };

instead of
let options = { topics: [ null, '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2', null ] };
?

@dan-nathan
Copy link

dan-nathan commented Jun 29, 2022

I'm running similar issues with my erigon node running on Ubuntu 22.04 on commit aa79853. If I only have one log subscription it works fine, but running multiple at once with different topic/address queries causes unexpected behaviour. I'm testing using go-ethereum with this simple program, which subscribes to logs with the option to filter by address 0xdac17f958d2ee523a2206206994597c13d831ec7 (USDT) using the flag -usdt, and the option to filter by topics ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"] (ERC20 Transfer event hash) using the flag -erc20

package main

import (
	"context"
	"flag"
	"fmt"
	"os"

	ethereum "github.com/ethereum/go-ethereum"
	ethCommon "github.com/ethereum/go-ethereum/common"
	"github.com/ethereum/go-ethereum/core/types"
	ethClient "github.com/ethereum/go-ethereum/ethclient"
)

const (
	usdtAddress         = "0xdac17f958d2ee523a2206206994597c13d831ec7"
	erc20TransferTopic0 = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
)

func main() {
	websocketAddress := os.Getenv("ERIGON_WEBSOCKET_URL")
	if websocketAddress == "" {
		fmt.Println("Please set ERIGON_WEBSOCKET_URL environment variable")
		return
	}
	client, err := ethClient.Dial(websocketAddress)
	if err != nil {
		fmt.Printf("Failed to connect to websocket: %v\n", err)
		return
	}

	erc20, uniswap := parseCliArgs()

	var filter ethereum.FilterQuery
	if erc20 {
		filter.Topics = [][]ethCommon.Hash{{ethCommon.HexToHash(erc20TransferTopic0)}}
	}
	if uniswap {
		filter.Addresses = []ethCommon.Address{ethCommon.HexToAddress(usdtAddress)}
	}

	receivedLogs := make(chan types.Log)
	subscription, err := client.SubscribeFilterLogs(context.Background(), filter, receivedLogs)
	if err != nil {
		fmt.Printf("Unable to create subscription: %v\n", err)
		return
	}
	defer subscription.Unsubscribe()
	fmt.Println("Subscription successful")
	errChan := subscription.Err()

	for {
		select {
		case log := <-receivedLogs:
			fmt.Printf("Received log %d of block %d from address %s and topic0 %s\n",
				log.Index,
				log.BlockNumber,
				log.Address.String(),
				log.Topics[0].String(),
			)
		case err := <-errChan:
			fmt.Printf("Subscription error encountered: %v\n", err)
			return
		}
	}
}

// Parses command line flags, and returns whether or not to filter by the ERC20
// transfer topic 0 hash, and whether or not to filter by USDT address
func parseCliArgs() (bool, bool) {
	erc20TransferFlag := flag.Bool("erc20", false, "Whether to filter logs by topic 0 hash "+erc20TransferTopic0)
	usdtFlag := flag.Bool("usdt", false, "Whether to filter logs by address "+usdtAddress)
	flag.Parse()
	return *erc20TransferFlag, *usdtFlag
}

When I run this program multiple time concurrently, I get the following behaviour:

  • Running one with no flags and one with the -erc20 flag, neither receive any logs
  • Running one with no flags and one with the -usdt flag, neither receive any logs
  • Running one with no flags, one with the -erc20 flag, and one with both flags, the former two erroneously only receive logs from the usdt address
  • Running one with no flags, one with the -usdt flag, and one with both flags, the former two erroneously only receive logs with the erc20 transfer topic 0 hash
  • Running one with no flags, one with the -usdt flag, one with the -erc20 flag, and one with both flags, all four only receive logs with the erc20 transfer topic 0 hash from the usdt address
  • Running one with the -usdt flag, one with the -erc20 flag, and one with both flags, all behave as expected
  • Running one with no flags, and one with both flags, both behave as expected

@samsondav
Copy link

samsondav commented Aug 8, 2022

This is a blocking bug for us to use Chainlink with Erigon. We require topic filtering on log streams.

@revitteth revitteth added the gateway Issue assigned to gateway team label Aug 8, 2022
@revitteth revitteth self-assigned this Aug 8, 2022
@hexoscott hexoscott self-assigned this Sep 12, 2022
@hexoscott hexoscott added the status:in-progress Actively being worked on (picked up by developer) label Sep 14, 2022
@revitteth
Copy link
Collaborator

Closing issue as fix is merged.

@rcastellaj
Copy link

I'm not sure this is fixed, as we continue to see this issue. I cannot use eth_getLogs with any filters (address, topics). I can only get the full bloom from a log range. It's not the end of the world as I can filter in the client, but it adds a lot of latency and bandwidth requirements which in turn make it lose its advantage against geth.

@hexoscott
Copy link
Collaborator

Hi @rcastellaj, could you send some examples over of what you're running and I can take another look, thanks.

@drcliche
Copy link

@hexoscott This looks related to a similar issue with examples of usage and intermittent/different responses being generated: #6598

@hexoscott hexoscott reopened this Feb 1, 2023
@danieloadebimpe
Copy link

danieloadebimpe commented Feb 6, 2023

I don't believe this was fixed. As @samsondav said, this is required for chainlink.

I am running erigon v2.36.0 and I get the following error when trying to subscribe to a topic on goerli:

root@ip<erigonIP> bin]# curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_subscribe","params":["logs", {"address": "0x005B0d11379c4c04C0B726eE0BE55feb50b59f81", "topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"]}], "id":1}' <Node address and port>
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"notifications not supported"}}

My system info:
Kernel -> Linux version 4.14.243-185.433.amzn2.x86_64
Distribution -> (gcc version 7.3.1 20180712 (Red Hat 7.3.1-13) (GCC))

@hexoscott
Copy link
Collaborator

I'm taking a look into this now, could you send me the flags you're using @danieloadebimpe please

@danieloadebimpe
Copy link

@hexoscott

command:
      - erigon
      - --datadir=/home/erigon/.local/share/erigon
      - --authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex
      - --chain=goerli
      - --externalcl
      - --private.api.addr=0.0.0.0:9090
      - --http
      - --http.port=8546
      - --http.api=eth,erigon,engine,net,web3
      - --snapshots=false
      - --rpc.accessList=/home/erigon/rules.json
      - --http.addr=0.0.0.0
      - --http.vhosts=*
      - --ws
      - --metrics
      - --metrics.port=6060
      - --authrpc.addr=0.0.0.0
      - --authrpc.port=8551
      - --authrpc.vhosts=*
      ```

@hexoscott
Copy link
Collaborator

Just a quick check that eth_subscribe is in your rules.json?

@danieloadebimpe
Copy link

danieloadebimpe commented Feb 6, 2023

Yes, heres the file:

{ "allow": [ "web3_clientVersion", "web3_sha3", "net_listening", "net_peerCount", "net_version", "eth_blockNumber", "eth_chainID", "eth_chainId", "eth_protocolVersion", "eth_syncing", "eth_gasPrice",
    "eth_maxPriorityFeePerGas", "eth_feeHistory", "eth_getBlockByHash", "eth_getBlockByNumber", "eth_getBlockTransactionCountByHash", "eth_getBlockTransactionCountByNumber",
    "eth_getUncleByBlockHashAndIndex", "eth_getUncleByBlockNumberAndIndex", "eth_getUncleCountByBlockHash", "eth_getUncleCountByBlockNumber", "eth_getTransactionByHash", "eth_getRawTransactionByHash",
    "eth_getTransactionByBlockHashAndIndex", "eth_retRawTransactionByBlockHashAndIndex", "eth_getTransactionByBlockNumberAndIndex", "eth_retRawTransactionByBlockNumberAndIndex",
    "eth_getTransactionReceipt", "eth_getBlockReceipts", "eth_estimateGas", "eth_getBalance", "eth_getCode", "eth_getTransactionCount", "eth_getStorageAt", "eth_call", "eth_callBundle",
    "eth_createAccessList", "eth_newFilter", "eth_newBlockFilter", "eth_newPendingTransactionFilter", "eth_getFilterChanges", "eth_getFilterLogs", "eth_uninstallFilter",
    "eth_getLogs", "eth_getProof", "eth_sendRawTransaction", "eth_coinbase", "eth_hashrate", "eth_submitHashrate", "eth_getWork", "eth_submitWork", "**eth_subscribe**", "eth_unsubscribe",
    "debug_accountRange", "debug_accountAt", "debug_getModifiedAccountsByNumber", "debug_getModifiedAccountsByHash", "debug_storageRangeAt", "debug_traceBlockByHash", "debug_traceBlockByNumber",
    "debug_traceTransaction", "debug_traceCall", "trace_call", "trace_callMany", "trace_rawTransaction", "trace_replayBlockTransactions", "trace_replayTransaction", "trace_block", "trace_filter",
    "trace_get", "trace_transaction", "txpool_content", "txpool_status", "erigon_getHeaderByHash", "erigon_getHeaderByNumber", "erigon_getLogsByHash", "erigon_forks", "erigon_issuance",
    "erigon_GetBlockByTimestamp"
  ]
}

@hexoscott
Copy link
Collaborator

Ah I see the issue here, you need to open a websocket connection to erigon first then send your request via that. Subscriptions are only enabled via sockets.

You'll get back a subscription ID which you can then use to cancel the subscription in the future if you wish.

An example sent via postman:

Image

@ca42
Copy link

ca42 commented Feb 16, 2023

@hexoscott is this issue being actively worked on? we've been seeing this issue with eth_getLogs per #6598. is there any additional info i could provide which would be helpful?

@hexoscott
Copy link
Collaborator

Hi @ca42, it is, but I'm juggling it with other issues. I'll be taking another look into this today.

@hexoscott
Copy link
Collaborator

@ca42 - do you have any recent examples that are re-produceable? Could you let me know the chain as well.

@hexoscott
Copy link
Collaborator

I'm going to close this issue down, the problem mentioned above around eth_getLogs was initially an mdbx version issue from the looks of things which is now resolved, and a typo in a test file leading to inconsistent results. Chances are the mdbx fix will have resolved this issue?

If the issue still persists, please leave examples and information such as chain and anything else that would be helpful and I'll pick the thread back up.

@yperbasis yperbasis removed the status:in-progress Actively being worked on (picked up by developer) label Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gateway Issue assigned to gateway team
Projects
None yet
Development

No branches or pull requests