feat(etherscan): add caching #1108

onbjerg · 2022-04-04T15:18:20Z

Supersedes #935

ethers-etherscan/src/lib.rs

onbjerg · 2022-04-04T16:14:46Z

This should work now / it works when testing locally on mds1/drai

onbjerg · 2022-04-04T16:17:05Z

Actually, I should probably add an actual error for rate limits so we can catch it in Foundry for retries

onbjerg · 2022-04-04T17:15:29Z

This fails when the file doesn't exist, going back in draft

ethers-etherscan/src/contract.rs

gakonst · 2022-04-05T19:25:48Z

need to change how the responses are parsed
then add ttl

then should be good to merge

onbjerg · 2022-04-05T22:58:51Z

ethers-etherscan/src/contract.rs

        let query = self.create_query("contract", "getabi", HashMap::from([("address", address)]));
        let resp: Response<String> = self.get_json(&query).await?;
+        if resp.result.starts_with("Max rate limit reached") {


we have to dupe the check here cus it deserializes message as a string.. really annoying but no way around it really

if etherscan would give me access i would make their api better

onbjerg · 2022-04-05T23:21:23Z

ethers-etherscan/src/lib.rs

+                expiry: SystemTime::now()
+                    .checked_add(self.ttl)
+                    .expect("cache ttl overflowed")
+                    .duration_since(UNIX_EPOCH)
+                    .expect("system time is before unix epoch")
+                    .as_secs(),


Is there a simpler way to do this?

onbjerg · 2022-04-05T23:21:29Z

ethers-etherscan/src/lib.rs

+        let reader = std::io::BufReader::new(std::fs::File::open(path)?);
+        if let Ok(inner) = serde_json::from_reader::<_, CacheEnvelope<T>>(reader) {
+            // If this does not return None then we have passed the expiry
+            if SystemTime::now().checked_sub(Duration::from_secs(inner.expiry)).is_some() {


Same as above

onbjerg · 2022-04-05T23:22:00Z

Oh wait, I re-requested review before I cached the "not verified" objects

onbjerg · 2022-04-05T23:39:22Z

ethers-etherscan/src/contract.rs

+        if let Some(ref cache) = self.cache {
+            // If this is None, then we have a cache miss
+            if let Some(src) = cache.get_source(address) {
+                // If this is None, then the contract is not verified
+                return match src {
+                    Some(src) => Ok(src),
+                    None => Err(EtherscanError::ContractCodeNotVerified(address)),
+                }
+            }
+        }


This is really ugly, but the alternative was to cache Result<T, EtherscanError>. Not sure. Any suggestions?

you can prob do something like if let Some(Some(src)) = cache.get_source(address) but nbd. feel free to do in follow up

onbjerg · 2022-04-05T23:40:30Z

ethers-etherscan/src/lib.rs

+        }
+    }
+
+    fn get<T: DeserializeOwned>(&self, prefix: &str, address: Address) -> Option<T> {


Switched this to an option since we don't really care about the errors:

If we can't deserialize it, it's malformed and we need to rewrite it

If we can't open the file, the directory doesn't exist (we probably care here, should we add tracing?)

gakonst · 2022-04-06T01:58:22Z

ethers-etherscan/src/contract.rs

+        if let Some(ref cache) = self.cache {
+            // If this is None, then we have a cache miss
+            if let Some(src) = cache.get_source(address) {
+                // If this is None, then the contract is not verified
+                return match src {
+                    Some(src) => Ok(src),
+                    None => Err(EtherscanError::ContractCodeNotVerified(address)),
+                }
+            }
+        }


you can prob do something like if let Some(Some(src)) = cache.get_source(address) but nbd. feel free to do in follow up

gakonst · 2022-04-06T02:01:41Z

Merging - let's hope it works well :)

gakonst/ethers-rs#1108

* feat: etherscan identifier * chore: bump ethers for the caching PR gakonst/ethers-rs#1108 * feat: add cache ttl to etherscan identifier * chore: clippy * chore: re-add ethers patch section * build: bump ethers * test: fix tests * fix: trace macros * bump color eyre and lock tracing-subscriber (#1220) * bump color eyre * lock tracing-subscriber 0.3.9 * feat: pull etherscan api key and eth rpc from env * refactor: readability in `trace.addresses` * refactor: add `AddressIdentity` type * refactor: add a rate limited etherscan stream * test: don't set `ETH_RPC_URL` as its not used * refactor: smol nit Co-authored-by: Georgios Konstantopoulos <[email protected]> Co-authored-by: Matthias Seitz <[email protected]>

gakonst and others added 3 commits April 4, 2022 16:02

feat(etherscan): add caching

c2ac65d

feat: write to cache if not exists

549d913

fix: flush BufWriter

8d11bac

onbjerg changed the title ~~Feat/etherscan cache~~ feat(etherscan): add caching Apr 4, 2022

This was referenced Apr 4, 2022

feat: identify addresses from etherscan when forking foundry-rs/foundry#1190

Merged

feat(etherscan): add caching #935

Closed

gakonst reviewed Apr 4, 2022

View reviewed changes

ethers-etherscan/src/lib.rs Outdated Show resolved Hide resolved

onbjerg added 3 commits April 4, 2022 18:10

fix: fix serialization

b1c5791

fix: read cache without truncating the file

05d3c28

chore: remove comments

bfc28bf

onbjerg marked this pull request as ready for review April 4, 2022 16:14

onbjerg added 2 commits April 4, 2022 18:26

feat: rate limit errors

1cc734f

test: fix tests

a4179e0

onbjerg marked this pull request as draft April 4, 2022 17:15

onbjerg added 2 commits April 4, 2022 19:23

test: fix tests

b44ab84

fix: don't fail if cache doesn't exist

0577072

onbjerg marked this pull request as ready for review April 4, 2022 18:13

gakonst approved these changes Apr 4, 2022

View reviewed changes

ethers-etherscan/src/contract.rs Outdated Show resolved Hide resolved

fix: catch all rate limits

a625248

onbjerg commented Apr 5, 2022

View reviewed changes

feat: add ttl

d5bf785

onbjerg commented Apr 5, 2022

View reviewed changes

onbjerg requested a review from gakonst April 5, 2022 23:21

feat: also cache when contracts are not verified

18865c0

onbjerg commented Apr 5, 2022

View reviewed changes

gakonst approved these changes Apr 6, 2022

View reviewed changes

chore: clippy

0a30a7e

gakonst merged commit c436d19 into gakonst:master Apr 6, 2022

gakonst added a commit to foundry-rs/foundry that referenced this pull request Apr 6, 2022

chore: bump ethers for the caching PR

c524d77

gakonst/ethers-rs#1108

onbjerg pushed a commit to foundry-rs/foundry that referenced this pull request Apr 6, 2022

chore: bump ethers for the caching PR

8ccd584

gakonst/ethers-rs#1108

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(etherscan): add caching #1108

feat(etherscan): add caching #1108

onbjerg commented Apr 4, 2022

onbjerg commented Apr 4, 2022 •

edited

Loading

onbjerg commented Apr 4, 2022

onbjerg commented Apr 4, 2022

gakonst commented Apr 5, 2022

onbjerg Apr 5, 2022 •

edited

Loading

onbjerg Apr 5, 2022

onbjerg Apr 5, 2022

onbjerg commented Apr 5, 2022

onbjerg Apr 5, 2022

gakonst Apr 6, 2022

onbjerg Apr 5, 2022

gakonst Apr 6, 2022

gakonst commented Apr 6, 2022

feat(etherscan): add caching #1108

feat(etherscan): add caching #1108

Conversation

onbjerg commented Apr 4, 2022

onbjerg commented Apr 4, 2022 • edited Loading

onbjerg commented Apr 4, 2022

onbjerg commented Apr 4, 2022

gakonst commented Apr 5, 2022

onbjerg Apr 5, 2022 • edited Loading

Choose a reason for hiding this comment

onbjerg Apr 5, 2022

Choose a reason for hiding this comment

onbjerg Apr 5, 2022

Choose a reason for hiding this comment

onbjerg commented Apr 5, 2022

onbjerg Apr 5, 2022

Choose a reason for hiding this comment

gakonst Apr 6, 2022

Choose a reason for hiding this comment

onbjerg Apr 5, 2022

Choose a reason for hiding this comment

gakonst Apr 6, 2022

Choose a reason for hiding this comment

gakonst commented Apr 6, 2022

onbjerg commented Apr 4, 2022 •

edited

Loading

onbjerg Apr 5, 2022 •

edited

Loading