DOSsing S3 with GetSize calls even with bloom filter

[ianopolous]

Our S3 based ipfs instances have been flooding S3 with requests for GetSize requests. We're seeing a constant 5Mb/s of requests to S3 whilst we are not using it. It seemed to start around August 21st 2020. The only correlated event I can think of that may be significant is the filecoin space race launch. The log looks like:

2021-06-16T10:46:43.995Z	ERROR	engine	blockstore.GetSize(QmUCRuDoHafwt4CCkWy6gr45a6voofy5b4JktWstbW18NQ) error: ServiceUnavailable: Service Unavailable
	status code: 503, request id: tx000000000000000000000-0000000000-0000-default, host id: 
2021-06-16T10:46:44.025Z	ERROR	engine	blockstore.GetSize(Qmets7C6YhHgvm5BvWPT9dyXd1U6JLMUekhJwNm2P7acTd) error: ServiceUnavailable: Service Unavailable
	status code: 503, request id: tx000000000000000000000-0000000000-0000-default, host id: 
2021-06-16T10:46:44.056Z	ERROR	engine	blockstore.GetSize(QmcSB5Exmymh3P9stGFKvJyWP8XPp4L92FpXN8mAT8hkxT) error: ServiceUnavailable: Service Unavailable
	status code: 503, request id: tx000000000000000000000-0000000000-0000-default, host id:

We have the bloom filter configured with:

"BloomFilterSize": 268435456,

The S3 bucket in question has ~ 900k blocks in it. The S3 service rate limits us to 750 req/s/IP address.

My understanding is incoming requests will be served even before the bloom filter is built on startup. Is it possible that this is flooding S3 and preventing the bloom filter from being built at all, and thus the flood continues? Is there any other way to stop this?

[ianopolous]

It looks like this will fix it: ipfs/go-ipfs-blockstore#77

[aschmahmann]

Thanks for putting up ipfs/go-ipfs-blockstore#78. Have you tried running it, does it help?

[ianopolous]

No, I haven't tried building and running it in ipfs. We don't have a way to run a non release version on our infrastructure at the moment.

[ianopolous]

If it's incorporated into a pre-release version and I figure out how to build the S3plugin again with the latest then I can try that on our test infrastructure.

[aschmahmann]

Are you building the plugin in-tree or out-of-tree? If it's in-tree then doing a build from a branch of go-ipfs should be pretty straight forward. Out of tree might be a pain though.

[ianopolous]

So far we've done it out of tree (I assume that means a stand-alone build of go-ds-s3 as a plugin).

[ianopolous]

If I build ipfs locally, how do I ensure the go.mod points to the correct go-ipfs-blockstore repo/version? Just edit it?

[aschmahmann]

Yep, you can also use go mod replace directives (https://golang.org/ref/mod#go-mod-file-replace) on top level binaries (replace directives in libraries aren't utilized in things that import them). I also mistyped (sorry), the thing that will tend to make your life super easy (at least for testing) is just deploying a single binary using preloaded-plugins https://github.com/ipfs/go-ipfs/blob/master/docs/plugins.md#preloaded-plugins.

[ianopolous]

I've tried adding this line to go-ipfs/go.mod, and added the tag to the fork of blockstore

replace github.com/ipfs/go-ipfs-blockstore v0.1.4 => github.com/peergos/go-ipfs-blockstore v0.1.5

But make build fails with

missing go.sum entry for module providing package github.com/ipfs/go-ipfs-blockstore (imported by github.com/ipfs/go-ipfs/core); to add:
go get github.com/ipfs/go-ipfs/core

But running that command doesn't fix it. I don't normally work in Go, so sorry if these are trivial problems.

[aschmahmann]

@ianopolous no worries. You need to run go mod tidy first. However, you need to to base your blockstore branch off of the v0.1.4 tag instead of master/the v1 tag or it go-ipfs won't build.

Note: you don't need to tag the release in your repo if you don't need to you can just drop in the git commit hash.

[aschmahmann]

I put up a branch based of off v0 you can you in ipfs/go-ipfs-blockstore#79. This should work replace github.com/ipfs/go-ipfs-blockstore v0.1.4 => github.com/ipfs/go-ipfs-blockstore v0.1.5-0.20210617180433-625cea8e7c8e (or just putting the new version number where the old one is without a replace directive is also fine).

Hopefully ipfs/kubo#6815 will be resolved by go-ipfs v0.10.0 and we can put this multiple blockstore version nonsense behind us 🙏.

[ianopolous]

Thank you, I've got ipfs building now. Trying to build this plugin is failing (I normally build against ipfs releases):

make IPFS_VERSION=/home/ian/go-ipfs

go mod: -replace=github.com/ipfs/[email protected]=github.com/ipfs/go-ipfs-blockstore: unversioned new path must be local directory

This is on master of this repo. I'll try not using a replace directive in go-ipfs..

[ianopolous]

I'll try not using a replace directive in go-ipfs..

Yep that fixes it.

[ianopolous]

I've tried to test it, but the rebuilt plugin won't load with the locally built version of ipfs (from the release-v0.9.0 branch):

plugin was built with a different version of package github.com/jbenet/goprocess

I've hit this error before. I'll try and find the fix.

[ianopolous]

Last time it was a difference in the goflags, but that doesn't seem to be the current cause? #63

[ianopolous]

Yep that's definitely fixed it.