Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

swarm/network: Correct neighborhood depth #18066

Merged
merged 16 commits into from
Nov 26, 2018

Conversation

nolash
Copy link
Contributor

@nolash nolash commented Nov 8, 2018

This PR cherry-picks and amends fix by @zelig to neighborhood depth calculation from branch ethersphere/push-syncing-pss.

Before this fix, the depth would always be at the first proxpeer.

This fix makes sure that if there are vacant bins between the nearest non-prox peer and the first proxpeer, the first vacant bin will be the depth. It adjusts necessary tests accordingly.

It also removes number literals in kademlia params initialization.

@nolash nolash added this to the 1.8.18 milestone Nov 8, 2018
@nolash nolash requested review from janos and zelig November 8, 2018 12:42
@nonsense
Copy link
Member

nonsense commented Nov 8, 2018

I tried to run the TestNeighborhoodDepth on this branch and on master, but when I run the tests from swarm/network/kademlia_test.go on my machine, they panic.

@nolash
Copy link
Contributor Author

nolash commented Nov 9, 2018

@nonsense I can't see this panic on Travis, and I can't reproduce locally. Do you have a stack trace please?

swarm/network/kademlia.go Outdated Show resolved Hide resolved
swarm/network/kademlia.go Outdated Show resolved Hide resolved
Copy link
Contributor

@holisticode holisticode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a couple of comments about code comments....

I believe a lot of this code is obscure when you first hit it, and some more documentation would certainly be helpful.

Nevertheless, I don't mean to stop this PR from merging if it is urgent.

@@ -430,17 +442,44 @@ func (k *Kademlia) eachAddr(base []byte, o int, f func(*BzzAddr, int, bool) bool
// the nearest neighbour set with cardinality >= MinProxBinSize
// if there is altogether less than MinProxBinSize peers it returns 0
// caller must hold the lock
Copy link
Contributor

@holisticode holisticode Nov 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment still accurate? The lock is actually held inside the function now, no?

swarm/network/kademlia.go Outdated Show resolved Hide resolved
swarm/network/kademlia.go Show resolved Hide resolved
swarm/network/kademlia.go Outdated Show resolved Hide resolved
swarm/network/kademlia.go Outdated Show resolved Hide resolved
swarm/network/kademlia_test.go Show resolved Hide resolved
swarm/network/kademlia_test.go Show resolved Hide resolved
swarm/network/simulation/kademlia_test.go Outdated Show resolved Hide resolved
@nonsense nonsense modified the milestones: 1.8.18, 1.8.19 Nov 14, 2018
Copy link
Contributor

@janos janos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with existing comments. And I do not see the reason for some code changes that actually are not related to the problem that is solved.

swarm/network/simulation/kademlia_test.go Outdated Show resolved Hide resolved
swarm/network/kademlia.go Outdated Show resolved Hide resolved
swarm/network/kademlia.go Outdated Show resolved Hide resolved
@nolash
Copy link
Contributor Author

nolash commented Nov 14, 2018

Is there any insight what is causing the flakiness?

@janos My guess is that the healthy-checks are engineered to work with the previous (but faulty) kademlia calculations as source of truth. Now that the kademlia calculation is correct (which evaluation is the object of this PR) the tests relying on these calculations have to be re-evaluated. Before that happens, it's likely that dependent tests will flake. We can leave them, but then Travis may fail and we can't merge (or?)

@janos
Copy link
Contributor

janos commented Nov 14, 2018

Is there any insight what is causing the flakiness?

My guess is that the healthy-checks are engineered to work with the previous (but faulty) kademlia calculations as source of truth. Now that the kademlia calculation is correct (which evaluation is the object of this PR) the tests relying on these calculations have to be re-evaluated. Before that happens, it's likely that dependent tests will flake. We can leave them, but then Travis may fail and we can't merge (or?)

As @holisticode said, other tests are relying on this calculations. This means that they will be flaky also. Should we disable them, too?

@nolash
Copy link
Contributor Author

nolash commented Nov 14, 2018

As @holisticode said, other tests are relying on this calculations. This means that they will be flaky also. Should we disable them, too?

@janos Hm. Well. Some of them already flake, I guess, and maybe even because the calculations were incorrect. I don't have a good answer. And I don't know which ones all of htem are.

@janos
Copy link
Contributor

janos commented Nov 14, 2018

As @holisticode said, other tests are relying on this calculations. This means that they will be flaky also. Should we disable them, too?

@janos Hm. Well. Some of them already flake, I guess, and maybe even because the calculations were incorrect. I don't have a good answer. And I don't know which ones all of htem are.

It is easy to identify the test that need to be disabled, all of them that are using the tested functionality of already disabled test: Simulation.WaitTillHealthy function. A simple grep will do.

If your guess is that it would be the best to disable this test and leave others, from experience while working on this PR, then that is fine for me. But please, comment why the test is disabled as it would be a great help in the future.

Copy link

@ruthlesstom ruthlesstom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0

Spam

@nolash
Copy link
Contributor Author

nolash commented Nov 14, 2018

It is easy to identify the test that need to be disabled, all of them that are using the tested functionality of already disabled test: Simulation.WaitTillHealthy function. A simple grep will do.

Yes, of course. However, I meant of all tests that somehow depend on the wrong neighborhood depth calc. That said, I'll put the cart back behind the horse and see what flakes come out.

@janos
Copy link
Contributor

janos commented Nov 14, 2018

Thanks. For skipping TestWaitTillHealthy, just a simple comment with some context would be much appreciated.

@nolash nolash requested a review from acud as a code owner November 14, 2018 12:34
@nolash nolash removed the request for review from acud November 14, 2018 12:37
* swarm/network: simplified depth calculation

* swarm/network: changed depth calls to method not under read lock. reenabled waitTillHealty

* swarm/network: moved comment, replaced neighborhoodDepth with depthForPot

* swarm/network: revert to po==256

* swarm/network: Rename wrong var in iterator
@nonsense nonsense merged commit 1cd007e into ethereum:master Nov 26, 2018
@frncmx frncmx deleted the kademlia-neighbourhooddepth branch December 5, 2018 17:10
TuitionCoin added a commit to FinTechToken/go-ethereum that referenced this pull request Jun 16, 2019
* core: better printout of receipts in bad block reports (ethereum#18156)

* core/blockchain: better printout of receipts in bad block reports

* fix splleing

* params: add Constantinople block to AllXYZProtocolChanges (ethereum#18162)

* params: Add Constantinople block to AllCliqueProtocolChanges

* params: Add Constantinople block to AllEthashProtocolChanges

* trie: fix overflow in write cache parent tracking (ethereum#18165)

trie/database: fix overflow in parent tracking

* travis: increase open file limits (ethereum#18155)

* cmd/swarm: FUSE do not require --ipcpath (ethereum#18112)

- Have `${DataDir}/bzzd.ipc` as IPC path default.
- Respect the `--datadir` flag.
- Keep only the global `--ipcpath` flag and drop the local `--ipcpath` flag
  as flags might overwrite each other. (Note: before global `--ipcpath`
  was ignored even if it was set)

fixes ethersphere#795

* eth: increase timeout in TestBroadcastBlock (ethereum#18064)

* core: return error if repair block failed (ethereum#18126)

* core: return error if repair block failed

* make error a bit shorter

* config: add constantinople block to testchainconfig

* fix mixHash/nonce for parity compatible network (ethereum#18166)

* les: fix fetcher syncing logic (ethereum#18072)

* swarm/network/simulation: fix New function for-loop scope (ethereum#18161)

* swarm/api: improve not found error msg (ethereum#18171)

* light: odrTrie tryUpdate should use update (ethereum#18107)

TryUpdate does not call t.trie.TryUpdate(key, value) and calls t.trie.TryDelete
instead. The update operation simply deletes the corresponding entry, though
it could retrieve later by odr. However, it adds further network overhead.

* swarm/pss: Message handler refactor (ethereum#18169)

* tests: Add flag to use EVMC for state tests (ethereum#18084)

* Remove multihash from Swarm bzz:// for Feeds (ethereum#18175)

* Accounting metrics reporter (ethereum#18136)

* swarm/network: Correct neighborhood depth (ethereum#18066)

* cmd/swarm: update should error on manifest mismatch (ethereum#18047)

* cmd/swarm: fix ethersphere/swarm#979:

update should error on manifest mistmatch

* cmd/swarm: fixed comments and remove sprintf from log.Info

* cmd/swarm: remove unnecessary comment

* swarm: add database abstractions (shed package) (ethereum#18183)

* Increase bzz version (ethereum#18184)

* swarm/network/stream/: added stream protocol version match tests

* Increase BZZ version due to streamer version change; version tests

* swarm/network: increased hive and test protocol version

* p2p/protocols: fix minor comments typo (ethereum#18185)

* p2p/discv5: minor code simplification (ethereum#18188)

* Update net.go

more simple

* Update net.go

* p2p/discv5: gofmt

* core: more detailed metrics for block processing (ethereum#18119)

* eth/downloader: fix light client cht binary search issue

* params: update CHTs for the v1.8.19 release

* params, swarm: release Geth v1.8.19 and Swarm v0.3.7

* params, swarm: start Geth v1.8.20 and Swarm v0.3.8 release cycle

* remove unrelated code

* cmd/evm: commit statedb if dump is requested (ethereum#18208)

Add a call `statedb.Commit(true)` if the `Dump` flag is on, as otherwise the `storage` output in the dump is always empty.

* tests, core: update tests and make STATICCALL cause touch-delete (ethereum#18187)

* vendor: update leveldb

* remove a no-op line in the code (ethereum#17760)

* mobile: added constructor for BigInts (ethereum#17828)

* accounts/keystore: delete the redundant keystore in filename (ethereum#17930)

* accounts/keystore: reduce file name length

* accounts/keystore: reduce code line width

* cmd/utils: max out the OS file allowance, don't cap to 2K

* cmd/swarm: add flag for application name (swarm or swarm-private) (ethereum#18189)

* cmd/swarm: add flag for application name (swarm or swarm-private)

* cmd/swarm/swarm-smoke: return correct exit code

* cmd/swarm/swarm-smoke: remove colorable

* remove swarm/grafana_dashboards

* vendor: update github.com/karalabe/hid (ethereum#18213)

Fixes ethereum#15101 because hidapi is no longer being called from an
init function.

* p2p: use errors.New instead of fmt.Errorf (ethereum#18193)

* swarm/pss: Add same api interface for all Send* methods (ethereum#18218)

* signer/core: Fixes typo of method name in comment. (ethereum#18222)

* Changed http:// to https:// on JSON-RPC link (ethereum#18224)

Changed http:// to https:// on JSON-RPC link in README.md

* cmd/puppeth: enforce lowercase network names

* cmd/puppeth: implement chainspec converters

* whisperv6: remove duplicated code (ethereum#18015)

* cmd/puppeth: chain import/export via wizard, minor polishes

* Add packing for dynamic array and slice types (ethereum#18051)

* added tests for new abi encoding features (#4)

* added tests from bytes32[][] and string[]

* added offset to other types

* formatting

* Abi/dynamic types (#5)

* Revert "Abi/dynamic types (#5)" (#6)

This reverts commit dabca31.

* Abi/dynamic types (#7)

* some cleanup

* Apply suggestions from code review

apply suggestions

Co-Authored-By: vedhavyas <[email protected]>

* added better formatting (#8)

* review chnages

* better comments

* swarm/network/stream: Debug log instead of Warn for retrieval failure (ethereum#18246)

* swarm/api/http: add resetting timer metrics to requests (ethereum#18249)

* core, internal, eth, miner, les: Take VM config from BlockChain (ethereum#17955)

Until this commit, when sending an RPC request that called `NewEVM`, a blank `vm.Config`
would be taken so as to set some options, based on the default configuration. If some extra
configuration switches were passed to the blockchain, those would be ignored.

This PR adds a function to get the config from the blockchain, and this is what is now used
for RPC calls.

Some subsequent changes need to be made, see ethereum#17955 (review)
for the details of the discussion.

* swarm: snapshot load improvement (ethereum#18220)

* swarm/network: Hive - do not notify peer if discovery is disabled

* p2p/simulations: validate all connections on loading a snapshot

* p2p/simulations: track all connections in on snapshot loading

* p2p/simulations: add snapshotLoadTimeout variable

* p2p/simulations: ignore control events in snapshot load

* p2p/simulations: simplify event loop synchronization

* p2p/simulations: return already connected error from Load function

* p2p/simulations: log warning on snapshot loading disconnection

* node: warn when using deprecated config/resource files (ethereum#18199)

* cmd/puppeth: fix rogue quote in alethGenesisSpec JSON (ethereum#18262)

* eth/tracers: fixed incorrect storage from prestate_tracer (ethereum#18253)

* eth: fixed incorrect storage from prestate_tracer

* eth/tracers: updated assets.go

* params: set mainnet and Rinkeby Constantinople fork blocks

* cmd/evm, core/vm, eth: implement api methods to do stdjson dump to local filesystem

* eth, internal/web3ext: tiny polishes in tracers

* light: fix duplicated argument in bytes.Equal call

Most probably a copy/paste kind of error.
Found with gocritic `dupArg` checker.

* ethereum: fix typo in interfaces.go (ethereum#18266)

* Fix typo in interfaces.go

* Update interfaces.go

* cmd, eth: Add support for `--whitelist <blocknum>=<hash>,...` flag

* Rejects peers that respond with a different hash for any of the passed in block numbers.
* Meant for emergency situations when the network forks unexpectedly.

* cmd/utils, eth: minor polishes on whitelist code

* params: update CHTs for the 1.8.20 release

* swarm/network: Correct ambiguity in compared addresses (ethereum#18251)

* cmd/swarm, metrics, swarm/api/client, swarm/storage, swarm/metrics, swarm/api/http: add instrumentation (ethereum#18274)

* cmd/faucet: fix faucet static peer regression

* cmd/puppeth: support latest docker compose, expose faucet UDP

* cmd/geth, core, eth: implement Constantinople override flag (ethereum#18273)

* geth/core/eth: implement constantinople override flag

* les: implemnent constantinople override flag for les clients

* cmd/geth, eth, les: fix typo, move flag to experimentals

* params, swarm: release Geth v1.8.20 and Swarm v0.3.8

* params, swarm: begin Geth v1.9.0 family, Swarm v0.3.9 cycle

* swarm/shed: add metrics to each shed db (ethereum#18277)

* swarm/shed: add metrics to each shed db

* swarm/shed: push metrics prefix up

* swarm/shed: rename prefix to metricsPrefix

* swarm/shed: unexport Meter, remove Mutex for quit channel

* swarm/storage: simplify ChunkValidator interface (ethereum#18285)

* usbwallet: check returned error when decoding hexstr (ethereum#18056)

* usbwallet: check returned error when decoding hexstr

* Update accounts/usbwallet/ledger.go

Co-Authored-By: CoreyLin <[email protected]>

* usbwallet: check hex decode error

* crypto/secp256k1: Fix invalid document link (ethereum#18297)

* accounts/abi: argument type and name were reversed (ethereum#17947)

argument type and name were reversed

* rpc: add application/json-rpc as accepted content type, fixes ethereum#18293 (ethereum#18310)

* Comment error (ethereum#18303)

* Change issue labels in bot configs to the new prefixed version

* core/state: rename 'new' variable (ethereum#18301)

* p2p/discv5: don't hash findnode target in lookup against table (ethereum#18309)

* .github: add @gballet as abi codeowner (ethereum#18306)

* fix slice unpack bug in accounts/abi (ethereum#18321)

* fix slice unpack bug in accounts/abi

* swarm/storage/feed: remove unused code (ethereum#18324)

* p2p/simulation: move connection methods from swarm/network/simulation (ethereum#18323)

* Update visualized snapshot test (ethereum#18286)

* swarm/network/stream: fix visualized_snapshot_sync_sim_test

* swarm/network/stream: updated visualized snapshot-test;data in p2p event

* swarm/network/stream: cleanup visualized snapshot sync test

* swarm/network/stream: re-enable t.Skip for visualized test

* swarm/network/stream: addressed PR comments

* swarm/network/simulation:commented out unreachable code-avoid vet errors (ethereum#18263)

* swarm/pss: Reduce input vulnerabilities (ethereum#18304)

* swarm/storage: remove unused methods from Chunk interface (ethereum#18283)

* downloader: fix edgecase where returned index is OOB for downloader (ethereum#18335)

* downloader: fix edgecase where returned index is OOB for downloader

* downloader: documentation

Co-Authored-By: holiman <[email protected]>

* core: sanitize more TxPoolConfig fields (ethereum#17210)

* core: sanitize more TxPoolConfig fields

* core: fix TestTransactionPendingMinimumAllowance

* p2p/simulation: Test snapshot correctness and minimal benchmark (ethereum#18287)

* p2p/simulation: WIP minimal snapshot test

* p2p/simulation: Add snapshot create, load and verify to snapshot test

* build: add test tag for tests

* p2p/simulations, build: Revert travis change, build test sym always

* p2p/simulations: Add comments, timeout check on additional events

* p2p/simulation: Add benchmark template for minimal peer protocol init

* p2p/simulations: Remove unused code

* p2p/simulation: Correct timer reset

* p2p/simulations: Put snapshot check events in buffer and call blocking

* p2p/simulations: TestSnapshot fail if Load function returns early

* p2p/simulations: TestSnapshot wait for all connections before returning

* p2p/simulation: Revert to before wait for snap load (5e75594)

* p2p/simulations: add "conns after load" subtest to TestSnapshot

and nudge

* swarm/pss: forwarding function refactoring (ethereum#18353)

* eth/downloader: progress in stateSync not used anymore (ethereum#17998)

* p2p/protocols: accounting metrics rpc (ethereum#18336)

* p2p/protocols: accounting metrics rpc added (ethereum#847)

* p2p/protocols: accounting api documentation added (ethereum#847)

* p2p/protocols: accounting api doc updated (ethereum#847)

* p2p/protocols: accounting api doc update (ethereum#847)

* p2p/protocols: accounting api doc update (ethereum#847)

* p2p/protocols: fix file is not gofmted

* fix lint error

* updated comments after review

* add account balance to rpc

* naming changed after review

* swarm/network: Revised depth and health for Kademlia (ethereum#18354)

* swarm/network: Revised depth calculation with tests

* swarm/network: WIP remove redundant "full" function

* swarm/network: WIP peerpot refactor

* swarm/network: Make test methods submethod of peerpot and embed kad

* swarm/network: Remove commented out code

* swarm/network: Rename health test functions

* swarm/network: Too many n's

* swarm/network: Change hive Healthy func to accept addresses

* swarm/network: Add Healthy proxy method for api in hive

* swarm/network: Skip failing test out of scope for PR

* swarm/network: Skip all tests dependent on SuggestPeers

* swarm/network: Remove commented code and useless kad Pof member

* swarm/network: Remove more unused code, add counter on depth test errors

* swarm/network: WIP Create Healthy assertion tests

* swarm/network: Roll back health related methods receiver change

* swarm/network: Hardwire network minproxbinsize in swarm sim

* swarm/network: Rework Health test to strict

Pending add test for saturation
And add test for as many as possible up to saturation

* swarm/network: Skip discovery tests (dependent on SuggestPeer)

* swarm/network: Remove useless minProxBinSize in stream

* swarm/network: Remove unnecessary testing.T param to assert health

* swarm/network: Implement t.Helper() in checkHealth

* swarm/network: Rename check back to assert now that we have helper magic

* swarm/network: Revert WaitTillHealthy change (deferred to nxt PR)

* swarm/network: Kademlia tests GotNN => ConnectNN

* swarm/network: Renames and comments

* swarm/network: Add comments

* accounts/abi: add support for unpacking returned bytesN arrays (ethereum#15242)

* accounts/abi: Brings out the msg defined at require statement in SC function (ethereum#17328)

* swarm: remove unused/dead code (ethereum#18351)

* fix string array unpack bug in accounts/abi (ethereum#18364)

* core/types: update incorrect comment

* accounts/abi: change unpacking of abi fields w/ underscores (ethereum#16513)

* accounts/abi: fix name styling when unpacking abi fields w/ underscores

ABI fields with underscores that are being unpacked
into structs expect structs with following form:

int_one -> Int_one

whereas in abigen the generated structs are camelcased

int_one -> IntOne

so updated the unpack method to expect camelcased structs as well.

* accounts/abi: fix case of generated java functions (ethereum#18372)

* rpc: Warn the user when the path name is too long for the Unix ipc endpoint (ethereum#18330)

* swarm/storage: change Proximity function and add TestProximity test (ethereum#18379)

* build: add LGPL license at update-license.go (ethereum#18377)

* add LGPL licence at update-licence.go

* add empty line

* travis, appveyor: bump to Go 1.11.4 (ethereum#18314)

* travis, appveyor: bump to Go 1.11.4

* internal/build: revert comment changes

* accounts/abi/bind: add optional block number for calls (ethereum#17942)

* vendor: vendor/github.com/mattn/go-isatty - add missing files (reported by mksully22) (ethereum#18376)

* swarm/docker: Dockerfile for swarm:edge docker image (ethereum#18386)

* vendor, crypto, swarm: switch over to upstream sha3 package

* vendor: update the entire golang.org/x/crypto dependency

* cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses (ethereum#18344)

* cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses

* cmd/puppeth: don't need to handle duplicate set

* eth/tracer: extend create2 (ethereum#18318)

* eth/tracer: extend create2

* eth/tracers: fix create2-flaw in prestate_tracer

* eth/tracers: fix test

* eth/tracers: update assets

* accounts/keystore: fix comment typo (ethereum#18395)

* A few minor code inspection fixes (ethereum#18393)

* swarm/network: fix code inspection problems

- typos
- redundant import alias

* p2p/simulations: fix code inspection problems

- typos
- unused function parameters
- redundant import alias
- code style issue: snake case

* swarm/network: fix unused method parameters inspections

* accounts/keystore: small code simplification (ethereum#18394)

* internal/ethapi: ask transaction pool for pending nonce (ethereum#15794)

* cmd/geth: support dumpconfig optionally saving to file (ethereum#18327)

* Changed dumpConfig function to optionally save to file

* Added O_TRUNC flag to file open and cleaned up code

* swarm: Shed Index and Uint64Field additions (ethereum#18398)

* swarm/storage/mock/test: fix T.Fatal inside a goroutine (ethereum#18399)

* swarm, p2p/protocols: Stream accounting (ethereum#18337)

* swarm: completed 1st phase of swap accounting

* swarm, p2p/protocols: added stream pricing

* swarm/network/stream: gofmt simplify stream.go

* swarm: fixed review comments

* swarm: used snapshots for swap tests

* swarm: custom retrieve for swap (less cascaded requests at any one time)

* swarm: addressed PR comments

* swarm: log output formatting

* swarm: removed parallelism in swap tests

* swarm: swap tests simplification

* swarm: removed swap_test.go

* swarm/network/stream: added prefix space for comments

* swarm/network/stream: unit test for prices

* swarm/network/stream: don't hardcode price

* swarm/network/stream: fixed invalid price check

* github: remove swarm github codeowners (ethereum#18412)

* swarm: Fix T.Fatal inside a goroutine in tests (ethereum#18409)

* swarm/storage: fix T.Fatal inside a goroutine

* swarm/network/simulation: fix T.Fatal inside a goroutine

* swarm/network/stream: fix T.Fatal inside a goroutine

* swarm/network/simulation: consistent failures in TestPeerEventsTimeout

* swarm/network/simulation: rename sendRunSignal to triggerSimulationRun

* swarm/network: remove isproxbin bool from kad.Each* iterfunc (ethereum#18239)

* swarm/network, swarm/pss: remove isproxbin bool from kad.Each* iterfunc

* swarm/network: restore comment and unskip snapshot sync tests

* accounts/abi: tuple support (ethereum#18406)

* swarm/network: Rename minproxbinsize, add as member of simulation (ethereum#18408)

* swarm/network: Rename minproxbinsize, add as member of simulation

* swarm/network: Deactivate WaitTillHealthy, unreliable pending suggestpeer

* accounts/abi: Extra slice tests (ethereum#18424)

Co-authored-by: weimumu <[email protected]>

* p2p/simulations: eliminate concept of pivot (ethereum#18426)

* core, eth: fix database version (ethereum#18429)

* core, eth: fix database version

* eth: polish error message

* Stream subscriptions (ethereum#18355)

* swarm/network: eachBin now starts at kaddepth for nn

* swarm/network: fix Kademlia.EachBin

* swarm/network: fix kademlia.EachBin

* swarm/network: correct EachBin implementation according to requirements

* swarm/network: less addresses simplified tests

* swarm: calc kad depth outside loop in EachBin test

* swarm/network: removed printResults

* swarm/network: cleanup imports

* swarm/network: remove kademlia.EachBin; fix RequestSubscriptions and add unit test

* swarm/network/stream: address PR comments

* swarm/network/stream: package-wide subscriptionFunc

* swarm/network/stream: refactor to kad.EachConn

* swarm/pot: pot.remove fixed (ethereum#18431)

* swarm/pot: refactored pot.remove(), updated comments

* swarm/pot: comments updated

* swarm/pot: each() functions refactored (ethereum#18452)

* accounts/abi: Add tests for reflection ahead of refactor (ethereum#18434)

* params: postpone Constantinople due to net SSTORE reentrancy

* params, swarm: release Geth v1.8.21 and Swarm v0.3.9

* core, cmd/puppeth: implement constantinople fix, disable EIP-1283 (ethereum#18486)

This PR adds a new fork which disables EIP-1283. Internally it's called Petersburg,
but the genesis/config field is ConstantinopleFix.

The block numbers are:

    7280000 for Constantinople on Mainnet
    7280000 for ConstantinopleFix on Mainnet
    4939394 for ConstantinopleFix on Ropsten
    9999999 for ConstantinopleFix on Rinkeby (real number decided later)

This PR also defaults to using the same ConstantinopleFix number as whatever
Constantinople is set to. That is, it will default to mainnet behaviour if ConstantinopleFix
is not set.This means that for private networks which have already transitioned
to Constantinople, this PR will break the network unless ConstantinopleFix is
explicitly set!

* travis, appveyor: bump to Go 1.11.5 (ethereum#18947)

* build: tweak debian source package build/upload options (ethereum#18962)

dput --passive should make repo pushes from Travis work again.
dput --no-upload-log works around an issue I had while uploading locally.

debuild -d says that debuild shouldn't check for build dependencies when
creating the source package. This option is needed to make builds work
in environments where the installed Go version doesn't match the
declared dependency in the source package.

* p2p/discover, p2p/enode: rework endpoint proof handling, packet logging (ethereum#18963)

This change resolves multiple issues around handling of endpoint proofs.
The proof is now done separately for each IP and completing the proof
requires a matching ping hash.

Also remove waitping because it's equivalent to sleep. waitping was
slightly more efficient, but that may cause issues with findnode if
packets are reordered and the remote end sees findnode before pong.

Logging of received packets was hitherto done after handling the packet,
which meant that sent replies were logged before the packet that
generated them. This change splits up packet handling into 'preverify'
and 'handle'. The error from 'preverify' is logged, but 'handle' happens
after the message is logged. This fixes the order. Packet logs now
contain the node ID.

* params: new CHTs (ethereum#18577)

* p2p/discover: improve table addition code (ethereum#18974)

This change clears up confusion around the two ways in which nodes
can be added to the table.

When a neighbors packet is received as a reply to findnode, the nodes
contained in the reply are added as 'seen' entries if sufficient space
is available.

When a ping is received and the endpoint verification has taken place,
the remote node is added as a 'verified' entry or moved to the front of
the bucket if present. This also updates the node's IP address and port
if they have changed.

* params, swarm/version: Geth 1.8.22-stable, Swarm 0.3.10-stable

* eth: make tracers respect pre- EIP 158/161 rule

* core: fix error in block iterator (ethereum#18986)

* cmd/puppeth: handle pre-set Petersburg number, save changed fork rules

* core: repro ethereum#18977

* core: fix pruner panic when importing low-diff-large-sidechain

* cmd, core, params: add support for Goerli

(cherry picked from commit b0ed083)

* build: use SFTP for launchpad uploads (ethereum#19037)

* build: use sftp for launchpad uploads

* .travis.yml: configure sftp export

* build: update CI docs

(cherry picked from commit 3de19c8)

* common/fdlimit: cap on MacOS file limits, fixes ethereum#18994 (ethereum#19035)

* common/fdlimit: cap on MacOS file limits, fixes ethereum#18994

* common/fdlimit: fix Maximum-check to respect OPEN_MAX

* common/fdlimit: return error if OPEN_MAX is exceeded in Raise()

* common/fdlimit: goimports

* common/fdlimit: check value after setting fdlimit

* common/fdlimit: make comment a bit more descriptive

* cmd/utils: make fdlimit happy path a bit cleaner

(cherry picked from commit f48da43)

* .travis.yml: fix upload destination (ethereum#19043)


(cherry picked from commit edf976e)

* build: avoid dput and upload with sftp directly (ethereum#19067)


(cherry picked from commit a8ddf7a)

* common/fdlimit: fix windows build (ethereum#19068)


(cherry picked from commit ba90a4a)

* vendor: update syscalls dependency

(cherry picked from commit dcc045f)

* vendor: pull in upstream syscall fixes for non-linux/arm64

(cherry picked from commit 9d3ea8d)

* trie: fix error in node decoding (ethereum#19111)

* vendor: update bigcache

(cherry picked from commit 37e5a90)

* travis.yml: add launchpad SSH public key (ethereum#19115)


(cherry picked from commit 75a9314)

* build: explicitly force .xz compression (old debuild picks gzip) (ethereum#19118)


(cherry picked from commit c0b9c76)

* swarm/network: fix data race in TestNetworkID test (ethereum#18460)

(cherry picked from commit 96c7c18)

* swarm/storage: fix mockNetFetcher data races (ethereum#18462)

fixes: ethersphere/swarm#1117
(cherry picked from commit f728837)

* cmd/swarm/swarm-snapshot: swarm snapshot generator (ethereum#18453)

* cmd/swarm/swarm-snapshot: add binary to create network snapshots

* cmd/swarm/swarm-snapshot: refactor and extend tests

* p2p/simulations: remove unused triggerChecks func and fix linter

* internal/cmdtest: raise the timeout for killing TestCmd

* cmd/swarm/swarm-snapshot: add more comments and other minor adjustments

* cmd/swarm/swarm-snapshot: remove redundant check in createSnapshot

* cmd/swarm/swarm-snapshot: change comment wording

* p2p/simulations: revert Simulation.Run from master

https://github.com/ethersphere/go-ethereum/pull/1077/files#r247078904

* cmd/swarm/swarm-snapshot: address pr comments

* swarm/network/simulations/discovery: removed snapshot write to file

* cmd/swarm/swarm-snapshot, swarm/network/simulations: removed redundant connection event check, fixed lint error

(cherry picked from commit 34f11e7)

* swarm/network: rewrite of peer suggestion engine, fix skipped tests (ethereum#18404)

* swarm/network: fix skipped tests related to suggestPeer

* swarm/network: rename depth to radius

* swarm/network: uncomment assertHealth and improve comments

* swarm/network: remove commented code

* swarm/network: kademlia suggestPeer algo correction

* swarm/network: kademlia suggest peer

 * simplify suggest Peer code
 * improve peer suggestion algo
 * add comments
 * kademlia testing improvements
   * assertHealth -> checkHealth (test helper)
   * testSuggestPeer -> checkSuggestPeer (test helper)
   * remove testSuggestPeerBug and TestKademliaCase

* swarm/network: kademlia suggestPeer cleanup, improved comments

* swarm/network: minor comment, discovery test default arg

(cherry picked from commit bcb2594)

* swarm/network: fix data race warning on TestBzzHandshakeLightNode (ethereum#18459)

(cherry picked from commit 81e26d5)

* Upload speed (ethereum#18442)

(cherry picked from commit 257bfff)

* cmd/swarm/swarm-snapshot: disable tests on windows (ethereum#18478)

(cherry picked from commit 632135c)

* cmd/swarm: use resetting timer to measure fetch time (ethereum#18474)

(cherry picked from commit a0b0db6)

* cmd/swarm/swarm-smoke: use ResettingTimer instead of Counters for times (ethereum#18479)

(cherry picked from commit 5609577)

* p2p/simulations: fix data race on swarm/network/simulations (ethereum#18464)

(cherry picked from commit 85a79b3)

* swarm: bootnode-mode, new bootnodes and no p2p package discovery (ethereum#18498)

(cherry picked from commit bbd1203)

* swarm: fix a data race on startTime (ethereum#18511)

(cherry picked from commit fa34429)

* swarm/version: commit version added (ethereum#18510)

(cherry picked from commit ad13d2d)

* p2p/testing, swarm: remove unused testing.T in protocol tester (ethereum#18500)

(cherry picked from commit 2abeb35)

* swarm/network: Remove extra random peer, connect test sanity, comments (ethereum#18964)

(cherry picked from commit f9401ae)

* swarm: fix flaky delivery tests (ethereum#18971)

(cherry picked from commit 592bf6a)

* swarm/storage: Get all chunk references for a given file (ethereum#19002)

(cherry picked from commit 3eff652)

* cmd/swarm: hashes command (ethereum#19008)

(cherry picked from commit 7f55b0c)

* cmd/swarm/swarm-smoke: sliding window test (ethereum#18967)

(cherry picked from commit b91bf08)

* cmd/swarm/swarm-smoke: remove wrong metrics (ethereum#18970)

(cherry picked from commit c5c9cef)

* cmd/swarm/swarm-smoke: refactor generateEndpoints (ethereum#19006)

(cherry picked from commit d212535)

* swarm: Debug API and HasChunks() API endpoint (ethereum#18980)

(cherry picked from commit 41597c2)

* swarm/storage: fix test timeout with -race by increasing mget timeout

(cherry picked from commit 1c3aa8d)

* swarm/pss: Remove pss service leak in test (ethereum#18992)

(cherry picked from commit 7c60d0a)

* swarm/storage/localstore: new localstore package (ethereum#19015)

(cherry picked from commit 4f3d22f)

* swarm/network: fix data race in fetcher_test.go (ethereum#18469)

(cherry picked from commit 19bfcbf)

* swarm/network, swarm/storage: Preserve opentracing contexts (ethereum#19022)

(cherry picked from commit 0c10d37)

* swarm/pss: transition to whisper v6 (ethereum#19023)

(cherry picked from commit cde02e0)

* swarm: GetPeerSubscriptions RPC (ethereum#18972)

(cherry picked from commit 43e1b7b)

* swarm/network: refactor simulation tests bootstrap (ethereum#18975)

(cherry picked from commit 597597e)

* cmd/swarm/global-store: global store cmd (ethereum#19014)

(cherry picked from commit 33d0a0e)

* swarm: CI race detector test adjustments (ethereum#19017)

(cherry picked from commit 27e3f96)

* swarm/docker: add global-store and split docker images (ethereum#19038)

(cherry picked from commit 6cb7d52)

* swarm/pss: mutex lifecycle fixed (ethereum#19045)

(cherry picked from commit b30109d)

* swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (ethereum#19028)

* swarm/storage: fix HashExplore concurrency bug ethersphere#1211

*  swarm/storage: lock as value not pointer

* swarm/storage: wait for  to complete

* swarm/storage: fix linter problems

* swarm/storage: append to nil slice

(cherry picked from commit 3d22a46)

* swarm: fix uptime gauge update goroutine leak by introducing cleanup functions (ethereum#19040)

(cherry picked from commit d596bea)

* swarm: fix network/stream data races (ethereum#19051)

* swarm/network/stream: newStreamerTester cleanup only if err is nil

* swarm/network/stream: raise newStreamerTester waitForPeers timeout

* swarm/network/stream: fix data races in GetPeerSubscriptions

* swarm/storage: prevent data race on LDBStore.batchesC

ethersphere/swarm#1198 (comment)

* swarm/network/stream: fix TestGetSubscriptionsRPC data race

ethersphere/swarm#1198 (comment)

* swarm/network/stream: correctly use Simulation.Run callback

ethersphere/swarm#1198 (comment)

* swarm/network: protect addrCountC in Kademlia.AddrCountC function

ethersphere/swarm#1198 (comment)

* p2p/simulations: fix a deadlock calling getRandomNode with lock

ethersphere/swarm#1198 (comment)

* swarm/network/stream: terminate disconnect goruotines in tests

* swarm/network/stream: reduce memory consumption when testing data races

* swarm/network/stream: add watchDisconnections helper function

* swarm/network/stream: add concurrent counter for tests

* swarm/network/stream: rename race/norace test files and use const

* swarm/network/stream: remove watchSim and its panic

* swarm/network/stream: pass context in watchDisconnections

* swarm/network/stream: add concurrent safe bool for watchDisconnections

* swarm/storage: fix LDBStore.batchesC data race by not closing it

(cherry picked from commit 3fd6db2)

* swarm/storage/netstore: add fetcher cancellation on shutdown (ethereum#19049)

swarm/network/stream: remove netstore internal wg
swarm/network/stream: run individual tests with t.Run

(cherry picked from commit 3ee09ba)

* swarm/network: Saturation check for healthy networks (ethereum#19071)

* swarm/network: new saturation for  implementation

* swarm/network: re-added saturation func in Kademlia as it is used elsewhere

* swarm/network: saturation with higher MinBinSize

* swarm/network: PeersPerBin with depth check

* swarm/network: edited tests to pass new saturated check

* swarm/network: minor fix saturated check

* swarm/network/simulations/discovery: fixed renamed RPC call

* swarm/network: renamed to isSaturated and returns bool

* swarm/network: early depth check

(cherry picked from commit 2af2472)

* swarm/storage: fix influxdb gc metrics report (ethereum#19102)

(cherry picked from commit 5b8ae78)

* swarm/pss: refactoring (ethereum#19110)

* swarm/pss: split pss and keystore

* swarm/pss: moved whisper to keystore

* swarm/pss: goimports fixed

(cherry picked from commit 12ca3b1)

* p2p, swarm: fix node up races by granular locking (ethereum#18976)

* swarm/network: DRY out repeated giga comment

I not necessarily agree with the way we wait for event propagation.
But I truly disagree with having duplicated giga comments.

* p2p/simulations: encapsulate Node.Up field so we avoid data races

The Node.Up field was accessed concurrently without "proper" locking.
There was a lock on Network and that was used sometimes to access
the  field. Other times the locking was missed and we had
a data race.

For example: ethereum#18464
The case above was solved, but there were still intermittent/hard to
reproduce races. So let's solve the issue permanently.

resolves: ethersphere/swarm#1146

* p2p/simulations: fix unmarshal of simulations.Node

Making Node.Up field private in 13292ee
broke TestHTTPNetwork and TestHTTPSnapshot. Because the default
UnmarshalJSON does not handle unexported fields.

Important: The fix is partial and not proper to my taste. But I cut
scope as I think the fix may require a change to the current
serialization format. New ticket:
ethersphere/swarm#1177

* p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON

* p2p/simulations: revert back to defer Unlock() pattern for Network

It's a good patten to call `defer Unlock()` right after `Lock()` so
(new) error cases won't miss to unlock. Let's get back to that pattern.

The patten was abandoned in 85a79b3,
while fixing a data race. That data race does not exist anymore,
since the Node.Up field got hidden behind its own lock.

* p2p/simulations: consistent naming for test providers Node.UnmarshalJSON

* p2p/simulations: remove JSON annotation from private fields of Node

As unexported fields are not serialized.

* p2p/simulations: fix deadlock in Network.GetRandomDownNode()

Problem: GetRandomDownNode() locks -> getDownNodeIDs() ->
GetNodes() tries to lock -> deadlock

On Network type, unexported functions must assume that `net.lock`
is already acquired and should not call exported functions which
might try to lock again.

* p2p/simulations: ensure method conformity for Network

Connect* methods were moved to p2p/simulations.Network from
swarm/network/simulation. However these new methods did not follow
the pattern of Network methods, i.e., all exported method locks
the whole Network either for read or write.

* p2p/simulations: fix deadlock during network shutdown

`TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock.
The execution was stuck on two locks, i.e, `Kademlia.lock` and
`p2p/simulations.Network.lock`. Usually the test got stuck once in each
20 executions with high confidence.

`Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in
`Network.Stop()`.

Solution: in `Network.Stop()` `net.lock` must be released before
calling `node.Stop()` as stopping a node (somehow - I did not find
the exact code path) causes `Network.InitConn()` to be called from
`Kademlia.SuggestPeer()` and that blocks on `net.lock`.

Related ticket: ethersphere/swarm#1223

* swarm/state: simplify if statement in DBStore.Put()

* p2p/simulations: remove faulty godoc from private function

The comment started with the wrong method name.

The method is simple and self explanatory. Also, it's private.
=> Let's just remove the comment.

(cherry picked from commit 50b872b)

* cmd/swarm/swarm-smoke: Trigger chunk debug on timeout (ethereum#19101)

* cmd/swarm/swarm-smoke: first version trigger has-chunks on timeout

* cmd/swarm/swarm-smoke: finalize trigger to chunk debug

* cmd/swarm/swarm-smoke: fixed httpEndpoint for trigger

* cmd/swarm/swarm-smoke: port

* cmd/swarm/swarm-smoke: ws not rpc

* cmd/swarm/swarm-smoke: added debug output

* cmd/swarm/swarm-smoke: addressed PR comments

* cmd/swarm/swarm-smoke: renamed track-timeout and track-chunks

(cherry picked from commit 62d7688)

* contracts/*: golint updates for this or self warning

(cherry picked from commit 53b823a)

* swarm: Reinstate Pss Protocol add call through swarm service (ethereum#19117)

* swarm: Reinstate Pss Protocol add call through swarm service

* swarm: Even less self

(cherry picked from commit d88c6ce)

* swarm/metrics: Send the accounting registry to InfluxDB (ethereum#18470)

(cherry picked from commit f28da4f)

* cmd/utils: allow for multiple influxdb tags (ethereum#18520)

This PR is replacing the metrics.influxdb.host.tag cmd-line flag with metrics.influxdb.tags - a comma-separated key/value tags, that are passed to the InfluxDB reporter, so that we can index measurements with multiple tags, and not just one host tag.

This will be useful for Swarm, where we want to index measurements not just with the host tag, but also with bzzkey and git commit version (for long-running deployments).

(cherry picked from commit 21acf0b)

* params, swarm: release Geth v1.8.23, Swarm v0.3.11

* light, params: update CHTs, integrate CHT for Goerli too

* common/fdlimit: fix macos file descriptors for Go 1.12

* all: simplify timestamps to uint64 (ethereum#19372)

* all: simplify timestamps to uint64

* tests: update definitions

* clef, faucet, mobile: leftover uint64 fixups

* ethash: fix tests

* graphql: update schema for timestamp

* ethash: remove unused variable

* trie: there's no point in retrieving the metaroot

* travis: update builders to xenial to shadow Go releases

* core: re-omit new log event when logs rebirth

* core: minor code polishes + rebase fixes

* params: set Rinkeby Petersburg fork block (4th May, 2019)

* p2p/discover: bump failure counter only if no nodes were provided (ethereum#19362)

This resolves a minor issue where neighbors responses containing less
than 16 nodes would bump the failure counter, removing the node. One
situation where this can happen is a private deployment where the total
number of extant nodes is less than 16.

Issue found by @Jsying.

* eth, les, geth: implement cli-configurable global gas cap for RPC calls (ethereum#19401)

* eth, les, geth: implement cli-configurable global gas cap for RPC calls

* graphql, ethapi: place gas cap in DoCall

* ethapi: reformat log message

* params, swarm: release Geth v1.8.24 (noop Swarm 0.3.12)

* cmd/geth: fix accidental --rpccorsdomain and --rpcvhosts removal

* params, swarm: hotfix Geth v1.8.25 release to restore rpc flags

* les: backported new SendTx cost calculation

* les: check required message types in cost table

* params, swarm: release Geth v1.8.26 (+noop Swarm v0.3.14)

* eth, les, light: enforce CHT checkpoints on fast-sync too

* params, swarm: release Geth v1.8.27 (noop Swarm v0.3.15)

* forward to 1.8.27
TuitionCoin added a commit to FinTechToken/go-ethereum that referenced this pull request Jun 16, 2019
* trie: fix overflow in write cache parent tracking (ethereum#18165)

trie/database: fix overflow in parent tracking

* travis: increase open file limits (ethereum#18155)

* cmd/swarm: FUSE do not require --ipcpath (ethereum#18112)

- Have `${DataDir}/bzzd.ipc` as IPC path default.
- Respect the `--datadir` flag.
- Keep only the global `--ipcpath` flag and drop the local `--ipcpath` flag
  as flags might overwrite each other. (Note: before global `--ipcpath`
  was ignored even if it was set)

fixes ethersphere#795

* eth: increase timeout in TestBroadcastBlock (ethereum#18064)

* core: return error if repair block failed (ethereum#18126)

* core: return error if repair block failed

* make error a bit shorter

* config: add constantinople block to testchainconfig

* fix mixHash/nonce for parity compatible network (ethereum#18166)

* les: fix fetcher syncing logic (ethereum#18072)

* swarm/network/simulation: fix New function for-loop scope (ethereum#18161)

* swarm/api: improve not found error msg (ethereum#18171)

* light: odrTrie tryUpdate should use update (ethereum#18107)

TryUpdate does not call t.trie.TryUpdate(key, value) and calls t.trie.TryDelete
instead. The update operation simply deletes the corresponding entry, though
it could retrieve later by odr. However, it adds further network overhead.

* swarm/pss: Message handler refactor (ethereum#18169)

* tests: Add flag to use EVMC for state tests (ethereum#18084)

* Remove multihash from Swarm bzz:// for Feeds (ethereum#18175)

* Accounting metrics reporter (ethereum#18136)

* swarm/network: Correct neighborhood depth (ethereum#18066)

* cmd/swarm: update should error on manifest mismatch (ethereum#18047)

* cmd/swarm: fix ethersphere/swarm#979:

update should error on manifest mistmatch

* cmd/swarm: fixed comments and remove sprintf from log.Info

* cmd/swarm: remove unnecessary comment

* swarm: add database abstractions (shed package) (ethereum#18183)

* Increase bzz version (ethereum#18184)

* swarm/network/stream/: added stream protocol version match tests

* Increase BZZ version due to streamer version change; version tests

* swarm/network: increased hive and test protocol version

* p2p/protocols: fix minor comments typo (ethereum#18185)

* p2p/discv5: minor code simplification (ethereum#18188)

* Update net.go

more simple

* Update net.go

* p2p/discv5: gofmt

* core: more detailed metrics for block processing (ethereum#18119)

* eth/downloader: fix light client cht binary search issue

* params: update CHTs for the v1.8.19 release

* params, swarm: release Geth v1.8.19 and Swarm v0.3.7

* params, swarm: start Geth v1.8.20 and Swarm v0.3.8 release cycle

* remove unrelated code

* cmd/evm: commit statedb if dump is requested (ethereum#18208)

Add a call `statedb.Commit(true)` if the `Dump` flag is on, as otherwise the `storage` output in the dump is always empty.

* tests, core: update tests and make STATICCALL cause touch-delete (ethereum#18187)

* vendor: update leveldb

* remove a no-op line in the code (ethereum#17760)

* mobile: added constructor for BigInts (ethereum#17828)

* accounts/keystore: delete the redundant keystore in filename (ethereum#17930)

* accounts/keystore: reduce file name length

* accounts/keystore: reduce code line width

* cmd/utils: max out the OS file allowance, don't cap to 2K

* cmd/swarm: add flag for application name (swarm or swarm-private) (ethereum#18189)

* cmd/swarm: add flag for application name (swarm or swarm-private)

* cmd/swarm/swarm-smoke: return correct exit code

* cmd/swarm/swarm-smoke: remove colorable

* remove swarm/grafana_dashboards

* vendor: update github.com/karalabe/hid (ethereum#18213)

Fixes ethereum#15101 because hidapi is no longer being called from an
init function.

* p2p: use errors.New instead of fmt.Errorf (ethereum#18193)

* swarm/pss: Add same api interface for all Send* methods (ethereum#18218)

* signer/core: Fixes typo of method name in comment. (ethereum#18222)

* Changed http:// to https:// on JSON-RPC link (ethereum#18224)

Changed http:// to https:// on JSON-RPC link in README.md

* cmd/puppeth: enforce lowercase network names

* cmd/puppeth: implement chainspec converters

* whisperv6: remove duplicated code (ethereum#18015)

* cmd/puppeth: chain import/export via wizard, minor polishes

* Add packing for dynamic array and slice types (ethereum#18051)

* added tests for new abi encoding features (#4)

* added tests from bytes32[][] and string[]

* added offset to other types

* formatting

* Abi/dynamic types (#5)

* Revert "Abi/dynamic types (#5)" (#6)

This reverts commit dabca31.

* Abi/dynamic types (#7)

* some cleanup

* Apply suggestions from code review

apply suggestions

Co-Authored-By: vedhavyas <[email protected]>

* added better formatting (#8)

* review chnages

* better comments

* swarm/network/stream: Debug log instead of Warn for retrieval failure (ethereum#18246)

* swarm/api/http: add resetting timer metrics to requests (ethereum#18249)

* core, internal, eth, miner, les: Take VM config from BlockChain (ethereum#17955)

Until this commit, when sending an RPC request that called `NewEVM`, a blank `vm.Config`
would be taken so as to set some options, based on the default configuration. If some extra
configuration switches were passed to the blockchain, those would be ignored.

This PR adds a function to get the config from the blockchain, and this is what is now used
for RPC calls.

Some subsequent changes need to be made, see ethereum#17955 (review)
for the details of the discussion.

* swarm: snapshot load improvement (ethereum#18220)

* swarm/network: Hive - do not notify peer if discovery is disabled

* p2p/simulations: validate all connections on loading a snapshot

* p2p/simulations: track all connections in on snapshot loading

* p2p/simulations: add snapshotLoadTimeout variable

* p2p/simulations: ignore control events in snapshot load

* p2p/simulations: simplify event loop synchronization

* p2p/simulations: return already connected error from Load function

* p2p/simulations: log warning on snapshot loading disconnection

* node: warn when using deprecated config/resource files (ethereum#18199)

* cmd/puppeth: fix rogue quote in alethGenesisSpec JSON (ethereum#18262)

* eth/tracers: fixed incorrect storage from prestate_tracer (ethereum#18253)

* eth: fixed incorrect storage from prestate_tracer

* eth/tracers: updated assets.go

* params: set mainnet and Rinkeby Constantinople fork blocks

* cmd/evm, core/vm, eth: implement api methods to do stdjson dump to local filesystem

* eth, internal/web3ext: tiny polishes in tracers

* light: fix duplicated argument in bytes.Equal call

Most probably a copy/paste kind of error.
Found with gocritic `dupArg` checker.

* ethereum: fix typo in interfaces.go (ethereum#18266)

* Fix typo in interfaces.go

* Update interfaces.go

* cmd, eth: Add support for `--whitelist <blocknum>=<hash>,...` flag

* Rejects peers that respond with a different hash for any of the passed in block numbers.
* Meant for emergency situations when the network forks unexpectedly.

* cmd/utils, eth: minor polishes on whitelist code

* params: update CHTs for the 1.8.20 release

* swarm/network: Correct ambiguity in compared addresses (ethereum#18251)

* cmd/swarm, metrics, swarm/api/client, swarm/storage, swarm/metrics, swarm/api/http: add instrumentation (ethereum#18274)

* cmd/faucet: fix faucet static peer regression

* cmd/puppeth: support latest docker compose, expose faucet UDP

* cmd/geth, core, eth: implement Constantinople override flag (ethereum#18273)

* geth/core/eth: implement constantinople override flag

* les: implemnent constantinople override flag for les clients

* cmd/geth, eth, les: fix typo, move flag to experimentals

* params, swarm: release Geth v1.8.20 and Swarm v0.3.8

* params, swarm: begin Geth v1.9.0 family, Swarm v0.3.9 cycle

* swarm/shed: add metrics to each shed db (ethereum#18277)

* swarm/shed: add metrics to each shed db

* swarm/shed: push metrics prefix up

* swarm/shed: rename prefix to metricsPrefix

* swarm/shed: unexport Meter, remove Mutex for quit channel

* swarm/storage: simplify ChunkValidator interface (ethereum#18285)

* usbwallet: check returned error when decoding hexstr (ethereum#18056)

* usbwallet: check returned error when decoding hexstr

* Update accounts/usbwallet/ledger.go

Co-Authored-By: CoreyLin <[email protected]>

* usbwallet: check hex decode error

* crypto/secp256k1: Fix invalid document link (ethereum#18297)

* accounts/abi: argument type and name were reversed (ethereum#17947)

argument type and name were reversed

* rpc: add application/json-rpc as accepted content type, fixes ethereum#18293 (ethereum#18310)

* Comment error (ethereum#18303)

* Change issue labels in bot configs to the new prefixed version

* core/state: rename 'new' variable (ethereum#18301)

* p2p/discv5: don't hash findnode target in lookup against table (ethereum#18309)

* .github: add @gballet as abi codeowner (ethereum#18306)

* fix slice unpack bug in accounts/abi (ethereum#18321)

* fix slice unpack bug in accounts/abi

* swarm/storage/feed: remove unused code (ethereum#18324)

* p2p/simulation: move connection methods from swarm/network/simulation (ethereum#18323)

* Update visualized snapshot test (ethereum#18286)

* swarm/network/stream: fix visualized_snapshot_sync_sim_test

* swarm/network/stream: updated visualized snapshot-test;data in p2p event

* swarm/network/stream: cleanup visualized snapshot sync test

* swarm/network/stream: re-enable t.Skip for visualized test

* swarm/network/stream: addressed PR comments

* swarm/network/simulation:commented out unreachable code-avoid vet errors (ethereum#18263)

* swarm/pss: Reduce input vulnerabilities (ethereum#18304)

* swarm/storage: remove unused methods from Chunk interface (ethereum#18283)

* downloader: fix edgecase where returned index is OOB for downloader (ethereum#18335)

* downloader: fix edgecase where returned index is OOB for downloader

* downloader: documentation

Co-Authored-By: holiman <[email protected]>

* core: sanitize more TxPoolConfig fields (ethereum#17210)

* core: sanitize more TxPoolConfig fields

* core: fix TestTransactionPendingMinimumAllowance

* p2p/simulation: Test snapshot correctness and minimal benchmark (ethereum#18287)

* p2p/simulation: WIP minimal snapshot test

* p2p/simulation: Add snapshot create, load and verify to snapshot test

* build: add test tag for tests

* p2p/simulations, build: Revert travis change, build test sym always

* p2p/simulations: Add comments, timeout check on additional events

* p2p/simulation: Add benchmark template for minimal peer protocol init

* p2p/simulations: Remove unused code

* p2p/simulation: Correct timer reset

* p2p/simulations: Put snapshot check events in buffer and call blocking

* p2p/simulations: TestSnapshot fail if Load function returns early

* p2p/simulations: TestSnapshot wait for all connections before returning

* p2p/simulation: Revert to before wait for snap load (5e75594)

* p2p/simulations: add "conns after load" subtest to TestSnapshot

and nudge

* swarm/pss: forwarding function refactoring (ethereum#18353)

* eth/downloader: progress in stateSync not used anymore (ethereum#17998)

* p2p/protocols: accounting metrics rpc (ethereum#18336)

* p2p/protocols: accounting metrics rpc added (ethereum#847)

* p2p/protocols: accounting api documentation added (ethereum#847)

* p2p/protocols: accounting api doc updated (ethereum#847)

* p2p/protocols: accounting api doc update (ethereum#847)

* p2p/protocols: accounting api doc update (ethereum#847)

* p2p/protocols: fix file is not gofmted

* fix lint error

* updated comments after review

* add account balance to rpc

* naming changed after review

* swarm/network: Revised depth and health for Kademlia (ethereum#18354)

* swarm/network: Revised depth calculation with tests

* swarm/network: WIP remove redundant "full" function

* swarm/network: WIP peerpot refactor

* swarm/network: Make test methods submethod of peerpot and embed kad

* swarm/network: Remove commented out code

* swarm/network: Rename health test functions

* swarm/network: Too many n's

* swarm/network: Change hive Healthy func to accept addresses

* swarm/network: Add Healthy proxy method for api in hive

* swarm/network: Skip failing test out of scope for PR

* swarm/network: Skip all tests dependent on SuggestPeers

* swarm/network: Remove commented code and useless kad Pof member

* swarm/network: Remove more unused code, add counter on depth test errors

* swarm/network: WIP Create Healthy assertion tests

* swarm/network: Roll back health related methods receiver change

* swarm/network: Hardwire network minproxbinsize in swarm sim

* swarm/network: Rework Health test to strict

Pending add test for saturation
And add test for as many as possible up to saturation

* swarm/network: Skip discovery tests (dependent on SuggestPeer)

* swarm/network: Remove useless minProxBinSize in stream

* swarm/network: Remove unnecessary testing.T param to assert health

* swarm/network: Implement t.Helper() in checkHealth

* swarm/network: Rename check back to assert now that we have helper magic

* swarm/network: Revert WaitTillHealthy change (deferred to nxt PR)

* swarm/network: Kademlia tests GotNN => ConnectNN

* swarm/network: Renames and comments

* swarm/network: Add comments

* accounts/abi: add support for unpacking returned bytesN arrays (ethereum#15242)

* accounts/abi: Brings out the msg defined at require statement in SC function (ethereum#17328)

* swarm: remove unused/dead code (ethereum#18351)

* fix string array unpack bug in accounts/abi (ethereum#18364)

* core/types: update incorrect comment

* accounts/abi: change unpacking of abi fields w/ underscores (ethereum#16513)

* accounts/abi: fix name styling when unpacking abi fields w/ underscores

ABI fields with underscores that are being unpacked
into structs expect structs with following form:

int_one -> Int_one

whereas in abigen the generated structs are camelcased

int_one -> IntOne

so updated the unpack method to expect camelcased structs as well.

* accounts/abi: fix case of generated java functions (ethereum#18372)

* rpc: Warn the user when the path name is too long for the Unix ipc endpoint (ethereum#18330)

* swarm/storage: change Proximity function and add TestProximity test (ethereum#18379)

* build: add LGPL license at update-license.go (ethereum#18377)

* add LGPL licence at update-licence.go

* add empty line

* travis, appveyor: bump to Go 1.11.4 (ethereum#18314)

* travis, appveyor: bump to Go 1.11.4

* internal/build: revert comment changes

* accounts/abi/bind: add optional block number for calls (ethereum#17942)

* vendor: vendor/github.com/mattn/go-isatty - add missing files (reported by mksully22) (ethereum#18376)

* swarm/docker: Dockerfile for swarm:edge docker image (ethereum#18386)

* vendor, crypto, swarm: switch over to upstream sha3 package

* vendor: update the entire golang.org/x/crypto dependency

* cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses (ethereum#18344)

* cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses

* cmd/puppeth: don't need to handle duplicate set

* eth/tracer: extend create2 (ethereum#18318)

* eth/tracer: extend create2

* eth/tracers: fix create2-flaw in prestate_tracer

* eth/tracers: fix test

* eth/tracers: update assets

* accounts/keystore: fix comment typo (ethereum#18395)

* A few minor code inspection fixes (ethereum#18393)

* swarm/network: fix code inspection problems

- typos
- redundant import alias

* p2p/simulations: fix code inspection problems

- typos
- unused function parameters
- redundant import alias
- code style issue: snake case

* swarm/network: fix unused method parameters inspections

* accounts/keystore: small code simplification (ethereum#18394)

* internal/ethapi: ask transaction pool for pending nonce (ethereum#15794)

* cmd/geth: support dumpconfig optionally saving to file (ethereum#18327)

* Changed dumpConfig function to optionally save to file

* Added O_TRUNC flag to file open and cleaned up code

* swarm: Shed Index and Uint64Field additions (ethereum#18398)

* swarm/storage/mock/test: fix T.Fatal inside a goroutine (ethereum#18399)

* swarm, p2p/protocols: Stream accounting (ethereum#18337)

* swarm: completed 1st phase of swap accounting

* swarm, p2p/protocols: added stream pricing

* swarm/network/stream: gofmt simplify stream.go

* swarm: fixed review comments

* swarm: used snapshots for swap tests

* swarm: custom retrieve for swap (less cascaded requests at any one time)

* swarm: addressed PR comments

* swarm: log output formatting

* swarm: removed parallelism in swap tests

* swarm: swap tests simplification

* swarm: removed swap_test.go

* swarm/network/stream: added prefix space for comments

* swarm/network/stream: unit test for prices

* swarm/network/stream: don't hardcode price

* swarm/network/stream: fixed invalid price check

* github: remove swarm github codeowners (ethereum#18412)

* swarm: Fix T.Fatal inside a goroutine in tests (ethereum#18409)

* swarm/storage: fix T.Fatal inside a goroutine

* swarm/network/simulation: fix T.Fatal inside a goroutine

* swarm/network/stream: fix T.Fatal inside a goroutine

* swarm/network/simulation: consistent failures in TestPeerEventsTimeout

* swarm/network/simulation: rename sendRunSignal to triggerSimulationRun

* swarm/network: remove isproxbin bool from kad.Each* iterfunc (ethereum#18239)

* swarm/network, swarm/pss: remove isproxbin bool from kad.Each* iterfunc

* swarm/network: restore comment and unskip snapshot sync tests

* accounts/abi: tuple support (ethereum#18406)

* swarm/network: Rename minproxbinsize, add as member of simulation (ethereum#18408)

* swarm/network: Rename minproxbinsize, add as member of simulation

* swarm/network: Deactivate WaitTillHealthy, unreliable pending suggestpeer

* accounts/abi: Extra slice tests (ethereum#18424)

Co-authored-by: weimumu <[email protected]>

* p2p/simulations: eliminate concept of pivot (ethereum#18426)

* core, eth: fix database version (ethereum#18429)

* core, eth: fix database version

* eth: polish error message

* Stream subscriptions (ethereum#18355)

* swarm/network: eachBin now starts at kaddepth for nn

* swarm/network: fix Kademlia.EachBin

* swarm/network: fix kademlia.EachBin

* swarm/network: correct EachBin implementation according to requirements

* swarm/network: less addresses simplified tests

* swarm: calc kad depth outside loop in EachBin test

* swarm/network: removed printResults

* swarm/network: cleanup imports

* swarm/network: remove kademlia.EachBin; fix RequestSubscriptions and add unit test

* swarm/network/stream: address PR comments

* swarm/network/stream: package-wide subscriptionFunc

* swarm/network/stream: refactor to kad.EachConn

* swarm/pot: pot.remove fixed (ethereum#18431)

* swarm/pot: refactored pot.remove(), updated comments

* swarm/pot: comments updated

* swarm/pot: each() functions refactored (ethereum#18452)

* accounts/abi: Add tests for reflection ahead of refactor (ethereum#18434)

* params: postpone Constantinople due to net SSTORE reentrancy

* params, swarm: release Geth v1.8.21 and Swarm v0.3.9

* core, cmd/puppeth: implement constantinople fix, disable EIP-1283 (ethereum#18486)

This PR adds a new fork which disables EIP-1283. Internally it's called Petersburg,
but the genesis/config field is ConstantinopleFix.

The block numbers are:

    7280000 for Constantinople on Mainnet
    7280000 for ConstantinopleFix on Mainnet
    4939394 for ConstantinopleFix on Ropsten
    9999999 for ConstantinopleFix on Rinkeby (real number decided later)

This PR also defaults to using the same ConstantinopleFix number as whatever
Constantinople is set to. That is, it will default to mainnet behaviour if ConstantinopleFix
is not set.This means that for private networks which have already transitioned
to Constantinople, this PR will break the network unless ConstantinopleFix is
explicitly set!

* travis, appveyor: bump to Go 1.11.5 (ethereum#18947)

* build: tweak debian source package build/upload options (ethereum#18962)

dput --passive should make repo pushes from Travis work again.
dput --no-upload-log works around an issue I had while uploading locally.

debuild -d says that debuild shouldn't check for build dependencies when
creating the source package. This option is needed to make builds work
in environments where the installed Go version doesn't match the
declared dependency in the source package.

* p2p/discover, p2p/enode: rework endpoint proof handling, packet logging (ethereum#18963)

This change resolves multiple issues around handling of endpoint proofs.
The proof is now done separately for each IP and completing the proof
requires a matching ping hash.

Also remove waitping because it's equivalent to sleep. waitping was
slightly more efficient, but that may cause issues with findnode if
packets are reordered and the remote end sees findnode before pong.

Logging of received packets was hitherto done after handling the packet,
which meant that sent replies were logged before the packet that
generated them. This change splits up packet handling into 'preverify'
and 'handle'. The error from 'preverify' is logged, but 'handle' happens
after the message is logged. This fixes the order. Packet logs now
contain the node ID.

* params: new CHTs (ethereum#18577)

* p2p/discover: improve table addition code (ethereum#18974)

This change clears up confusion around the two ways in which nodes
can be added to the table.

When a neighbors packet is received as a reply to findnode, the nodes
contained in the reply are added as 'seen' entries if sufficient space
is available.

When a ping is received and the endpoint verification has taken place,
the remote node is added as a 'verified' entry or moved to the front of
the bucket if present. This also updates the node's IP address and port
if they have changed.

* params, swarm/version: Geth 1.8.22-stable, Swarm 0.3.10-stable

* eth: make tracers respect pre- EIP 158/161 rule

* core: fix error in block iterator (ethereum#18986)

* cmd/puppeth: handle pre-set Petersburg number, save changed fork rules

* core: repro ethereum#18977

* core: fix pruner panic when importing low-diff-large-sidechain

* cmd, core, params: add support for Goerli

(cherry picked from commit b0ed083)

* build: use SFTP for launchpad uploads (ethereum#19037)

* build: use sftp for launchpad uploads

* .travis.yml: configure sftp export

* build: update CI docs

(cherry picked from commit 3de19c8)

* common/fdlimit: cap on MacOS file limits, fixes ethereum#18994 (ethereum#19035)

* common/fdlimit: cap on MacOS file limits, fixes ethereum#18994

* common/fdlimit: fix Maximum-check to respect OPEN_MAX

* common/fdlimit: return error if OPEN_MAX is exceeded in Raise()

* common/fdlimit: goimports

* common/fdlimit: check value after setting fdlimit

* common/fdlimit: make comment a bit more descriptive

* cmd/utils: make fdlimit happy path a bit cleaner

(cherry picked from commit f48da43)

* .travis.yml: fix upload destination (ethereum#19043)


(cherry picked from commit edf976e)

* build: avoid dput and upload with sftp directly (ethereum#19067)


(cherry picked from commit a8ddf7a)

* common/fdlimit: fix windows build (ethereum#19068)


(cherry picked from commit ba90a4a)

* vendor: update syscalls dependency

(cherry picked from commit dcc045f)

* vendor: pull in upstream syscall fixes for non-linux/arm64

(cherry picked from commit 9d3ea8d)

* trie: fix error in node decoding (ethereum#19111)

* vendor: update bigcache

(cherry picked from commit 37e5a90)

* travis.yml: add launchpad SSH public key (ethereum#19115)


(cherry picked from commit 75a9314)

* build: explicitly force .xz compression (old debuild picks gzip) (ethereum#19118)


(cherry picked from commit c0b9c76)

* swarm/network: fix data race in TestNetworkID test (ethereum#18460)

(cherry picked from commit 96c7c18)

* swarm/storage: fix mockNetFetcher data races (ethereum#18462)

fixes: ethersphere/swarm#1117
(cherry picked from commit f728837)

* cmd/swarm/swarm-snapshot: swarm snapshot generator (ethereum#18453)

* cmd/swarm/swarm-snapshot: add binary to create network snapshots

* cmd/swarm/swarm-snapshot: refactor and extend tests

* p2p/simulations: remove unused triggerChecks func and fix linter

* internal/cmdtest: raise the timeout for killing TestCmd

* cmd/swarm/swarm-snapshot: add more comments and other minor adjustments

* cmd/swarm/swarm-snapshot: remove redundant check in createSnapshot

* cmd/swarm/swarm-snapshot: change comment wording

* p2p/simulations: revert Simulation.Run from master

https://github.com/ethersphere/go-ethereum/pull/1077/files#r247078904

* cmd/swarm/swarm-snapshot: address pr comments

* swarm/network/simulations/discovery: removed snapshot write to file

* cmd/swarm/swarm-snapshot, swarm/network/simulations: removed redundant connection event check, fixed lint error

(cherry picked from commit 34f11e7)

* swarm/network: rewrite of peer suggestion engine, fix skipped tests (ethereum#18404)

* swarm/network: fix skipped tests related to suggestPeer

* swarm/network: rename depth to radius

* swarm/network: uncomment assertHealth and improve comments

* swarm/network: remove commented code

* swarm/network: kademlia suggestPeer algo correction

* swarm/network: kademlia suggest peer

 * simplify suggest Peer code
 * improve peer suggestion algo
 * add comments
 * kademlia testing improvements
   * assertHealth -> checkHealth (test helper)
   * testSuggestPeer -> checkSuggestPeer (test helper)
   * remove testSuggestPeerBug and TestKademliaCase

* swarm/network: kademlia suggestPeer cleanup, improved comments

* swarm/network: minor comment, discovery test default arg

(cherry picked from commit bcb2594)

* swarm/network: fix data race warning on TestBzzHandshakeLightNode (ethereum#18459)

(cherry picked from commit 81e26d5)

* Upload speed (ethereum#18442)

(cherry picked from commit 257bfff)

* cmd/swarm/swarm-snapshot: disable tests on windows (ethereum#18478)

(cherry picked from commit 632135c)

* cmd/swarm: use resetting timer to measure fetch time (ethereum#18474)

(cherry picked from commit a0b0db6)

* cmd/swarm/swarm-smoke: use ResettingTimer instead of Counters for times (ethereum#18479)

(cherry picked from commit 5609577)

* p2p/simulations: fix data race on swarm/network/simulations (ethereum#18464)

(cherry picked from commit 85a79b3)

* swarm: bootnode-mode, new bootnodes and no p2p package discovery (ethereum#18498)

(cherry picked from commit bbd1203)

* swarm: fix a data race on startTime (ethereum#18511)

(cherry picked from commit fa34429)

* swarm/version: commit version added (ethereum#18510)

(cherry picked from commit ad13d2d)

* p2p/testing, swarm: remove unused testing.T in protocol tester (ethereum#18500)

(cherry picked from commit 2abeb35)

* swarm/network: Remove extra random peer, connect test sanity, comments (ethereum#18964)

(cherry picked from commit f9401ae)

* swarm: fix flaky delivery tests (ethereum#18971)

(cherry picked from commit 592bf6a)

* swarm/storage: Get all chunk references for a given file (ethereum#19002)

(cherry picked from commit 3eff652)

* cmd/swarm: hashes command (ethereum#19008)

(cherry picked from commit 7f55b0c)

* cmd/swarm/swarm-smoke: sliding window test (ethereum#18967)

(cherry picked from commit b91bf08)

* cmd/swarm/swarm-smoke: remove wrong metrics (ethereum#18970)

(cherry picked from commit c5c9cef)

* cmd/swarm/swarm-smoke: refactor generateEndpoints (ethereum#19006)

(cherry picked from commit d212535)

* swarm: Debug API and HasChunks() API endpoint (ethereum#18980)

(cherry picked from commit 41597c2)

* swarm/storage: fix test timeout with -race by increasing mget timeout

(cherry picked from commit 1c3aa8d)

* swarm/pss: Remove pss service leak in test (ethereum#18992)

(cherry picked from commit 7c60d0a)

* swarm/storage/localstore: new localstore package (ethereum#19015)

(cherry picked from commit 4f3d22f)

* swarm/network: fix data race in fetcher_test.go (ethereum#18469)

(cherry picked from commit 19bfcbf)

* swarm/network, swarm/storage: Preserve opentracing contexts (ethereum#19022)

(cherry picked from commit 0c10d37)

* swarm/pss: transition to whisper v6 (ethereum#19023)

(cherry picked from commit cde02e0)

* swarm: GetPeerSubscriptions RPC (ethereum#18972)

(cherry picked from commit 43e1b7b)

* swarm/network: refactor simulation tests bootstrap (ethereum#18975)

(cherry picked from commit 597597e)

* cmd/swarm/global-store: global store cmd (ethereum#19014)

(cherry picked from commit 33d0a0e)

* swarm: CI race detector test adjustments (ethereum#19017)

(cherry picked from commit 27e3f96)

* swarm/docker: add global-store and split docker images (ethereum#19038)

(cherry picked from commit 6cb7d52)

* swarm/pss: mutex lifecycle fixed (ethereum#19045)

(cherry picked from commit b30109d)

* swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (ethereum#19028)

* swarm/storage: fix HashExplore concurrency bug ethersphere#1211

*  swarm/storage: lock as value not pointer

* swarm/storage: wait for  to complete

* swarm/storage: fix linter problems

* swarm/storage: append to nil slice

(cherry picked from commit 3d22a46)

* swarm: fix uptime gauge update goroutine leak by introducing cleanup functions (ethereum#19040)

(cherry picked from commit d596bea)

* swarm: fix network/stream data races (ethereum#19051)

* swarm/network/stream: newStreamerTester cleanup only if err is nil

* swarm/network/stream: raise newStreamerTester waitForPeers timeout

* swarm/network/stream: fix data races in GetPeerSubscriptions

* swarm/storage: prevent data race on LDBStore.batchesC

ethersphere/swarm#1198 (comment)

* swarm/network/stream: fix TestGetSubscriptionsRPC data race

ethersphere/swarm#1198 (comment)

* swarm/network/stream: correctly use Simulation.Run callback

ethersphere/swarm#1198 (comment)

* swarm/network: protect addrCountC in Kademlia.AddrCountC function

ethersphere/swarm#1198 (comment)

* p2p/simulations: fix a deadlock calling getRandomNode with lock

ethersphere/swarm#1198 (comment)

* swarm/network/stream: terminate disconnect goruotines in tests

* swarm/network/stream: reduce memory consumption when testing data races

* swarm/network/stream: add watchDisconnections helper function

* swarm/network/stream: add concurrent counter for tests

* swarm/network/stream: rename race/norace test files and use const

* swarm/network/stream: remove watchSim and its panic

* swarm/network/stream: pass context in watchDisconnections

* swarm/network/stream: add concurrent safe bool for watchDisconnections

* swarm/storage: fix LDBStore.batchesC data race by not closing it

(cherry picked from commit 3fd6db2)

* swarm/storage/netstore: add fetcher cancellation on shutdown (ethereum#19049)

swarm/network/stream: remove netstore internal wg
swarm/network/stream: run individual tests with t.Run

(cherry picked from commit 3ee09ba)

* swarm/network: Saturation check for healthy networks (ethereum#19071)

* swarm/network: new saturation for  implementation

* swarm/network: re-added saturation func in Kademlia as it is used elsewhere

* swarm/network: saturation with higher MinBinSize

* swarm/network: PeersPerBin with depth check

* swarm/network: edited tests to pass new saturated check

* swarm/network: minor fix saturated check

* swarm/network/simulations/discovery: fixed renamed RPC call

* swarm/network: renamed to isSaturated and returns bool

* swarm/network: early depth check

(cherry picked from commit 2af2472)

* swarm/storage: fix influxdb gc metrics report (ethereum#19102)

(cherry picked from commit 5b8ae78)

* swarm/pss: refactoring (ethereum#19110)

* swarm/pss: split pss and keystore

* swarm/pss: moved whisper to keystore

* swarm/pss: goimports fixed

(cherry picked from commit 12ca3b1)

* p2p, swarm: fix node up races by granular locking (ethereum#18976)

* swarm/network: DRY out repeated giga comment

I not necessarily agree with the way we wait for event propagation.
But I truly disagree with having duplicated giga comments.

* p2p/simulations: encapsulate Node.Up field so we avoid data races

The Node.Up field was accessed concurrently without "proper" locking.
There was a lock on Network and that was used sometimes to access
the  field. Other times the locking was missed and we had
a data race.

For example: ethereum#18464
The case above was solved, but there were still intermittent/hard to
reproduce races. So let's solve the issue permanently.

resolves: ethersphere/swarm#1146

* p2p/simulations: fix unmarshal of simulations.Node

Making Node.Up field private in 13292ee
broke TestHTTPNetwork and TestHTTPSnapshot. Because the default
UnmarshalJSON does not handle unexported fields.

Important: The fix is partial and not proper to my taste. But I cut
scope as I think the fix may require a change to the current
serialization format. New ticket:
ethersphere/swarm#1177

* p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON

* p2p/simulations: revert back to defer Unlock() pattern for Network

It's a good patten to call `defer Unlock()` right after `Lock()` so
(new) error cases won't miss to unlock. Let's get back to that pattern.

The patten was abandoned in 85a79b3,
while fixing a data race. That data race does not exist anymore,
since the Node.Up field got hidden behind its own lock.

* p2p/simulations: consistent naming for test providers Node.UnmarshalJSON

* p2p/simulations: remove JSON annotation from private fields of Node

As unexported fields are not serialized.

* p2p/simulations: fix deadlock in Network.GetRandomDownNode()

Problem: GetRandomDownNode() locks -> getDownNodeIDs() ->
GetNodes() tries to lock -> deadlock

On Network type, unexported functions must assume that `net.lock`
is already acquired and should not call exported functions which
might try to lock again.

* p2p/simulations: ensure method conformity for Network

Connect* methods were moved to p2p/simulations.Network from
swarm/network/simulation. However these new methods did not follow
the pattern of Network methods, i.e., all exported method locks
the whole Network either for read or write.

* p2p/simulations: fix deadlock during network shutdown

`TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock.
The execution was stuck on two locks, i.e, `Kademlia.lock` and
`p2p/simulations.Network.lock`. Usually the test got stuck once in each
20 executions with high confidence.

`Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in
`Network.Stop()`.

Solution: in `Network.Stop()` `net.lock` must be released before
calling `node.Stop()` as stopping a node (somehow - I did not find
the exact code path) causes `Network.InitConn()` to be called from
`Kademlia.SuggestPeer()` and that blocks on `net.lock`.

Related ticket: ethersphere/swarm#1223

* swarm/state: simplify if statement in DBStore.Put()

* p2p/simulations: remove faulty godoc from private function

The comment started with the wrong method name.

The method is simple and self explanatory. Also, it's private.
=> Let's just remove the comment.

(cherry picked from commit 50b872b)

* cmd/swarm/swarm-smoke: Trigger chunk debug on timeout (ethereum#19101)

* cmd/swarm/swarm-smoke: first version trigger has-chunks on timeout

* cmd/swarm/swarm-smoke: finalize trigger to chunk debug

* cmd/swarm/swarm-smoke: fixed httpEndpoint for trigger

* cmd/swarm/swarm-smoke: port

* cmd/swarm/swarm-smoke: ws not rpc

* cmd/swarm/swarm-smoke: added debug output

* cmd/swarm/swarm-smoke: addressed PR comments

* cmd/swarm/swarm-smoke: renamed track-timeout and track-chunks

(cherry picked from commit 62d7688)

* contracts/*: golint updates for this or self warning

(cherry picked from commit 53b823a)

* swarm: Reinstate Pss Protocol add call through swarm service (ethereum#19117)

* swarm: Reinstate Pss Protocol add call through swarm service

* swarm: Even less self

(cherry picked from commit d88c6ce)

* swarm/metrics: Send the accounting registry to InfluxDB (ethereum#18470)

(cherry picked from commit f28da4f)

* cmd/utils: allow for multiple influxdb tags (ethereum#18520)

This PR is replacing the metrics.influxdb.host.tag cmd-line flag with metrics.influxdb.tags - a comma-separated key/value tags, that are passed to the InfluxDB reporter, so that we can index measurements with multiple tags, and not just one host tag.

This will be useful for Swarm, where we want to index measurements not just with the host tag, but also with bzzkey and git commit version (for long-running deployments).

(cherry picked from commit 21acf0b)

* params, swarm: release Geth v1.8.23, Swarm v0.3.11

* light, params: update CHTs, integrate CHT for Goerli too

* common/fdlimit: fix macos file descriptors for Go 1.12

* all: simplify timestamps to uint64 (ethereum#19372)

* all: simplify timestamps to uint64

* tests: update definitions

* clef, faucet, mobile: leftover uint64 fixups

* ethash: fix tests

* graphql: update schema for timestamp

* ethash: remove unused variable

* trie: there's no point in retrieving the metaroot

* travis: update builders to xenial to shadow Go releases

* core: re-omit new log event when logs rebirth

* core: minor code polishes + rebase fixes

* params: set Rinkeby Petersburg fork block (4th May, 2019)

* p2p/discover: bump failure counter only if no nodes were provided (ethereum#19362)

This resolves a minor issue where neighbors responses containing less
than 16 nodes would bump the failure counter, removing the node. One
situation where this can happen is a private deployment where the total
number of extant nodes is less than 16.

Issue found by @Jsying.

* eth, les, geth: implement cli-configurable global gas cap for RPC calls (ethereum#19401)

* eth, les, geth: implement cli-configurable global gas cap for RPC calls

* graphql, ethapi: place gas cap in DoCall

* ethapi: reformat log message

* params, swarm: release Geth v1.8.24 (noop Swarm 0.3.12)

* cmd/geth: fix accidental --rpccorsdomain and --rpcvhosts removal

* params, swarm: hotfix Geth v1.8.25 release to restore rpc flags

* les: backported new SendTx cost calculation

* les: check required message types in cost table

* params, swarm: release Geth v1.8.26 (+noop Swarm v0.3.14)

* eth, les, light: enforce CHT checkpoints on fast-sync too

* params, swarm: release Geth v1.8.27 (noop Swarm v0.3.15)

* 1.8.27 with changes redone to compile

* fix compile

* runner update
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants