This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Add greylist limit - develop #8044
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…othing). Clean up usage of the max virtual multiplier throughout the code by using one constant defined in config.hpp. Enhance the resource_limits_test/sanity_check test.
This was referenced Oct 9, 2019
heifner
suggested changes
Oct 9, 2019
arhag
added a commit
that referenced
this pull request
Oct 10, 2019
arhag
added a commit
that referenced
this pull request
Oct 10, 2019
heifner
approved these changes
Oct 10, 2019
roman-tik
added a commit
to Remmeauth/remprotocol
that referenced
this pull request
Dec 28, 2019
* add some paranoia around GS usage * update version to 2.0.0-develop * clean up error checking throughout the tierup processes * executor needs to check its inital map of code * be more explict about testing the alignment of code/data segments * make sure EOS-VM OC is given at least 1 thread * don't leak memfd if memory fails to be ctored * coalesce executor's per-action cleanup lambda * be more aggressive on evicting cache when compiler monitor indicates too full * bump the code cache header to force invalidation of previous caches this is not the intended way of deprecating old generated code -- that would be the codegen_version field. But this is a quick way to zap any code caches that were created by versions of EOS-VM OC before now. They could have had leaks/problems * disallow WAVM with EOS-VM OC not compatible * Increase stability of test by spreading push to all nodes * Revert to old behavior of appending subjective failure to allow possibily cheaper trx in * Change log level of index write from dlog to ilog * Fixed block reporting to work for trimmed block logs. * Moved trim_data into block_log. GH EOSIO#7939 * Adding missed invarients to existing code. GH EOSIO#7939 * Movede trim_blocklog_front logic into block_log and added support for version 3. GH EOSIO#7939 * Added integration tests for trimming front of block log. GH EOSIO#7939 * ensure eos-vm-oc headers get installed * Added comments and cleanup. GH EOSIO#7939 * Retain persisted trx until expired on speculative nodes * get the linker order right for eosio tester cmakes * when linking with a libchain containing EOS-VM OC, need to wrap main * &> /dev/null -> 1> /dev/null so we can see errors * Replaced cached trim_data variables with methods that determine those values. GH EOSIO#7939 * When using the `set_proposed_producers_ex` intrinsic with the new format (`packed_producer_format == 1`) avoid calling `valid()` on each of the keys. * Added passing in the temp directory for temporary storage of new files and eventual storage of old versions. GH EOSIO#7939 * Fixed variable name. GH EOSIO#7939 * Need to ensure temp directory exists. GH EOSIO#7939 * Change default subjective cpu leeway to 9000us from 3000us and make it configurable * Add optional ability to disable WASM Spec Tests * Change assert that always failed to EOS_THROW. GH EOSIO#7939 * Update subjective-cpu-leeway-us description and set default_subjective_cpu_leeway_us to 31ms * Fix build script LLVM symlinking For unpinned builds the build script symlinks the system LLVM7 directory to LLVM_ROOT. But the command only successfully works the first time. Fix that. * Fixed skip logic and added assert to verify at or past first block when searching for a block number location. GH EOSIO#7939 * update EOS VM Optimized Compiler naming convention * Fixed logic for reverse reading the block log for truncation. GH EOSIO#7939 * Fixed code so last_block is set before we call block_pos ... GH EOSIO#7939 which validates that n is less than or equal to last_block + 1. * update eos-vm ref and install eos-vm license * Fixed trimmed file to write buffer in reverse order. GH EOSIO#7939 and other fixes * Fixed comment. GH EOSIO#7939 * install license * update ref to temporary branch with license fixes * Distinguish between block not found and on fork to give a better error message. Also allow reconnect for benign_other go_away which is what we now send for unknown block * EOS VM OC license updates * fix up markup that leaked in to license file * bump version to v2.0.0-rc1 * eosio/producer -> eosio/ci * Add global-greylist-limit option to producer_plugin (currently does nothing). Clean up usage of the max virtual multiplier throughout the code by using one constant defined in config.hpp. Enhance the resource_limits_test/sanity_check test. * Support global-greylist-limit * add whitelist_blacklist_tests/global_greylist_tests test * don't rebuild llvm unnecessarily during pinned builds * address @heifner's review comments on EOSIO#8044 * remove unnecessary reflection of controller::config * EOSIO#8054: fix commas in ship abi * Update to support new producer schedule * Updated to use new version library for version and full-version reporting * Add server_full_version_string to get info * Update to v2.0 of appbase which includes set version * Remove read delay timer and just disconnect when queue is full. Also drop trx when too many in progress. * Fix variable shadowing * Reduce some error level logging to info as they are normal responses to client requests when using snaphot. Also send benign_other in corner case of unable to find block. * 10.14.4 -> 10.14.6 * Handle cases where version_* not specified * Require version_minor along with version_major * Add separate build/test fleet. * Pass correct number of JOBS to unit test step. * Added support for reporting extensions_type parameters in RPC and eosio-blocklog json output. For release/2.0.x * bcast block even with syncing with peer as there can be race condition around determination * Modify --print-default-config to exit with success * Bump commit. * Split WASM spec tests to own label and steps. * mac steps get a min of 1 hour * Add debug log of sync vars in request_next_chunck * Add debug logging * Call sync_recv_block for already received block since accounting of blocks is required * added new build step & comments to help describe what variables are available for logic creation * added new build step & comments to help describe what variables are available for logic creation * ensure script name change * when building pinned boost on macOS, make sure SDKROOT is set * Move platforms into distinct folders to support future platform types. * Add support for platform directory. * Do not start mongod if it does not exist. * Add support for platform type directory. * Fix missing LLVM. * Fix missing LLVM. * Fix quotes. * Post callback as medium priority to prevent ship running at maximum priority * Be careful to capture lambda values by value since execution happens later via app post * add harden flags to cicd & pinned builds * Fix for unpinned env and triggered wasm spec tests. * ensure GMP is always dynamically linked * Remove signs of UNPINNED in favor of only supporting one env. * Remove signs of UNPINNED in favor of only supporting one env. * 10 second sleep * Change to debug level log output * Consolidated Security Fixes for 2.0.x * bump version to 2.0.0-rc2 * Fixed test to not count failed attempts as actual sends and added some leeway in validation. * Rename sync_state struct to peer_sync_state to distinguish from sync_state enum * Fix self check to compare to this node's node_id * Remove dead code * Avoid wrapping generation * sync_recv_block should only be called if in lib catchup * Add full version to startup ilog message if diff than version * Add additional logging * Don't send handshakes to everyone as often as it just causes confusion. Also don't go into syncing mode for head catchup. * return more from producer_plugin's get_runtime_options() * Only send handshake if it has changed from last send. Log all incoming blocks. * chainbase sync to pick up DB shrink fix while in heap mode * If block less than lib received outside of lib catchup then tell peer to stop sending * Make sure handshake is first response to first handshake from peer. Add extra verification in blk_send_branch that we have valid handshake from peer. * Send head id along with verify request_message so if not on a fork the peer can send from head instead of from lib * explictly use openssl 1.1 via brew on macos * need to set brew bottle to [email protected] too * If verifying catchup and num is notice is less than lib, nothing to do. * remove OPENSSL_ROOT_DIR from build script -- not needed * Reduce received block and bcast block logging to debug level * Enable debug level logging for net_plugin, http_plugin, and producer_plugin for launcher logging config. debug level used to be enabled by default for net_plugin and http_plugin for the tests. * Add const and extra logging * Additional logging for when block is not applied to head * If peer sending less than lib and we have notified it of our lib, then close connection * Only log block not applied to head if not running in irreversible mode * Explicitly guard agaisnt null block_state->block * Updated C7 version. * Only cancel wait if not syncing * send handshake if head_id changed * Fix compile error * Updated C7 version. * Always cancel_wait on receive * on fork should be true if not found * Do not send a go away message if unable to fetch a block as we might have just switched to a diff fork * Fix for incorrect openssl and timeouts in Travis. * Add back in sending handshakes when becoming in_sync since it is possible for client to fall behind while catching up * Send handshake instead of go away message as client needs to know we have not been able to honor request * Add UTC to print * switch from strand dispatch to strand post. add extra checks for correct socket. * Add extra temp logging * Always clear out queue on async write callback otherwise we always believe we are still in the process of sending and will never send anything else. * Add clear of out queu on connect to match 1.8.x. Remove unused priority arguement. Remove temp logging. * known_blocks.pending is head NOT known_trx.pending * If requesting next chuck and determine nothing to request, send handshake * Use error code to avoid exceptions from boost asio * Add thread protection around peer logging variables * reset the new handler * uncomment, hopefully should work * Fixed corner case for failures and added debug logging. * Increased time to wait for shutdown. * Cleaned up log statements. * Fix race in fc::message_buffer and move message_buffer_tests to fc. * Update controller.cpp * Update controller.cpp * Do less work when closing during plugin shutdown to speed up shutdown * Allow tags to create Contract Builders. * Allow retries on passed steps. * Cumulative security fixes for 2.0 rc3 * increase tester state size from 8 MiB to 16 MiB * modifications to trigger LRTs and Multiver on anything but scheduled * update submod * update submod * bump version to 2.0.0-rc3 * Update compile_trampoline.cpp * update submodules * change version and project name in CMake * Update CMakeLists.txt * Update CMakeLists.txt * Update CMakeLists.txt * Update CMakeLists.txt * Update CMakeLists.txt Co-authored-by: Bucky Kittinger <[email protected]> Co-authored-by: Matt Witherspoon <[email protected]> Co-authored-by: arhag <[email protected]> Co-authored-by: Kevin Heifner <[email protected]> Co-authored-by: Brian Johnson <[email protected]> Co-authored-by: Nathan Pierce <[email protected]> Co-authored-by: Jeffrey Smith II <[email protected]> Co-authored-by: Todd Fleming <[email protected]> Co-authored-by: Scott Arnette <[email protected]> Co-authored-by: swatanabe-b1 <[email protected]> Co-authored-by: Roman Cherednik <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Description
This PR adds support for a greylist that applies to all accounts and makes the accounts' resource limits behave (as far as deciding on whether to subjectively reject the transaction) as if the elastic resource multiplier (which can vary from 1 to 1000 according to dynamics that are part of consensus) is limited to a locally configured ceiling. The value for this limit can be specified through the new
greylist-limitconfiguration option in the producer plugin. It can also be updated during runtime by specifying the new optional fieldgreylist_limitwithin theruntime_optionsprovided to the existingupdate_runtime_optionsRPC of the producer API.Consider a transaction that is billed to an account that is not on any explicit greylist and that hits a timeout deadline (or NET limit) driven by the quotas of the billed account (rather than other limits). When this greylist limit is not the limiting factor (e.g. the
greylist-limitis set to 500 but the network is used to the point where the multiplier is at 200), the system will continue to throw the usual exception:tx_cpu_usage_exceeded(for hitting the CPU limit) ortx_net_usage_exceeded(for hitting the NET limit). But when the greylist limit is the limiting factor, the greylist versions of the exceptions will be thrown instead:greylist_cpu_usage_exceededorgreylist_net_usage_exceeded. In the case of hitting NET limits, this can be the difference between the subjective exceptiongreylist_net_usage_exceeded(which would, for example, cause a deferred transaction to be ignored and not retired) and the objective exceptiontx_net_usage_exceeded(which could cause a deferred transaction to retire withsoft_failorhard_fail).This effectively acts as a subjective means of capping the maximum resource multiplier to a value lower than 1000. It is not possible to change that maximum value of 1000 in an objective way without a change to the consensus protocol rules, which this PR does not attempt to do.
This PR also:
eosio::chain::config::maximum_elastic_resource_multiplierconstant rather than hardcoding the number 1000 in various places through the code;resource_limits_test/sanity_checktest;whitelist_blacklist_tests/greylist_limit_teststest which tests the new greylist feature.Consensus Changes
API Changes
Added new configuration option
greylist-limitto the producer plugin which is an integer ranging from 1 to 1000 that determines the limit for the new greylist feature. The maximum value of 1000 (which is also the default) means that greylist is not the limiting factor and therefore basically has no effect.Added a new optional integer field
greylist_limitto theoptionsargument of theupdate_runtime_optionsRPC of the producer API which allows changing the greylist limit during runtime.Documentation Additions