This repository has been archived by the owner on Dec 26, 2022. It is now read-only.
Implement Github Actions caching #170
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
inickles-grapl
approved these changes
Jul 24, 2020
…ity/grapl into internal-6-faster-gh-actions
jgrillo-grapl
added a commit
that referenced
this pull request
Jul 28, 2020
* Internal 9 fix workflow (#81) * trigger on prereleases as well * actions/checkout@v2 should checkout the correct branch anyway? * bash * fix tagged builds * add graphql-endpoint to build scripts * fix graphql endpoint Dockerfile * docker cp * zip arguments reversed >_< * Add generic subgraph generator to build * Build generic-subgraph-generator * Add Dockerfile for generic-subgraph-generator * Internal 9 fix workflow (#83) * trigger on prereleases as well * actions/checkout@v2 should checkout the correct branch anyway? * bash * fix tagged builds * add graphql-endpoint to build scripts * fix graphql endpoint Dockerfile * docker cp * zip arguments reversed >_< * python lambda.zips are located at /home/grapl * Internal 9 fix workflow (#84) * trigger on prereleases as well * actions/checkout@v2 should checkout the correct branch anyway? * bash * fix tagged builds * add graphql-endpoint to build scripts * fix graphql endpoint Dockerfile * docker cp * zip arguments reversed >_< * python lambda.zips are located at /home/grapl * use step outputs for artifact names * New CDK. * sneaky entry in parent .gitignore left the most important subdir out! * Add grapl-notebook * Restoring edge_ux - oops. * Clean and rebuild of the edge_ux, as well as fixing graphql endpoint string replacement. * prefer arrow function. * Bump service queue size from 1 to 10. * Fixing perms granted by `allowReadWrite` to also grant write perms. * Reduce console logs in schema.js * Community docs (#85) * initial attempt at community docs * document branching strategy * add link to Dobi issue * Add script to download release zips by tag. * Update to CDK 1.41.0. * Update deploy_all.sh to reflect new stack name for UX. * add grapl_analyzerlib to monorepo and fix rust containers * only rename the binaries in the lambdas * Add S3 encryption. Make fetch_Zips a bit more resilient. * Move JWT to secretsmanager * Remove unused import * Publish grapl_graph_descriptions and grapl_analyzerlib to PyPI (#91) * publish grapl_graph_descriptions and grapl_analyzerlib to PyPI * fix a few mistakes * remove some stuff that was erroneously brought in with grapl_analyzerlib * run mypy in lint workflow * fix a bug in the release workflow * Fix release workflow (#92) * publish grapl_graph_descriptions and grapl_analyzerlib to PyPI * fix a few mistakes * remove some stuff that was erroneously brought in with grapl_analyzerlib * run mypy in lint workflow * fix a bug in the release workflow * fix workflow syntax * whitespace * Fix the grapl-release workflow (#93) * publish grapl_graph_descriptions and grapl_analyzerlib to PyPI * fix a few mistakes * remove some stuff that was erroneously brought in with grapl_analyzerlib * run mypy in lint workflow * fix a bug in the release workflow * fix workflow syntax * whitespace * what? * whoops (#94) * Internal 18 fix release workflow (#95) * whoops * >_< * Fix pypa/gh-action-pypi-publish version (#96) * whoops * >_< * use correct version for pypa/[email protected] * Add rate limiting * Remove unused import * Valid emails for PyPI (#97) * valid emails for PyPI * update grapl-build and grapl-lint workflows to run on PR updates, run cargo update * run lints and builds on the staging branch * listen to synchronized events instead of edited * set the working-directory before attempting cargo audit * synchronize, not synchronized >_< * update working-directory * apparently working-directory does not work with uses * make cargo audit actually work * make cargo audit its own job s.t. it shows up as a separate check * attempt to fix the grapl-cargo-audit job * Internal 18 more release workflow fixes (#99) * valid emails for PyPI * update grapl-build and grapl-lint workflows to run on PR updates, run cargo update * run lints and builds on the staging branch * listen to synchronized events instead of edited * set the working-directory before attempting cargo audit * synchronize, not synchronized >_< * update working-directory * apparently working-directory does not work with uses * make cargo audit actually work * make cargo audit its own job s.t. it shows up as a separate check * attempt to fix the grapl-cargo-audit job * fix utility service builds * add another depends_on (#100) * Don't create a stack just for the secret. Use arn, not name. * Fixing path join in fetch script. * Remove engagement-graph from grapl_analyzerlib * Bump versions for grapl_analyzerlib * Use bump2version for python library releases (#101) * version bumps for python libraries * remove unreachable exceptions * python3 (#102) * Configure python correctly (#103) * python3 * configure python correctly * Check whether versions need to be bumped before building (#104) * python3 * configure python correctly * WIP so close * exclude yanked packages * some updates and fixes * Remove .bumpversion.cfg and increment versions (#105) * python3 * configure python correctly * WIP so close * exclude yanked packages * some updates and fixes * bump versions, remove .bumpversion.cfgs * Complete removal of EngagementGraph * Fix merge conflict * Add caching to Dockerfile * Black and rustfmt (#106) * black * rustfmt * add a newline to trigger git checks * cargo update * run black once last time * build in debian images instead of alpine (#108) * bump python library versions (#109) * build in debian images instead of alpine * bump python lib versions * add PyPI checks to lint workflow * - Adding version info as description to Lambdas. - Naming cleanup. - Pinning dgraph version. This should have been in a previous commit. * DGraph - don't use standalone version in AWS. * Fix rust builds (#110) * build in debian images instead of alpine * bump python lib versions * add PyPI checks to lint workflow * try to reduce peak disk usage for 10GB Github Actions limit * Removing a file created from merge that shouldn't have made it in previous commit. * Move docs * - Cleanup SageMaker Notebook name. * - Remove encryption on UX bucket - this needs to be public. * build in debian images instead of alpine (#108) * bump python library versions (#109) * build in debian images instead of alpine * bump python lib versions * add PyPI checks to lint workflow * Fix rust builds (#110) * build in debian images instead of alpine * bump python lib versions * add PyPI checks to lint workflow * try to reduce peak disk usage for 10GB Github Actions limit * Renaming and syncing queue names in cdk with rest of the codebase. * Fixing Python builds to work in AWS. * Python and Rust formatting. * Fix LGTM alert, unused import. * Update sqs-lambda * Add waiting for s3 and sqs * Add aws_region to events * Add ip_address to OutboundConnection * - Adding version info as description to Lambdas. - Naming cleanup. - Pinning dgraph version. This should have been in a previous commit. * DGraph - don't use standalone version in AWS. * Removing a file created from merge that shouldn't have made it in previous commit. * - Cleanup SageMaker Notebook name. * - Remove encryption on UX bucket - this needs to be public. * Renaming and syncing queue names in cdk with rest of the codebase. * Fixing Python builds to work in AWS. * Python and Rust formatting. * Fix LGTM alert, unused import. * Remove engagement graph from model plugin deployer. * Bump graph generator lib version * Bump graph generator lib version * Move grapl-config and grapl-graph-descriptions to crates.io * Add license and description * Add license and description * Update Cargo.toml * Update Cargo.toml * Slip in a change to fix logging * Update dependencies * Syncing queue names. * Fix unused import reported by LGTM. * Attach reverse edges in DGraph (#114) * implement reverse edges * some bug fixes * remove unused local * remove some more engagement graph stuff * add some debug logging * Set up reverse edges, remove types * fix logging * put EngagementClient and EngagementView back in * update for review comments Co-authored-by: colin-grapl <[email protected]> Co-authored-by: colin <[email protected]> * Adding ability for generator lib s3 client to assume AWS STS role. * internal-13-ui-updates (#118) * fixed compiler warnings * Start dashboard * completed dashboard, page not found, extracted header, & routing * Add file handler * able to upload & delete plugins with UX changes * reformatted using black * lgtm changes * Remove hanging reference to engagement_graph * update version, fix encoding error Co-authored-by: colin <[email protected]> * Implement DGraph TTL cleanup job (#119) * basic ttl job structure * small fix * small fixes * code cleanup * write batch query * updates for review comments * WIP but close now * some updates from local testing * updates from local testing * make debug logs a little less spammy * add grapl-dgraph-ttl to local grapl * some more updates from local testing * CDK for grapl-dgraph-ttl (#121) * add --delay and --batch-size options to upload-sysmon-logs.py * attempt to CDK * add schedule rule * updates for review comments * address LGTM lint * cdk README * add extract-grapl-deployment-artifacts.sh * WIP build is broken * fix aws-cdk versions * some updates from local testing * WIP build still borked * README update * make dgraph-ttl a Construct instead of a NestedStack * fix an issue with the handler reference * some more updates from testing in sandbox * more fixes * normalize dgraph dns name handling (#126) * Upgrade aws cdk to 1.46 to take advantage of fix for aws/aws-cdk#3463 * - Adding deploy name to support deploying multiple instances of Grapl to a single AWS account. - All Constructs now use an interface for constructor params beyond scope and id. This essentially provides named params, making it clearer what the arguments are for. * Remove .env from .gitignore, env vars no longer used. * Fix Grapl prefix from stackName issue. * Fix EngagementUX Stack name. * unused imports cleanup. * Fix test code from template to new Grapl Stack params. * Fixing some hard-coded names to take a prefix. * Fix name for retry handler. * Fixing hard-coded table names, which now take from env vars. Also gonna mix up our naming convention. For fun. * jkz colin * Add asset_id_mappings table and remove node_id_retry_table. * Add new env vars to docker-compose.yml * Remove swp file that should not be there. * Moving dynamodb table name fetches for node-identifier to grapl-config. * Adding MG_ALPHAS to docker-compose for dgraph-ttl. * Integrate cdk-watchful (#124) * whitespace * add cdk-watchful dep * vendor cdk-watchful 0.5.1 * get vendored cdk-watchful to build * add watchful to the Grapl stack * some fixes * remove cdk-watchful DynamoDB stuff because we use on-demand pricing * remove unused import * instrument engagement and graphql * updates from local testing * make everything a NestedStack * internal-74 fix destroy_all.sh * remove dangerous S3 stuff * Updating CDK README to match changes from previous PR. * Routing (#131) * added paging for graphQL endpoint * added front-end pagination to plugins * graphQL paging * fixed duplicate pagination bug for lenses * added styling for paging * formatting * Revert extra lenses * Styling * fixed any types * revisions * refactored custom routing to use react-router-dom HashRouter * remove redirectTo function * added logic to check if logged in * removed unneeded props * Improve performance of plugin page and fix warnings * Update sqs-lambda * Update sqs-lambda * Bump versions for tokio, tokio-compat, other dependencies * Making the use of Watchful conditional on spec of email for watchful alerts. * Let's not forget to pass this new param to the Grapl Stack. * Update graph-generator-lib * Glob deps * Properly pass in UX Bucket URL to graphql endpoint * Fixing previous watcher commit. * Move ux bucket to grapl-cdk-stack * Fix import * Fix .gitignore in CDK to not ignore the JS in our UX. * modelPluginDeployer cdk hack * Lowercase the origin in engagement edge * Reformat * Fix endpoint URL * Use regional bucket domain with https * Fix some more issues related to cors, env vars, etc * Format engagement_edge * Only use one client per service * Reuse asset id db * Formatting via Prettier. Adding .prettierrc.toml. * Fixing bucket policies, which had a few issues: - ActionGetObject isn't a thing - resources for s3:GebObject action need path wildcard - extra perms in TODO note * no sids. * policy statement cleanup. * Create the Lmabda execution role so we can name it. Also rename publishesToBucket to writesToBucket. * Adding explicit lambda execution roles with leigble names to ModelPluginDeployer and DGraphTTL. * Add description text to CDK stacks. * Use dobi for builds (#146) * WIP * tag build images latest * replace docker-compose build with dobi * mark dobi-linux binary executable * split builds into separate jobs * small fix for release workflow * RefactorBugfix * Fixes a number of issues with localgrapl, and improves logging * Temporary fix for cors in python services. grapl_analyzerlib plugin retriever fix. Notebook ACL fix. * Fix acl for list of s3 * Improve error in dispatcher. Fix ACL * Use addResources * Fix a couple things: - pass new arg - change params to match scheme of others (extending Props) * Don't unwrap a failed DGraph upsert * Discard transaction on failure * Remove commented out code * Reformat python * fix py formatting issues. * Reformat python * Reformat python * Handle redeployment of plugins more gracefully * Update Dockerfile to cd before zip * It's a - not a _ * Fix underscores * Give Engagement Notebook role a ligble name and a description. * Fix a number of issues with the analyzer-executor * Revert prefix * Bump the grapl_analyzerlib VERSION * Add paging to get_lenses (#125) * added paging for graphQL endpoint * added front-end pagination to plugins * graphQL paging * fixed duplicate pagination bug for lenses * added styling for paging * formatting * Revert extra lenses * Styling * fixed any types * revisions * Reload page when login is successful and history changes. (#148) * added location.reload() when history changes * removed console.logs * removed console.log * Internal 135 error handling (#164) * added validation * added extra rows to plugin listing to prevent UI from jumping * added yup package * add back local_handlers for graph-merger and analyzer-dispatcher * staging changes * Tests in CI (#153) * move assetdb and sessiondb tests under integration feature flag * run rust tests in CI * fix dobi configs for rust deploy images * run python unit tests in builds * run tests in js builds * fix js tests * WIP tests run but they fail * WIP * python integration tests pass! * make rust integration tests run * make rust integration tests work and wire tests into github workflows * workflow improvements * use cached build artifacts in integration tests * remove bogus hypothesis examples * Internal 37 fix workflows (#168) * move assetdb and sessiondb tests under integration feature flag * run rust tests in CI * fix dobi configs for rust deploy images * run python unit tests in builds * run tests in js builds * fix js tests * WIP tests run but they fail * WIP * python integration tests pass! * make rust integration tests run * make rust integration tests work and wire tests into github workflows * workflow improvements * use cached build artifacts in integration tests * remove bogus hypothesis examples * fix failing release build * fix build * some fixes (#171) * some fixes * 😭 * Fix debug builds * Remove test build step * another release workflow fix (#175) * Internal 82 local auth (#173) * fixed bugs with local secretsmanager * fixed dashboard buttons, bug fixes for local auth * added UX_BUCKET_URL * added missing parameter to GraphQLEndpoint for ux_bucket per Colin * added dynamodb tables & service to graplprovision for local-grapl-user_auth * removed links to documentation * added welcome back * debugging for local auth * removed print statement * check for print statement in graph-merger * merge with staging * cargo formatting * python formatting * run black for python formatting * PR comment edits * update documentation with localgrapl creds, python formatting * reformatting * fml (#177) * Internal 37 moar workflow fixes (#178) * fml * another fix * Bump sqs-lambda to 0.20.20 * Internal 172 canvas bug (#180) * graphql query fixes * extended DGraph TTL to 13 months * remove console.log * cargo update * revert cargo update. * Implement Github Actions caching (#170) * WIP -- begin implementing gh actions caching * WIP * WIP * run unit tests in dobi * WIP workflows * update workflows for new jobs * add caching to build workflow * fix workflow * yaml syntax * fix build workflow * chicken or egg? * copy pasta * apparently artifacts are per-workflow * remove upload/download * fix build workflow * another fix Co-authored-by: inickles-grapl <[email protected]> * Fix release workflow (#182) * WIP -- begin implementing gh actions caching * WIP * WIP * run unit tests in dobi * WIP workflows * update workflows for new jobs * add caching to build workflow * fix workflow * yaml syntax * fix build workflow * chicken or egg? * copy pasta * apparently artifacts are per-workflow * remove upload/download * fix build workflow * another fix * release workflow updates Co-authored-by: inickles-grapl <[email protected]> * fix release flow (#185) * fix release flow * wat * >_< * Fix merge conflict (#187) * Update grapl_analyzerlib * Create CODE_OF_CONDUCT.md * version bumps * add newline for teh lulz Co-authored-by: colin <[email protected]> Co-authored-by: andrea-grapl <[email protected]> Co-authored-by: colin-grapl <[email protected]> * add check-pypi job to build workflow (#188) * V0.2.0 docs update (#189) * update CONTRIBUTING.md for v0.2.0 release * README.md reformat and some updates * add Grapl DFIR Slack invite link to CONTRIBUTING.md * move check-pypi job to the lint workflow * two more README.md updates Co-authored-by: colin <[email protected]> Co-authored-by: inickles-grapl <[email protected]> Co-authored-by: colin-grapl <[email protected]> Co-authored-by: inickles-grapl <[email protected]> Co-authored-by: andrea-grapl <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make builds faster by caching intermediate artifacts.