-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: streaming terraform logs in real-time #1937
feat: streaming terraform logs in real-time #1937
Conversation
* Current progress * Websocket implementation * Fixing routes * Adding url * Fixed websocket connection * Progress 7/9 * UI Fixes * Fixed test * Fixing failed tests * Update server/controllers/templates/web_templates.go Co-authored-by: Sarvar Muminov <[email protected]> * Update server/events/models/models.go Co-authored-by: Sarvar Muminov <[email protected]> Co-authored-by: Isata Sankoh <[email protected]> Co-authored-by: Isata Sankoh <[email protected]>
* Current progress * Websocket implementation * Fixing routes * Adding url * Fixed websocket connection * Progress 7/9 * Added some channel logic * Changes to models.go * Working on log streaming for plan * Beginning tests/Having errors * Log Streaming Logic * fixed test * Added Testing * Fixed websocket connection * Async implementation and test error fixes * Changes to variables in tests * Debugging * Fixed UI in webpage terminal * Fixing Run make-test coverage * Fixing tests * Testing * Deleted txt.act files * Fixing broken tests * Changes to terraform_client testing * Fixed tests in terraform_client_internal_test.go * Fixed failing test * Deleted lines causing test to fail * Reformating * More Reformatting * Run make check-lint corrections * Added error check * Fixing check-lint test * Suggested changes prior to merge Co-authored-by: Isata Sankoh <[email protected]>
This reverts commit a9e1712.
* Began channel refactoring * Fixing refactoring logic * Replacing terraformOutputChan * Making suggested changes * Changing attribute declarations * Fixed channel logic * Formatting * Changing tests to utilize mocks * Adding go files * Adding suggested changes * Suggested changes before merging * Reformatting * Deleting unnecesary lines * Buffer cleanup logic * Buffer cleanup & Testing * Trying to make test pass * Suggested changes * Fixing test * Fixing failing test * Error checking * Comment improvements * Adding buffer clearing for apply workflow * Removing clearing for apply workflow
* Adding URLs to PR * Changes for rebasing * Minor changes in comments * Removing unnecessary comments * Fixing bug in PR * Minor changes in comments * Removing unnecessary comments * Removing extra character * Allocating comments created in PR * Adding feature allocator to structs
* renaming log-streaming to jobs * Create a status per running project * Add SetJobUrlWithStatus to ProjectCommandOutputHandler It implements updating the PR with a new project status and adds URL to a project jobs page where customer can view the progress of their terraform command execution in realtime. * Added feature allocator into the project command output handler * Remove PullRequestJobs objects and structs * remove print statement * remove print * Adding feature allocator mocks to tests * fix tests * Fixing linting * fix linting * Add decorator with feature allocator * use different variable name * Moved logic into a decorator * fix e2e test * Address PR comments
* Do not display non essential tf commands * Add title with correct tabbing * fix ui issues * more fixes
* So for whatever reason tf show with big plan doesn't work in async * rename isValidCommand to isAsyncEligibleCommand
A number of things are going on here: Removed the feature aware related items for log streaming work Created websocket package which handles writing to individual websockets Created Multiplexor which primarily should be all the controller interacts with and moved all primary logic there Created PartitionKeyGenerator for pulling the projectinfo which makes it generic. Modified Receive/removeChan -> Register/Deregister to remove the reading logic from the outputhandler since that struct has way too many responsibilities right now.
@Aayyush it is safe to delete all |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did a first pass looks good, its worth calling out that we've been running this in production for a few months so the only risk with a large PR like this is maybe some weirdness with merging or with workflows that others in the community use that we don't.
…-test-job * 'master' of github.com:runatlantis/atlantis: (79 commits) release: 0.18.0 (#1965) feat: streaming terraform logs in real-time (#1937) build(deps): bump github.com/hashicorp/go-getter from 1.5.9 to 1.5.10 (#1961) docs: update website links (#1964) docs: clarify example for `--azuredevops-token` flag (#1712) docs: typo in heading level (#1960) fix: fallback to default TF version in apply step (#1931) feat: add GitHub team allowlist configuration option (#1694) Add in Dockerfile support for last Terraform 1.0.x version in AVAILABLE_TERRAFORM_VERSIONS (#1957) build(deps): bump github.com/spf13/viper from 1.10.0 to 1.10.1 (#1956) deps: terraform 1.1.2 (#1952) release: 0.17.6 (#1947) docker: make multi-platform atlantis image (#1943) docker(testing): fix arch ref for `linux/arm/v7` docker(testing): updating image build process docs: fix policy check documentation examples (#1945) build: make multi-platform image for testing-env atlantis-base: fix context atalntis-base: update platforms and path trigger atlantis-base: update platforms ...
Hello, if i might ask a question - does this feature support (or will) custom workflows? Currently I am using terragrunt as terraform wrapper |
Ideally yes we would want to support streaming all commands this was just the first step to get it out there. |
@nishkrishnan I would be interested to try and see If I can contribute - if possible. Thank you very much, cheers! |
@juozasget I opened #2054, feel free to contribute there. |
This feature allows users to view their terraform logs in real time when running atlantis plan/apply commands. This implementation uses websockets to stream the logs from the terraform client to the user's web browser.
How to use it?
This is inspired from @raxod502-plaid initial implementation outlined in this PR: #1315
Contributors: @isatasan @nishkrishnan @msarvar @Aayyush