package manager for vendor pull #768
Conversation
…uce package types
… package handling
We need a fallback for headless terminals. This is probably handled by charmbracelet as well. We might just not be using it. |
📝 Walkthrough📝 WalkthroughWalkthroughThis pull request introduces a comprehensive enhancement to the Atmos vendoring system, focusing on improving the Changes
Assessment against linked issues
Possibly related PRs
Suggested reviewers
📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (2)
🧰 Additional context used📓 Learnings (2)📓 Common learningsexamples/quick-start-advanced/Dockerfile (2)🔇 Additional comments (2)examples/quick-start-advanced/Dockerfile (1)
The update of website/docs/integrations/atlantis.mdx (1)
The Atmos version update in the GitHub Actions example maintains consistency with the version specified in the Dockerfile. Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Outside diff range and nitpick comments (9)
internal/exec/vendor_model.go (1)
148-148: Add missing space in completion messageThere's a missing space after the period in the string "Vendored %d components.Failed to vendor %d components.\n", which causes the message to read incorrectly. Adding a space will improve readability.
Apply this diff to fix the formatting:
return doneStyle.Render(fmt.Sprintf("Vendored %d components.Failed to vendor %d components.\n", n-m.failedPkg, m.failedPkg)) +return doneStyle.Render(fmt.Sprintf("Vendored %d components. Failed to vendor %d components.\n", n-m.failedPkg, m.failedPkg))internal/exec/vendor_model_component.go (5)
23-34: ExportpkgComponentVendorstructIf there's a possibility of using
pkgComponentVendoroutside this package in the future, consider exporting it by capitalizing the struct name. Adding comments to structs and fields will also enhance code readability.
135-137: Add missing space in status messageThere's a missing space after the period in the status message. This will improve the readability of the output.
-return doneStyle.Render(fmt.Sprintf("Vendored %d components.Failed to vendor %d components.\n", n-m.failedPkg, m.failedPkg)) +return doneStyle.Render(fmt.Sprintf("Vendored %d components. Failed to vendor %d components.\n", n-m.failedPkg, m.failedPkg))
166-199: Refactor to eliminate duplicate code indownloadComponentAndInstallThe function
downloadComponentAndInstallcontains similar code blocks for handling components and mixins. Refactoring can reduce duplication and simplify maintenance.Here's a possible refactor:
func downloadComponentAndInstall(p pkgComponentVendor, dryRun bool, cliConfig schema.CliConfiguration) tea.Cmd { return func() tea.Msg { if dryRun { time.Sleep(100 * time.Millisecond) return installedPkgMsg{ err: nil, name: p.name, } } - if p.IsComponent { - err := installComponent(p, cliConfig) - if err != nil { - return installedPkgMsg{ - err: err, - name: p.name, - } - } - return installedPkgMsg{ - err: nil, - name: p.name, - } - } - if p.IsMixins { - err := installMixin(p, cliConfig) - if err != nil { - return installedPkgMsg{ - err: err, - name: p.name, - } - } - return installedPkgMsg{ - err: nil, - name: p.name, - } - } + var err error + if p.IsComponent { + err = installComponent(p, cliConfig) + } else if p.IsMixins { + err = installMixin(p, cliConfig) + } else { + err = fmt.Errorf("unknown install operation") + } return installedPkgMsg{ err: err, name: p.name, } } }
258-259: Include package type in error messageAdding the package type to the error message will help in debugging unknown package types.
-return fmt.Errorf("unknown package type") +return fmt.Errorf("unknown package type: %v", p.pkgType)
297-298: Include package type in error messageSimilarly, in
installMixin, include the package type in the error message for better clarity.-return fmt.Errorf("unknown package type") +return fmt.Errorf("unknown package type: %v", p.pkgType)internal/exec/vendor_component_utils.go (3)
96-105: ImplementExecuteStackVendorInternalfunctionThe
ExecuteStackVendorInternalfunction currently returns a "not supported yet" error. Consider implementing the necessary logic or providing a more detailed message.Would you like assistance in drafting the implementation or creating a GitHub issue to track this task?
343-343: Wrap error with%wfor proper error handlingTo enable error unwrapping, replace
%vwith%wwhen formatting errors.Apply this diff:
- return fmt.Errorf("error initializing model: %v", err) + return fmt.Errorf("error initializing model: %w", err)
341-345: Handle errors from the TUI runEnsure that errors from
tea.NewProgram(model).Run()are properly handled to maintain robustness.Apply this diff:
if _, err := tea.NewProgram(model).Run(); err != nil { - return fmt.Errorf("running download error: %w", err) + return fmt.Errorf("error running TUI program: %w", err) }
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (5)
go.mod(1 hunks)internal/exec/vendor_component_utils.go(4 hunks)internal/exec/vendor_model.go(1 hunks)internal/exec/vendor_model_component.go(1 hunks)internal/exec/vendor_utils.go(5 hunks)
🔇 Additional comments (10)
go.mod (1)
96-96: LGTM! Clean dependency addition for TUI animations.
The addition of github.com/charmbracelet/harmonica aligns perfectly with implementing an improved vendor pull interface using Bubble Tea. This indirect dependency will provide smooth animation capabilities for the TUI.
Let's verify the harmonica package usage in the codebase:
internal/exec/vendor_utils.go (8)
10-10: Importing Bubble Tea library for TUI functionality
The addition of the tea import integrates the Bubble Tea library, enabling the new TUI enhancements.
178-179: Calling logInitialMessage to enhance user feedback
Incorporating logInitialMessage improves the user experience by logging initial processing information.
Line range hint 250-309: Refactoring source processing into modular functions
The introduction of shouldSkipSource and validateSourceFields enhances code readability and maintainability by encapsulating specific logic.
271-280: Clarifying source type determination logic
The determineSourceType function cleanly separates the logic for source type determination, improving code clarity.
311-321: Implementing TUI for package processing
Introducing the TUI enhances the user interface during package processing with Bubble Tea.
372-379: Defining logInitialMessage function
Centralizing initial logging supports better code organization and reduces redundancy.
380-392: Duplicate issue: Modifying a value receiver in validateSourceFields
As mentioned earlier, modifying s.File won't affect the original struct since s is passed by value.
435-491: Generating skip function for file copying
The generateSkipFunction effectively handles inclusion and exclusion patterns, enhancing modularity and reusability.
internal/exec/vendor_component_utils.go (1)
15-15: Bubble Tea import added for TUI integration
The addition of the Bubble Tea package is appropriate for implementing TUI features, aligning with the PR objectives.
|
Running 
|
|
@aknysh i updated the tests to check for this and they passed. https://github.com/cloudposse/atmos/actions/runs/12364998225/job/34509262900?pr=768 |
|
💥 This pull request now has conflicts. Could you fix it @haitham911? 🙏 |
There was a problem hiding this comment.
Actionable comments posted: 0
🔭 Outside diff range comments (2)
internal/exec/vendor_model_component.go (1)
Line range hint
312-315: Fix ineffectual assignment to pType in local file handling.The
continuestatement skips the package creation, making thepTypeassignment ineffective. This could lead to incorrect package type determination.Apply this fix:
if u.FileExists(absPath) { pType = pkgTypeLocal - continue + uri = absPath }internal/exec/vendor_component_utils.go (1)
Line range hint
600-632: Enhance URI validation with additional security checks.The URI validation could be strengthened with more comprehensive checks.
func validateURI(uri string) error { if uri == "" { return fmt.Errorf("URI cannot be empty") } + // Normalize URI before validation + normalizedURI := filepath.Clean(uri) + // Maximum length check if len(uri) > 2048 { return fmt.Errorf("URI exceeds maximum length of 2048 characters") } - // Add more validation as needed // Validate URI format if strings.Contains(uri, "..") { return fmt.Errorf("URI cannot contain path traversal sequences") } + // Check for encoded traversal attempts + if strings.Contains(normalizedURI, "%2e%2e") { + return fmt.Errorf("URI cannot contain encoded path traversal sequences") + } // ... rest of the validation
🧹 Nitpick comments (4)
internal/exec/vendor_model_component.go (2)
20-31: Add documentation for the pkgComponentVendor struct.Consider adding detailed documentation for each field to improve maintainability.
+// pkgComponentVendor represents a vendor component package with its metadata type pkgComponentVendor struct { + // uri is the source location of the component uri string + // name is the unique identifier of the component name string + // sourceIsLocalFile indicates if the source is a local file sourceIsLocalFile bool // ... add documentation for remaining fields
116-118: Standardize error message formatting.Error messages should follow a consistent format with proper capitalization.
Apply these improvements:
-return fmt.Errorf("Failed to create temp directory %s", err) +return fmt.Errorf("failed to create temp directory: %v", err) -return fmt.Errorf("Failed to download package %s error %s", p.name, err) +return fmt.Errorf("failed to download package %s: %v", p.name, err) -return fmt.Errorf("Failed to process OCI image %s error %s", p.name, err) +return fmt.Errorf("failed to process OCI image %s: %v", p.name, err) -return fmt.Errorf("failed to copy package %s error %s", p.name, err) +return fmt.Errorf("failed to copy package %s: %v", p.name, err)Also applies to: 139-140, 146-147, 166-167
internal/exec/vendor_component_utils.go (2)
Line range hint
402-405: Enhance non-TTY mode feedback.The warning message could be more descriptive about the fallback behavior.
-u.LogWarning(cliConfig, "No TTY detected. Falling back to basic output. This can happen when no terminal is attached or when commands are pipelined.") +u.LogWarning(cliConfig, "No TTY detected. Falling back to basic output mode. Progress will be logged as text. This typically occurs when running in CI/CD pipelines or when output is redirected.")
Line range hint
497-514: Enhance determineSourceType function documentation and immutability.The function modifies its input parameter and lacks comprehensive documentation.
+// determineSourceType analyzes the given URI and determines its type (OCI, local file, or remote). +// It handles OCI scheme prefixes and resolves local file paths relative to the vendor config path. +// +// Parameters: +// - uri: Pointer to the URI string that may be modified during processing +// - vendorConfigFilePath: Base path for resolving relative file paths +// +// Returns: +// - useOciScheme: true if the URI uses the OCI scheme +// - useLocalFileSystem: true if the URI points to the local filesystem +// - sourceIsLocalFile: true if the URI points to an existing local file func determineSourceType(uri *string, vendorConfigFilePath string) (bool, bool, bool) {
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
internal/exec/vendor_component_utils.go(4 hunks)internal/exec/vendor_model.go(1 hunks)internal/exec/vendor_model_component.go(1 hunks)internal/exec/vendor_utils.go(11 hunks)pkg/utils/file_utils.go(1 hunks)pkg/vender/vendor_config_test.go(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- pkg/utils/file_utils.go
- pkg/vender/vendor_config_test.go
- internal/exec/vendor_model.go
🧰 Additional context used
📓 Learnings (4)
📓 Common learnings
Learnt from: osterman
PR: cloudposse/atmos#768
File: website/docs/cheatsheets/vendoring.mdx:70-70
Timestamp: 2024-11-12T13:06:56.195Z
Learning: In `atmos vendor pull --everything`, the `--everything` flag uses the TTY for TUI but is not interactive.
internal/exec/vendor_model_component.go (1)
Learnt from: aknysh
PR: cloudposse/atmos#768
File: internal/exec/vendor_model_component.go:3-20
Timestamp: 2024-11-18T13:59:10.824Z
Learning: When replacing significant dependencies like `go-getter` that require extensive changes, prefer to address them in separate PRs.
internal/exec/vendor_component_utils.go (2)
Learnt from: aknysh
PR: cloudposse/atmos#768
File: internal/exec/vendor_component_utils.go:312-316
Timestamp: 2024-11-20T18:38:34.525Z
Learning: Ensure that mixin package type determination supports local files, handling them consistently with component package type determination.
Learnt from: osterman
PR: cloudposse/atmos#768
File: internal/exec/vendor_component_utils.go:354-360
Timestamp: 2024-11-12T05:52:05.088Z
Learning: In the vendoring process, a TTY can exist without being interactive. If the process does not prompt the user, we should not require interactive mode to display the TUI. The `CheckTTYSupport` function should check TTY support on stdout rather than stdin.
internal/exec/vendor_utils.go (3)
Learnt from: osterman
PR: cloudposse/atmos#768
File: internal/exec/vendor_utils.go:496-513
Timestamp: 2024-11-22T12:38:33.132Z
Learning: In the Atmos project, continue to flag path traversal issues in code reviews but acknowledge when they are expected and acceptable in specific cases.
Learnt from: osterman
PR: cloudposse/atmos#768
File: internal/exec/vendor_utils.go:496-513
Timestamp: 2024-11-19T14:50:16.194Z
Learning: In the Atmos project, path traversal is acceptable due to its role in automation. Do not flag path traversal as an issue in code reviews.
Learnt from: osterman
PR: cloudposse/atmos#768
File: internal/exec/vendor_utils.go:0-0
Timestamp: 2024-11-12T05:52:05.088Z
Learning: When TTY is not supported, log the downgrade message at the Warn level using `u.LogWarning(cliConfig, ...)` instead of `fmt.Println`.
🪛 golangci-lint (1.62.2)
internal/exec/vendor_component_utils.go
314-314: ineffectual assignment to pType
(ineffassign)
🔇 Additional comments (2)
internal/exec/vendor_model_component.go (1)
129-140: 🛠️ Refactor suggestion
Add timeout context for component downloads.
The component installation uses context.Background() without a timeout, which could lead to hanging operations.
Apply this improvement:
client := &getter.Client{
- Ctx: context.Background(),
+ Ctx: func() context.Context {
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
+ // Store cancel in struct for cleanup
+ return ctx
+ }(),
Dst: tempDir,
Src: p.uri,
Mode: getter.ClientModeAny,
}Likely invalid or redundant comment.
internal/exec/vendor_utils.go (1)
75-84: LGTM! Well-implemented default behavior for the --everything flag.
The logic correctly defaults to everything = true when no specific flags are set, providing a good user experience.
|
Important Cloud Posse Engineering Team Review RequiredThis pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes. To expedite this process, reach out to us on Slack in the |
It works if we do cd examples/tests
atmos vendor pullbut if we use the base_path: "./examples/tests"and execute The issue is prob that the code just uses It should do the following:
We'll release this as is, but @haitham911 please take a look at this and try to fix it so it woks from any folder (given that the cc: @osterman |
|
These changes were released in v1.130.0. |
|
looks like this broke the parsing: when sources:
- source: 'git::https://x-access-token:{{env "GITHUB_TOKEN"}}@github.com...... |
what
interactive shell for atmos vendor pull --everything
atmos vendor pull --everything </dev/null |& cat > atmos_vendor_pull.logscreenshots
example for process with error
example for process successfully done
why
Build an interface for a package manager using bubbletea
references
atmos vendor pull --tags ... --dry-rundoesn't do a full run #792atmos vendor pull --all#301Summary by CodeRabbit
New Features
--everythingflag for thevendor pullcommand, allowing users to vendor all components at once.Bug Fixes
Documentation
--everythingflag and its usage.vendor-manifestdocumentation regarding vendoring from OCI registries.Chores