feat: Add pattern matching support to atlantis plan/apply -d flag #2742

jmediald-te · 2022-11-30T19:39:48Z

what

This PR adds support for filepath pattern matching expressions to the -d/--dir flag in atlantis plan/apply commands. The user must use the --enable-regexp-cmd server configuration flag to activate this feature.
When we run atlantis plan/apply -d <pattern>, atlantis will behave exactly the same way as when running atlantis plan/apply. The only difference is that it will only plan/apply the projects whose relative dir matches with the specified pattern.

why

references

Closes Allow supplying wildcard for dir #686

…rn matching

jamengual · 2022-11-30T19:57:41Z

thanks @jmediald-te for the contribution
could you add docs with examples and tests for this new flag?

nitrocode · 2022-12-01T00:18:10Z

How is this different than --enable-regexp-cmd ?

jmediald-te · 2022-12-01T10:53:48Z

Thanks for your response @jamengual. I will try to work on the docs/tests later.

@nitrocode - Based on the documentation, --enable-regexp-cmd allows us to use regular expressions when using the -p command. It is often the case where we do not have all projects defined in atlantis.yaml so the -p flag won't work. The atlantis apply/plan -f <pattern> command would be equivalent to atlantis apply/plan -d <pattern> if the -d flag had support for pattern matching.

jamengual · 2022-12-01T13:11:29Z

why we don't add patterns matching to the - d instead? we can reuse the same flag used in - p to enable it. what do you think?

…

On Thu, Dec 1, 2022, 2:54 a.m. Josep Medialdea ***@***.***> wrote: Thanks for your response @jamengual <https://github.com/jamengual>. I will try to work on the docs/tests later. @nitrocode <https://github.com/nitrocode> - Based on the documentation, --enable-regexp-cmd allows us to use regular expressions when using the -p command. It is often the case where we do not have all projects defined in atlantis.yaml so the -p flag won't work. The atlantis apply/plan -f <pattern> command would be equivalent to atlantis apply/plan -d <pattern> if the -d flag had support for pattern matching. — Reply to this email directly, view it on GitHub <#2742 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERDQBB6Q3D676JUI3N3WLB7URANCNFSM6AAAAAASP7XH4U> . You are receiving this because you were mentioned.Message ID: ***@***.***>

jmediald-te · 2022-12-01T13:36:36Z

Makes a lot of sense. It will be more consistent.

I will work on that.

nitrocode · 2022-12-02T16:29:03Z

Related discussion in PR #2694 with @secustor. Unsure if you folks want to close your existing prs and create a new one or use an existing pr and/or work together?

See issue #259

secustor · 2022-12-02T17:09:04Z

@nitrocode I think this are two different use cases.

This PR is used to define the runs based on the folder structure. Ignoring defined projects if I'm not mistaken. Sry I'm on my mobile so I can't check the code reasonable.

My PR is about running all defined projects.

nitrocode · 2022-12-02T18:09:44Z

@secustor as of today, you are technically correct about both prs, however the discussion has led back to a -d regexp. Both you and this pr author have attempted different solutions to the same problem. Since that is the case, i am suggesting that one of you drives the suggested solution using -d regexp and the other contributes as a pr reviewer (or author) so we can achieve the desired outcome 😄

jmediald-te · 2022-12-05T11:12:04Z

Hey, sorry for my late reply. I have been very busy these days. I will be working on this today.

What I am doing is basically adding support for filepath pattern matching into the atlantis apply/plan -d flag. The pattern matching will only take into consideration the projects/plans that would be planned/applied by the current atlantis plan/apply (i.e. the project that are modifyed by the PR in the case of atlantis plan and the currently unapplied plans in the case of atlantis apply). At first, I created a new -f/--filter flag but as @jamengual mentioned it will be more consistent to add this functionality into the already existing -d flag.

If you want I can rename this PR to: feat: Add support for filepath pattern matching into atlantis plan/apply -d flag. Or create a new PR with the new name.

jmediald-te · 2022-12-05T13:21:29Z

I have just moved the functionality of the -f/--filter flag to the already existing -d flag. In order for this to work the user needs to use --enable-regexp-cmd.

nitrocode · 2022-12-05T18:51:07Z

server/events/project_command_builder.go

@@ -332,6 +342,15 @@ func (p *DefaultProjectCommandBuilder) buildPlanAllCommands(ctx *command.Context
 		ctx.Log.Debug("moduleInfo for %s (matching %q) = %v", repoDir, p.AutoDetectModuleFiles, moduleInfo)
 		modifiedProjects := p.ProjectFinder.DetermineProjects(ctx.Log, modifiedFiles, ctx.Pull.BaseRepo.FullName, repoDir, p.AutoplanFileList, moduleInfo)
 		ctx.Log.Info("automatically determined that there were %d projects modified in this pull request: %s", len(modifiedProjects), modifiedProjects)
+
+		if p.EnableRegExpCmd && cmd.RepoRelDir != "" {


This code looks to be the same as lines 303 to 309.

Can we encapsulate this into a function to keep the code as DRY as possible?

Or is there a better way to structure this so we can run this block for both if and else blocks?

nitrocode · 2022-12-05T18:52:02Z

server/events/project_command_builder.go

@@ -482,6 +501,14 @@ func (p *DefaultProjectCommandBuilder) buildAllProjectCommands(ctx *command.Cont
 		return nil, err
 	}

+	if p.EnableRegExpCmd && commentCmd.RepoRelDir != "" {


Another duplicated block here

nitrocode · 2022-12-05T18:53:06Z

server/events/project_command_builder.go

+	return filteredProjects, nil
+}
+
+func filterValidProjects(projects []valid.Project, filter string) ([]valid.Project, error) {


Can we combine both filterProjects and filterValidProjects ?

Only difference I see is models.Project and proj.Path vs valid.Project and proj.Dir

nitrocode · 2022-12-05T18:54:48Z

server/events/project_command_builder.go

+	return filteredProjects, nil
+}
+
+func filterPlans(plans []PendingPlan, filter string) ([]PendingPlan, error) {


This could also be combined with the above 2 functions. Only thing I see here that's different is using PendingPlan and plan.RepoRelDir

nitrocode · 2022-12-05T18:55:49Z

server/events/event_parser.go

@@ -112,6 +112,10 @@ func (c CommentCommand) IsForSpecificProject() bool {
 	return c.RepoRelDir != "" || c.Workspace != "" || c.ProjectName != ""


Please re-word the PR summary and title to reflect that this is no longer using the -f flag and instead will add wildcard filtering using the existing -d flag

nitrocode · 2022-12-05T18:56:12Z

server/events/event_parser.go

+func (c CommentCommand) HasDirPatternMatching() bool {
+	return strings.Contains(c.RepoRelDir, "*") || strings.Contains(c.RepoRelDir, "?") || strings.Contains(c.RepoRelDir, "[") || strings.Contains(c.RepoRelDir, "]")
+}
+
 // CommandName returns the name of this command.
 func (c CommentCommand) CommandName() command.Name {


Please add tests for the new functionality

nitrocode · 2022-12-07T02:26:32Z

If we add a -d flag, we will have to also allow setting a unique workspace along with it for people that use workspaces.

The following should work

# should plan everything using the dev workspace
atlantis plan -d .* -w dev
# should plan everything using the prod workspace
atlantis plan -d .* -w prod
# should plan everything using the default workspace
atlantis plan -d .*

nitrocode · 2022-12-19T04:35:51Z

@jmediald-te friendly ping here. We'd love to see this change in the next release if you have time.

jmediald-te · 2023-01-04T12:41:29Z

Hey @nitrocode, sorry for my late reply. I have been out of office during the holiday break. I will be working on this soon.

nitrocode · 2023-02-22T23:17:03Z

Hi @jmediald-te . Any update on this? We'd love to see this feature. It's currently the second highest feature requested. :)

https://github.com/runatlantis/atlantis/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc

github-actions · 2023-03-26T01:59:18Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.

github-actions · 2023-05-29T01:58:31Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.

jamengual · 2023-08-18T05:21:12Z

if someone is willing to work on this and finish this pr , we will be happy to review

github-actions · 2023-09-19T01:47:21Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.

jamengual · 2023-09-27T17:16:35Z

@jmediald-te do you think you can continue this work? Thanks.

github-actions · 2023-10-29T01:47:58Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.

jrodante · 2023-11-02T17:51:48Z

Would love to see this make it to the finish line, any updates on when work can be resumed on it?

jamengual · 2023-11-02T17:56:55Z

The original committer has not responded in a while if you have time @jrodante and golango knowledge go for it.

jamengual · 2023-11-15T05:25:13Z

if anyone is interested, please reopen

Add -f flag to filter which directories to run apply/plan using patte…

ba3d814

…rn matching

jmediald-te requested a review from a team as a code owner November 30, 2022 19:39

jamengual added feature New functionality/enhancement waiting-on-response Waiting for a response from the user labels Nov 30, 2022

Merge branch 'main' into main

d1983e6

Merge branch 'main' into main

aaaaf25

nitrocode marked this pull request as draft December 5, 2022 03:31

add support for filepath pattern matching to atlantis plan/apply -d flag

eeb6f3f

nitrocode reviewed Dec 5, 2022

View reviewed changes

nitrocode added the needs tests Change requires tests label Dec 5, 2022

nitrocode mentioned this pull request Dec 6, 2022

Allow supplying wildcard for dir #686

Open

jmediald-te changed the title ~~feat: Add -f flag to filter which directories to run apply/plan using patte…~~ feat: Add pattern matching support to atlantis plan/apply -d flag Dec 6, 2022

nitrocode added the work-in-progress label Dec 17, 2022

github-actions bot added the Stale label Mar 26, 2023

github-actions bot closed this Apr 27, 2023

jamengual reopened this Apr 27, 2023

github-actions bot removed the Stale label Apr 28, 2023

github-actions bot added the Stale label May 29, 2023

github-actions bot closed this Jun 29, 2023

jamengual reopened this Aug 18, 2023

github-actions bot removed the Stale label Aug 19, 2023

github-actions bot added the Stale label Sep 19, 2023

github-actions bot removed the Stale label Sep 28, 2023

github-actions bot added the Stale label Oct 29, 2023

github-actions bot removed the Stale label Nov 3, 2023

jamengual closed this Nov 15, 2023

nitrocode mentioned this pull request Jan 18, 2024

feat: set undefined project name to dir #4150

Draft

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add pattern matching support to atlantis plan/apply -d flag #2742

feat: Add pattern matching support to atlantis plan/apply -d flag #2742

jmediald-te commented Nov 30, 2022 •

edited by nitrocode

Loading

jamengual commented Nov 30, 2022

nitrocode commented Dec 1, 2022

jmediald-te commented Dec 1, 2022

jamengual commented Dec 1, 2022 via email

jmediald-te commented Dec 1, 2022

nitrocode commented Dec 2, 2022 •

edited

Loading

secustor commented Dec 2, 2022

nitrocode commented Dec 2, 2022

jmediald-te commented Dec 5, 2022

jmediald-te commented Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode Dec 5, 2022

nitrocode commented Dec 7, 2022

nitrocode commented Dec 19, 2022

jmediald-te commented Jan 4, 2023

nitrocode commented Feb 22, 2023

github-actions bot commented Mar 26, 2023

github-actions bot commented May 29, 2023

jamengual commented Aug 18, 2023

github-actions bot commented Sep 19, 2023

jamengual commented Sep 27, 2023

github-actions bot commented Oct 29, 2023

jrodante commented Nov 2, 2023

jamengual commented Nov 2, 2023

jamengual commented Nov 15, 2023

		@@ -112,6 +112,10 @@ func (c CommentCommand) IsForSpecificProject() bool {
		return c.RepoRelDir != "" \|\| c.Workspace != "" \|\| c.ProjectName != ""

feat: Add pattern matching support to atlantis plan/apply -d flag #2742

feat: Add pattern matching support to atlantis plan/apply -d flag #2742

Conversation

jmediald-te commented Nov 30, 2022 • edited by nitrocode Loading

what

why

references

jamengual commented Nov 30, 2022

nitrocode commented Dec 1, 2022

jmediald-te commented Dec 1, 2022

jamengual commented Dec 1, 2022 via email

jmediald-te commented Dec 1, 2022

nitrocode commented Dec 2, 2022 • edited Loading

secustor commented Dec 2, 2022

nitrocode commented Dec 2, 2022

jmediald-te commented Dec 5, 2022

jmediald-te commented Dec 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nitrocode commented Dec 7, 2022

nitrocode commented Dec 19, 2022

jmediald-te commented Jan 4, 2023

nitrocode commented Feb 22, 2023

github-actions bot commented Mar 26, 2023

github-actions bot commented May 29, 2023

jamengual commented Aug 18, 2023

github-actions bot commented Sep 19, 2023

jamengual commented Sep 27, 2023

github-actions bot commented Oct 29, 2023

jrodante commented Nov 2, 2023

jamengual commented Nov 2, 2023

jamengual commented Nov 15, 2023

jmediald-te commented Nov 30, 2022 •

edited by nitrocode

Loading

nitrocode commented Dec 2, 2022 •

edited

Loading