Resolves #151: mark incorrect usages of Succeed() #152

navijation · 2024-06-16T17:10:08Z

Description

Fixes #151. Add a new, partially suppressible error set for calls to Succeed.

Whenever the actual argument is not a single error value for synchronous calls, or not a function returning a single error value for asynchronous calls, this creates a non-suppressible error.

Whenever the actual argument is anything but a function call, e.g. Expect(err).To(Succeed()), this creates a suppressible error that can be suppressed with command line flags or comment directives, and suggests a replacement to Expect(err).To(HaveOccurred()).

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Added test case and related test data
- testdata/src/a/succeed/succeed.go for test cases in strict config
- testdata/src/a/succeedconfig/succeedconfig.go for test cases with suppressed config
- "Succeed" and ""test the suppress-succeed-assertion flag"" test cases in analyzer_test.go
Update the functional test
- Most counts make sense (existing +24 or +16) except one or two, but most likely not a real bug but rather "conflict"
  between two lint rules

Checklist:

I have performed a self-review of my own code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules
I ran golangci-lint

@nunnatsa

navijation · 2024-06-16T22:44:34Z

tests/e2e.sh

@@ -6,34 +6,36 @@ cp ginkgolinter testdata/src/a
 cd testdata/src/a

 # no suppress
-[[ $(./ginkgolinter a/... 2>&1 | wc -l) == 2604 ]]
+[[ $(./ginkgolinter a/... 2>&1 | wc -l) == 2628 ]]


Inspection of succeed/succeed.go and succeedconfig/succeed.go shows that 24 new errors should be reported. 8 of these assertions of the form Expect(err).To(Succeed()) vanish when succeed suppression is enabled, leaving 16 extra.

So for each test case here, the number of errors should rise by either 24 or 16.

navijation · 2024-06-17T05:23:02Z

tests/e2e.sh

 # suppress all but spec pollution
-[[ $(./ginkgolinter --suppress-nil-assertion=true --suppress-err-assertion=true --suppress-len-assertion=true --suppress-compare-assertion=true --suppress-async-assertion=true --forbid-spec-pollution=true --suppress-type-compare-assertion=true a/... 2>&1 | wc -l) == 253 ]]
+[[ $(./ginkgolinter --suppress-nil-assertion=true --suppress-err-assertion=true --suppress-len-assertion=true --suppress-compare-assertion=true --suppress-async-assertion=true --forbid-spec-pollution=true --suppress-type-compare-assertion=true --suppress-succeed-assertion=true a/... 2>&1 | wc -l) == 275 ]]


This one goes up by 22 errors... close to 24 but not quite. Probably fine though...

navijation · 2024-06-17T05:24:42Z

tests/e2e.sh

 # suppress spec pollution
-[[ $(./ginkgolinter --forbid-spec-pollution=true a/... 2>&1 | wc -l) == 2702 ]]
+[[ $(./ginkgolinter --forbid-spec-pollution=true a/... 2>&1 | wc -l) == 2732 ]]


This one goes up by 30 errors. I did not intend to add that many new errors. Maybe it's because my test data changes inadvertently had some spec pollution going on.

nunnatsa

Thanks you so much for this PR @navijation!

I added a few preliminary comments. I didn't reviewed the logic of the new rule, yet.

One general comment: please also update the Readme file.

nunnatsa · 2024-06-17T05:53:17Z

testdata/src/a/succeed/succeed.go

+func safeDivide(num int, denom int) (int, error) {
+	if denom == 0 {
+		return 0, errors.New("divide by zero")
+	}
+	return num / denom, nil
+}
+
+func attemptToCallNetwork(shouldTry bool) error {
+	if !shouldTry {
+		return nil
+	}
+	return errors.New("don't know how to connect to network")
+}


nit: thinking out load (not sure about it), but since we doing a static analysis here, we really don't care about the logic of these functions and what is actually been returned on runtime.

So what I'm not sure about is if using an actual logic will confuse a future reader. Because we don't know and don't care what the function will return on runtime, but only what it can return.

🤷 no problem I’ll rename

nunnatsa · 2024-06-17T05:54:02Z

testdata/src/a/succeed/succeed.go

+	wrap := struct {
+		err error
+	}{
+		err: err0,
+	}


should we care about wrapped errors? Is it a real usecase?

I just added this to test the case where the actual argument is a neither a function call nor an identifier. I could remove it, although more case coverage wouldn't hurt I'd think.

nunnatsa · 2024-06-17T06:10:53Z

testdata/src/a/succeedconfig/succeedconfig.go

+	value0, err0 := safeDivide(0, 0)
+	value1, _ := safeDivide(0, 1)
+
+	Expect(value0, err0).ToNot(Succeed()) // want `ginkgo-linter: Succeed matcher must be asserted against exactly one error value or a function call returning the same, or it will always fail`


There are two issues here: this is also not a function call. Is there any way to catch them both? One of the concepts I'm trying to apply in this linter, is that if the user fixes a warning, they won't get a new warning for the fixed code.

So skip directly to Expect(err0).ToNot(HaveOccurred())? This seems to imply that I should add a suggested fix for the original type error. It's a bit more challenging to do that in general. e.g. For Expect(os.ReadFile("someFile")).To(Succeed()) and Expect(err1, err2).To(Succeed()), what would the suggested fixes be? I can take a look at doing this but some guidance would be helpful.

IMO, multiple issues are not ideal but not the end of the world. It's like having a Go type error. Generally you need to fix those before fixing other lint errors, which will come after the fix.

Also note that some of these suggested fixes can either change the code semantics by dropping function calls or introduce compiler errors due to unused variables. It might be better to let the user figure out how they want to satisfy the type constraint.

The linter only suggests fixes if it can do it without changing the test logic. The suggested fixes is less important than the warnings.

For example, the "expect without assertion" rule does not offer any fix.

I may have misunderstood. Are you suggesting that that the linter should say “Succeed matcher must be asserted against a function call returning one error, or it will always fail”.

I can do that, but it’s sort of an inaccurate message because it is not an error to use it against a value, just a style issue. Some users might suppress the style issue. If that isn’t a problem I can simplify the message that way, or complexify it like “Succeed matcher must be asserted against a single expression of type error, ideally a function call, or it will always fail”.

@nunnatsa I'm still not entirely sure what you're proposing here so I held off on making a change.

Option 1

Keep it as is. The user can fix the type error first and then the style error if applicable based on their config.

Option 2

Change the linter to say "Succeed matcher must be asserted against a function call returning exactly one error, or it will always fail" so that the user changes the usage to be that of a function call immediately to avoid the style error and type error.

Option 3

Change the linter to say “Succeed matcher must be asserted against a single expression of type error, ideally a function call, or it will always fail” or something of that sort.

What I meant is that the reportBuilder supports multiple warnings. I'm not trying to build a special warning message for each case, but gather all the relevant warnings. see here for example:

ginkgolinter/testdata/src/a/matcherror/matcherror.go

Line 112 in b14486e

Expect(e1).To(MatchError(1, "not used"), "wrong type - int") // want `ginkgo-linter: multiple issues: MatchError first parameter \(1\) must be error, string, GomegaMatcher or func\(error\)bool are allowed; redundant MatchError arguments; consider removing them`

I see...

One problem is that the fixes are kind of conflicting. If you switch to using Expect(err).ToNot(HaveOccurred()), it's no longer relevant to fix the signature to match what Succeed expects.

And secondly, it means the linter needs to figure out a suggestion in the special case where exactly one argument is an error, e.g. Expect(5, err).To(Succeed()) can be suggest fixed as Expect(err).ToNot(HaveOccurred()), but Expect(5).To(Succeed()) has no suggested fix. And I'm not sure what the suggested fix should be for Expect(err1, err2).To(Succeed()).

So what I gather is:

if it is possible and configured to suggest a switch to Expect(...).NotTo(HaveOccurred()), then do that
a. "possible" meaning exactly a single argument implements error and the rest do not
b. although fixing this error as such will likely lead to unused variable errors from the Go compiler that the user will still have to fix

(maybe) Do not post the warning about how Succeed's expected argument types work if condition 1 holds

This complicates things. I can manage it, but something about this approach feels off to me, especially point 1b.

coveralls · 2024-06-17T06:21:25Z

Pull Request Test Coverage Report for Build 9542355837

Details

6 of 8 (75.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 64.545%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
types/config.go	5	7	71.43%

Totals
Change from base Build 9516146313:	0.1%
Covered Lines:	213
Relevant Lines:	330

💛 - Coveralls

nunnatsa

After went over the code itself, added some additional comments inline.

General comment: please add a file without dot import, to the testdata. Testing the full name, like gomega.Expect().To(gomega.Succeed() may raise some unexpected issues.

nunnatsa · 2024-06-18T11:44:11Z

testdata/src/a/forceExpectTo/gomegavar.go

@@ -7,16 +7,18 @@ import (

 var _ = Describe("gomega var", func() {
 	It("in a valid Eventually", func() {
-		Eventually(func(g Gomega) {
+		Eventually(func(g Gomega) error {


This is actually a valid usecase, documented here: https://onsi.github.io/ginkgo/#testing-external-systems

We must not trigger a warning for code like:

Eventually(func (g Gomega){ g.Expect(...) ... }).Should(Succeeed())

nunnatsa · 2024-06-18T11:47:58Z

testdata/src/a/succeedconfig/succeedconfig.go

+	value0, err0 := safeDivide(0, 0)
+	value1, _ := safeDivide(0, 1)
+
+	Expect(value0, err0).ToNot(Succeed()) // want `ginkgo-linter: Succeed matcher must be asserted against exactly one error value or a function call returning the same, or it will always fail`


The linter only suggests fixes if it can do it without changing the test logic. The suggested fixes is less important than the warnings.

For example, the "expect without assertion" rule does not offer any fix.

analyzer.go

nunnatsa · 2024-06-18T12:06:20Z

linter/ginkgo_linter.go

+
+	isAsync := false
+	return doCheckSucceed(
+		pass, assertionExpCopy, matcherCopy, actualArgs, handler, reportBuilder, isAsync,


isAsync is always false. so we don't need this variable.

This is more for readability purposes because boolean arguments otherwise are unreadable. I can remove though

nunnatsa · 2024-06-18T12:09:33Z

linter/ginkgo_linter.go

@@ -396,21 +401,25 @@ func doCheckExpression(pass *analysis.Pass, config types.Config, assertionExp *a

 	} else if checkMatchError(pass, assertionExp, actualArg, handler, reportBuilder, isExprError) {
 		return false
+	} else if checkSucceedSync(pass, assertionExp, actualArgs, handler, reportBuilder, bool(config.SuppressSucceed)) {


nit: can we split it to first check if we're using Succeed() here, and then validate the usage of it?

If I understand what you're saying correctly, i.e. something like if isSucceed(...) { return checkSucceed(...) }, this will prevent the rules below from being checked even if the usage of Succeed is fine.

Also, most of the checks here work the same way of doing both structure check as well as reporting. What's special about Succeed?

nunnatsa · 2024-06-18T13:15:04Z

linter/ginkgo_linter.go

@@ -1637,6 +1792,23 @@ func isExprError(pass *analysis.Pass, expr ast.Expr) bool {
 	return false
 }

+// Returns whether the expression implements type error or is a tuple of length 1 whose only
+// element implements type error.
+func isExprExactError(pass *analysis.Pass, expr ast.Expr) bool {


I'm confused by the name of this function. Maybe I'm wrong here, but could it be that the logic is reverse than what its name suggests?

This is checking to see if the expression evaluates to a single value which is an error, as opposed to multiple values of a tuple where the first is an error. Maybe something like isExprSingleError?

nunnatsa · 2024-06-18T13:26:52Z

linter/ginkgo_linter.go

+
+// Negate an assertion by negating the assertion function, e.g. turn `Expect(x).To(BeEmpty())`
+// to `Expect(x).ToNot(BeEmpty())`
+func negateAssertion(assertionExpr *ast.CallExpr, handler gomegahandler.Handler) bool {


This is done by function in the reverseassertion package. If these functions are not the prefect match here, please move this function to this package

nunnatsa · 2024-06-18T13:35:55Z

linter/ginkgo_linter.go

+	assertionExpCopy := astcopy.CallExpr(assertionExp)
+	matcherCopy := assertionExpCopy.Args[0].(*ast.CallExpr)
+
+	isAsync := true


this variable is not needed. It never changed

nunnatsa · 2024-06-18T13:39:45Z

linter/ginkgo_linter.go

+//
+//	// function signature is `os.ReadFile(string) ([]byte, error)`
+//	Eventually(func() ([]byte, error) { return os.ReadFile("someFile") }).Should(Succeed())
+func checkSucceedAsync(


Be careful here. The following code is valid, but will trigger a warning with the current logic:

Eventually(func(g Gomega){ ... g.Expect(...)... ... }).Should(Succeed())

coveralls · 2024-06-24T05:38:39Z

Pull Request Test Coverage Report for Build 9639348734

Details

7 of 9 (77.78%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 64.653%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
types/config.go	5	7	71.43%

Totals
Change from base Build 9516146313:	0.3%
Covered Lines:	214
Relevant Lines:	331

💛 - Coveralls

README.md

nunnatsa · 2024-06-24T06:20:36Z

@navijation - would you mind to change the commits, so they will include only changes from main, and not from each other? For example, if a commit added some code that was later modified or removed, please use the final version of the change in the commit. we have now 14 commits in this PR, and I think we can do that with 4-5 commits.

For a specific example, the "fix bug" commit should be rebased with the commit that added the code before the fix.

navijation · 2024-07-01T06:27:23Z

@nunnatsa I've rebased as 4 commits to the best of my ability. Although I would probably recommend squash merging this PR since the boundaries are a little blurry between each commit after merging them.

coveralls · 2024-07-02T05:28:54Z

Pull Request Test Coverage Report for Build 9739311110

Details

7 of 9 (77.78%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 64.653%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
types/config.go	5	7	71.43%

Totals
Change from base Build 9516146313:	0.3%
Covered Lines:	214
Relevant Lines:	331

💛 - Coveralls

nunnatsa · 2024-07-02T05:37:29Z

internal/interfaces/interfaces.go

+
+// Note: we cannot check for an argument which _implements_ gomega.Gomega without adding a Go
+// module dependency on gomega.  This is because the gomega.Gomega interface definition references
+// gomega.AsyncAssertion, whose methods return gomega.AsyncAssertion. Because Go does not have
+// interface covariance or contravariance, any "local copy" of gomega.AsyncAssertion cannot
+// be satisified by any actual `gomega.AsyncAssertion` implementation, as the methods do not return
+// local.AsyncAssertion but rather gomega.AsyncAssertion.
+//
+// Also, Gomega probably doesn't even accept an argument whose type implements the interface, but
+// rather whose type _is_ the interface. So this check should suffice.
+func IsGomega(t gotypes.Type) bool {
+	named, isNamed := t.(*gotypes.Named)
+	if !isNamed {
+		return false
+	}
+
+	obj := named.Obj()
+
+	if obj.Name() != "Gomega" {
+		return false
+	}
+
+	return isPackageSymbol(named.Obj(), "github.com/onsi/gomega/types", "Gomega")
+}
+
+func isPackageSymbol(obj types.Object, pkgPath, name string) bool {
+	if obj.Name() != name {
+		return false
+	}
+
+	vendorPieces := strings.Split(pkgPath, "/vendor/")
+
+	return pkgPath == vendorPieces[len(vendorPieces)-1]
+}


I think, but not sure, that we can use the gomegahandler package for this check. The handler only exists if the gomega package is imported, and holds the knowledge if it's dot import or named import.

So you'll only need to check if the type name is Gomega and the left side, if exists, is expected by the handler. I think it simpler. WDYT?

nunnatsa · 2024-07-02T05:47:27Z

testdata/src/a/succeed/succeed.go

+	resp := struct {
+		response any
+		err      error
+	}{
+		response: nil,
+		err:      err0,
+	}


Not sure testing an error field adds any value, over testing an error variable.

nunnatsa · 2024-07-02T06:06:30Z

linter/ginkgo_linter.go

+// Returns whether the expression is a function that takes in a single gomega.Gomega argument and
+// returns nothing, e.g. `func(g gomega.Gomega) { g.Expect(myStringMaker()).ToNot(BeEmpty()) }`


The Gomega parameter is not necessary a single parameter. The check should be about the first parameter, if exists.

nunnatsa · 2024-07-02T06:11:05Z

linter/ginkgo_linter.go

+	if t.Results().Len() != 0 || t.Params().Len() != 1 {
+		return false
+	}


Continue: this condition will give the wrong result for valid usages like:

Eventually(func(g Gomega, ctx context.Context){...})...

Or

Eventually(func(g Gomega, x, y int){ g.Expect(x).To(Equal(y)) }).WithArguments(1,2).Should(Succeed())

nunnatsa · 2024-07-02T06:30:29Z

linter/ginkgo_linter.go

+
+	if isAsync {
+		if len(actualArgs) != 1 ||
+			!(isExprErrFunc(pass, actualArgs[0]) || isExprGomegaFunc(pass, actualArgs[0])) {


the function under test is not always the first argument. e.g. this code is valid:

Eventually(ctx, func(...){...}).Should(Succeed())

Eventually is with very flexible api...

This is actually checked here: https://github.com/nunnatsa/ginkgolinter/blob/main/linter/ginkgo_linter.go#L743-L759

nunnatsa · 2024-10-29T04:37:54Z

@navijation - I made a massive refactoring, to address #153

Do you still want to keep working on it?

Resolves nunnatsa#151: mark incorrect usages of Succeed()

c721a1c

navijation commented Jun 16, 2024

View reviewed changes

navijation marked this pull request as ready for review June 17, 2024 00:12

navijation changed the title ~~[WIP] Resolves #151: mark incorrect usages of Succeed()~~ Resolves #151: mark incorrect usages of Succeed() Jun 17, 2024

navijation commented Jun 17, 2024

View reviewed changes

nunnatsa reviewed Jun 17, 2024

View reviewed changes

nunnatsa reviewed Jun 18, 2024

View reviewed changes

nunnatsa self-assigned this Jun 18, 2024