Library test case produces different error than expected in CI

[michael-valdron]

/kind bug

Which area is this bug related to?

/area ci
/area library

What versions of software are you using?

Go project

Operating System and version: Ubuntu 22.04.4 LTS

Go Pkg Version: 1.21

Bug Summary

Describe the bug:

Library test case Test_parseFromRegistry/should_fail_if_registryUrl_is_invalid produces different error server misbehaving instead of expected no such host. This only seems to happening in the CI, not local test runs of make test, after updating to Go 1.21 and Kubernetes 1.29.

To Reproduce:

One can attempt to reproduce by running make test with a Go 1.21 runtime, so far reproducing this difference in error does not work locally (for me in Fedora 39) where it run with expected error that passes the test case.

To reproduce in CI at the time of writing, produce a branch with updates to library to use Go 1.21 and Kubernete 1.29, including updating the Go runtime version in the workflow, then run the build workflow.

Expected behavior

Error should be the following to match the test case expectation Get .* dial tcp: lookup http: .*:

Get "http://http//invalid.com/devfiles/notexist/": dial tcp: lookup http: no such host

Any logs, error output, screenshots etc? Provide the devfile that sees this bug, if applicable

Currently getting this error in that test case:

Get "http://http//invalid.com/devfiles/notexist/": dial tcp: lookup http on 127.0.0.53:53: server misbehaving

Error Log

=== RUN   Test_parseFromRegistry/should_fail_if_registryUrl_is_invalid
    parse_test.go:4879: 
        	Error Trace:	/home/runner/work/library/library/pkg/devfile/parser/parse_test.go:4879
        	Error:      	Expect "Get "http://http//invalid.com/devfiles/notexist/": dial tcp: lookup http on 127.0.0.53:53: server misbehaving" to match "Get .* dial tcp: lookup http: .*"
        	Test:       	Test_parseFromRegistry/should_fail_if_registryUrl_is_invalid
        	Messages:   	Test_parseFromRegistry(): Error message should match
=== RUN   Test_parseFromRegistry/should_fail_if_registry_returns_a_bad_devfile_content
--- FAIL: Test_parseFromRegistry (1.00s)

Additional context

Any workaround?

Don't make these changes, but this blocks dependencies from updating the Kubernetes version.

Suggestion on how to fix the bug

None yet, needs investigating.

[michael-valdron]

blocking devfile/library#212 from merging

[michael-valdron]

Was able to reproduce within a Ubuntu 22.04 LTS VM:

From here can debug what is happening.

[michael-valdron]

Was able to reproduce within a Ubuntu 22.04 LTS VM:

From here can debug what is happening.

Running same make test on Fedora 39 with same Go version 1.21.10 all cases pass, issue seems isolated to Ubuntu.

[michael-valdron]

Was able to trace error back to here: https://github.com/golang/go/blob/752b009010df021c45f620e683ec062d22b552bf/src/net/dnsclient_unix.go#L230

Expected error should be traced here: https://github.com/golang/go/blob/752b009010df021c45f620e683ec062d22b552bf/src/net/dnsclient_unix.go#L209

The problem seems to be in how systemd-resolved is resolving the given hostname.

[michael-valdron]

Testing later Ubuntu version in workflows: devfile/library#214

[michael-valdron]

Testing later Ubuntu version in workflows: devfile/library#214

No good, still produces:

=== RUN   Test_parseFromRegistry/should_fail_if_registryUrl_is_invalid
    parse_test.go:4879: 
        	Error Trace:	/home/runner/work/library/library/pkg/devfile/parser/parse_test.go:4879
        	Error:      	Expect "Get "http://http//invalid.com/devfiles/notexist/": dial tcp: lookup http on 127.0.0.53:53: server misbehaving" to match "Get .* dial tcp: lookup http: .*"
        	Test:       	Test_parseFromRegistry/should_fail_if_registryUrl_is_invalid
        	Messages:   	Test_parseFromRegistry(): Error message should match

[thepetk]

I've been looking around this and I think it might make sense to remove the prefix from the test?

https://github.com/devfile/library/blob/main/pkg/devfile/parser/parse_test.go#L4857

[michael-valdron]

I've been looking around this and I think it might make sense to remove the prefix from the test?

https://github.com/devfile/library/blob/main/pkg/devfile/parser/parse_test.go#L4857

I found this answer too and I've tried this, it produces a different error as the function from net needs the prefix to determine which protocol to use.

[michael-valdron]

Was able to reproduce within a Ubuntu 22.04 LTS VM:
From here can debug what is happening.

Running same make test on Fedora 39 with same Go version 1.21.10 all cases pass, issue seems isolated to Ubuntu.

Isolated problem down to it only effecting Ubuntu Server on Go 1.21, all other instances including Ubuntu Desktop should be unaffected to my knowledge.

[michael-valdron]

Using a bypass error pattern to expect server misbehaving if Ubuntu Server: devfile/library#217

Tried these changes with changes in devfile/library#212 under a Ubuntu Server 22.04 LTS and got:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy
$ make test
...
=== RUN   Test_parseFromRegistry
=== RUN   Test_parseFromRegistry/should_fail_if_provided_registryUrl_does_not_have_protocol_prefix
=== RUN   Test_parseFromRegistry/should_be_able_to_parse_from_provided_registryUrl
=== RUN   Test_parseFromRegistry/should_be_able_to_parse_from_registry_URL_defined_in_tool
=== RUN   Test_parseFromRegistry/should_be_able_to_parse_from_provided_registryUrl_with_latest_version_specified
=== RUN   Test_parseFromRegistry/should_be_able_to_parse_from_provided_registryUrl_with_version_specified
=== RUN   Test_parseFromRegistry/should_fail_if_provided_registryUrl_cannot_download_resources
=== RUN   Test_parseFromRegistry/should_fail_if_version_does_not_exist
=== RUN   Test_parseFromRegistry/should_fail_if_registryId_does_not_exist
=== RUN   Test_parseFromRegistry/should_fail_if_registryUrl_is_not_provided,_and_no_registry_URLs_has_been_set_in_tool
=== RUN   Test_parseFromRegistry/should_fail_if_registryUrl_is_invalid
=== RUN   Test_parseFromRegistry/should_fail_if_registry_returns_a_bad_devfile_content
--- PASS: Test_parseFromRegistry (2.73s)
...