e2e test case "TestAntctl/testAntctlProxy/AgentIPv4" always fails on branch feature/externalnode

[wenyingd]

Describe the bug
On feature branch feature/externalnode, e2e test case "TestAntctl/testAntctlProxy/AgentIPv4" always fails in kind test and CI e2e-test job after rebasing on the main branch.

Having added some log info with the failed case, I got this output:

    antctl_test.go:273: 'antctl proxy' command stdout: <>, stderr:<Error: error when creating Agent client config: no Antrea Agent found for Node name jenkins-antrea-e2e-for-pull-request-961-lw9m7

Looking into the code, the error log is output when antctl fails to read AntreaAgentInfo with the target Node name. It is possible to be related with the changes move the workflow of AntreaAgentInfo creation from Agent to Controller.

Following is samples of failed logs:
http://10.176.27.169:8080/job/antrea-e2e-for-pull-request/961/console
https://github.com/wenyingd/antrea/runs/6702775343?check_suite_focus=true

To Reproduce

1. Rebase branch feature/externalnode on main
2. Create a PR on the changes
3. Run kind test or test-e2e

Expected
All tests are expected to pass

Actual behavior
Case "TestAntctl/testAntctlProxy/AgentIPv4" is failed.

Versions:
feature branch: feature/externalnode

[wenyingd]

Having added some debug info in antctl code where returns error, and got the followsing output:

antctl_test.go:273: 'antctl proxy' command stdout: <>, stderr:<Error: error when creating Agent client config: no Antrea Agent found for Node name kind-control-plane, exsiting agentInfo count 3, values kind-control-plane,kind-worker,kind-worker2

The changed code in path pkg/antctl/raw/helper.go is like this,

@@ -96,7 +98,12 @@ func CreateAgentClientCfg(k8sClientset kubernetes.Interface, antreaClientset ant
                }
        }
        if agentInfo == nil {
-               return nil, fmt.Errorf("no Antrea Agent found for Node name %s", nodeName)
+               existingAgentInfos := sets.String{}
+               for i := range agentInfoList.Items {
+                       ai := agentInfoList.Items[i]
+                       existingAgentInfos.Insert(ai.Name)
+               }
+               return nil, fmt.Errorf("no Antrea Agent found for Node name %s, exsiting agentInfo count %d, values %s", nodeName, len(existingAgentInfos), strings.Join(existingAgentInfos.List(), ","))
        }

[mengdie-song]

I am looking into it and will make a fix.

[mengdie-song]

Command "antctl proxy --agent-node" requires the information from AntreaAgentInfo.
As a result, we need to make sure the content of AntreaAgentInfo is populated before
we run the command.

Since we recently move AntreaAgentInfo creation from agent to controller for branch feature/externalnode, agent may
not set the content for AntreaAgentInfo when it starts if controller has not created the AntreaAgentInfo.
In this case, it will take a minute for agent to update the content in its next try.

I have merged a change which adds retry logics in the antctl proxy e2e test and it will only invoke the command
after AntreaAgentInfo is ready.

I haven't seen the failure with the change in the latest several runs, mark it fixed.