Antrea supports collecting support bundle tarballs, which include the information from Antrea Controller and Antrea Agent. The collected information can help debugging issues in the Kubernetes cluster.
Be aware that the generated support bundle includes a lot of information, including logs, so please review the contents before sharing it on Github and ensure that you do not share any sensitive information.
There are two ways of generating support bundles. Firstly, you can run antctl supportbundle
directly in the Antrea Agent Pod, Antrea Controller Pod, or on a host with a
kubeconfig file for the target cluster. Secondly, you can also apply
SupportBundleCollection CRs to create support bundles for K8s Nodes
or external Nodes. We name this feature as SupportBundleCollection in Antrea.
The details are provided in section Usage examples.
The antctl supportbundle command is supported in Antrea since version 0.7.0.
The SupportBundleCollection CRD is introduced in Antrea v1.10.0 as an alpha
feature. The feature gate must be enabled in both antrea-controller and
antrea-agent configurations. If you plan to collect support bundle on an external
Node, you should enable it in the configuration on the external Node as well.
antrea-agent.conf: |
featureGates:
# Enable collecting support bundle files with SupportBundleCollection CRD.
SupportBundleCollection: true antrea-controller.conf: |
featureGates:
# Enable collecting support bundle files with SupportBundleCollection CRD.
SupportBundleCollection: trueA single Namespace (e.g., default) is created for saving the Secrets that are used to access the support bundle file server, and the permission to read Secrets in this Namespace is given to antrea-controller by modifying and applying the RBAC file.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: antrea-read-secrets
namespace: default # Change the Namespace to where the Secret for file server's authentication credential is created.
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: antrea-secret-reader
subjects:
- kind: ServiceAccount
name: antrea-controller
namespace: kube-systemSupportBundleCollection CRD is introduced to supplement the antctl command
with three additional features:
- Allow users to collect support bundle files on external Nodes.
- Upload all the support bundle files into a user-provided SFTP Server.
- Support tracking status of a SupportBundleCollection CR.
Please refer to the antctl user guide section.
Note: antctl supportbundle only works on collecting the support bundles from
Antrea Controller and Antrea Agent that is running on a K8s Node, but it does
not work for the Agent on an external Node.
In this section, we will create two SupportBundleCollection CRs for K8s Nodes and external Nodes. Note, it is supported to specify Nodes/ExternalNodes by their names or by matching their labels in a SupportBundleCollection CR.
Assume we have a cluster with Nodes named "worker1" and "worker2". In addition,
we have set up two external Nodes named "vm1" and "vm2" in the "vm-ns" Namespace
by following the instruction of the VM installation guide.
In addition, an SFTP server needs to be provided in advance to collect the bundle.
You can host the SFTP server by applying YAML hack/externalnode/sftp-deployment.yml
or deploy one by yourself.
A Secret needs to be created in advance with the username and password of the SFTP
Server. The Secret will be referred as authSecret in the following YAML examples.
# Set username and password with `--from-literal=username='foo' --from-literal=password='pass'`
# if the sftp server is deployed with sftp-deployment.yml
kubectl create secret generic support-bundle-secret --from-literal=username='your-sftp-username' --from-literal=password='your-sftp-password'Then we can apply the following YAML files. The first one is to collect support bundle on K8s Nodes "worker1" and "worker2": "worker1" is specified by the name, and "worker2" is specified by matching label "role: workers". The second one is to collect support bundle on external Nodes "vm1" and "vm2" in Namespace "vm-ns": "vm1" is specified by the name, and "vm2" is specified by matching label "role: vms".
cat << EOF | kubectl apply -f -
apiVersion: crd.antrea.io/v1alpha1
kind: SupportBundleCollection
metadata:
name: support-bundle-for-nodes
spec:
nodes: # All Nodes will be selected if both nodeNames and matchLabels are empty
nodeNames:
- worker1
matchLabels:
role: workers
expirationMinutes: 10 # expirationMinutes is the requested duration of validity of the SupportBundleCollection. A SupportBundleCollection will be marked as Failed if it does not finish before expiration.
sinceTime: 2h # Collect the logs in the latest 2 hours. Collect all available logs if the time is not set.
fileServer:
url: sftp://yourtestdomain.com:22/root/test
authentication:
authType: "BasicAuthentication"
authSecret:
name: support-bundle-secret
namespace: default # antrea-controller must be given the permission to read Secrets in "default" Namespace.
EOFcat << EOF | kubectl apply -f -
apiVersion: crd.antrea.io/v1alpha1
kind: SupportBundleCollection
metadata:
name: support-bundle-for-vms
spec:
externalNodes: # All ExternalNodes in the Namespace will be selected if both nodeNames and matchLabels are empty
nodeNames:
- vm1
nodeSelector:
matchLabels:
role: vms
namespace: vm-ns # namespace is mandatory when collecting support bundle from external Nodes.
fileServer:
url: yourtestdomain.com:22/root/test # Scheme sftp can be omitted. The url of "$controlplane_node_ip:30010/upload" is used if deployed with sftp-deployment.yml.
authentication:
authType: "BasicAuthentication"
authSecret:
name: support-bundle-secret
namespace: default # antrea-controller must be given the permission to read Secrets in "default" Namespace.
EOFFor more information about the supported fields in a "SupportBundleCollection" CR, please refer to the CRD definition
You can check the status of SupportBundleCollection by running command
kubectl get supportbundlecollections [NAME] -ojson.
The following example shows a successful realization of SupportBundleCollection.
desiredNodes shows the expected number of Nodes/ExternalNodes to collect with
this request, while collectedNodes shows the number of Nodes/ExternalNodes
which have already uploaded bundle files to the target file server. If the
collection completes successfully, collectedNodes and desiredNodesshould
have an equal value which should match the number of Nodes/ExternalNodes you
want to collect support bundle.
If the following two conditions are presented, it means a bundle collection succeeded,
- "Completed" is true
- "CollectionFailure" is false.
If any expected Node/ExternalNode failed to upload the bundle files in the required time, the "CollectionFailure" condition will be set to true.
$ kubectl get supportbundlecollections support-bundle-for-nodes -ojson
...
"status": {
"collectedNodes": 1,
"conditions": [
{
"lastTransitionTime": "2022-12-08T06:49:35Z",
"status": "True",
"type": "Started"
},
{
"lastTransitionTime": "2022-12-08T06:49:41Z",
"status": "True",
"type": "BundleCollected"
},
{
"lastTransitionTime": "2022-12-08T06:49:35Z",
"status": "False",
"type": "CollectionFailure"
},
{
"lastTransitionTime": "2022-12-08T06:49:41Z",
"status": "True",
"type": "Completed"
}
],
"desiredNodes": 1
}The collected bundle should include three tarballs. To access these files, you
can download the files from the SFTP server yourtestdomain.com. There will be
two tarballs for support-bundle-for-nodes: "support-bundle-for-nodes_worker1.tar.gz"
and "support-bundle-for-nodes_worker2.tar.gz", and two for support-bundle-for-vms:
"support-bundle-for-vms_vm1.tar.gz" and "support-bundle-for-vms_vm2.tar.gz", in
the /root/test folder. Run the tar xvf $TARBALL_NAME command to extract the
files from the tarballs.
Depending on the methods you use to collect the support bundle, the contents in the bundle may differ. The following table shows the differences.
We use agent,controller, outside to represent running command
antctl supportbundle in Antrea Agent, Antrea Controller, out-of-cluster
respectively. Also, we use Node and ExternalNode to represent
"create SupportBundleCollection CR for Nodes" and "create SupportBundleCollection
CR for external Nodes".
| Collected Item | Supported Collecting Method | Explanation |
|---|---|---|
| Antrea Agent Log | agent, outside, Node, ExternalNode |
Antrea Agent log files |
| Antrea Controller Log | controller, outside |
Antrea Controller log files |
| iptables (Linux Only) | agent, outside, Node, ExternalNode |
Output of ip6tables-save and iptable-save with counters |
| OVS Ports | agent, outside, Node, ExternalNode |
Output of ovs-ofctl dump-ports-desc |
| NetworkPolicy Resources | agent, controller, outside, Node, ExternalNode |
YAML output of antctl get appliedtogroups and antctl get addressgroups commands |
| Heap Pprof | agent, controller, outside, Node, ExternalNode |
Output of pprof.WriteHeapProfile |
| HNSResources (Windows Only) | agent, outside, Node, ExternalNode |
Output of Get-HNSNetwork and Get-HNSEndpoint commands |
| Antrea Agent Info | agent, outside, Node, ExternalNode |
YAML output of antctl get agentinfo |
| Antrea Controller Info | controller, outside |
YAML output of antctl get controllerinfo |
| IP Address Info | agent, outside, Node, ExternalNode |
Output of ip address command on Linux or ipconfig /all command on Windows |
| IP Route Info | agent, outside, Node, ExternalNode |
Output of ip route on Linux or route print on Windows |
| IP Link Info | agent, outside, Node, ExternalNode |
Output of ip link on Linux or Get-NetAdapter on Windows |
| Cluster Information | outside |
Dump of resources in the cluster, including: 1. all Pods, Deployments, Replicasets and Daemonsets in all Namespaces with any resourceVersion. 2. all Nodes with any resourceVersion. 3. all ConfigMaps in all Namespaces with any resourceVersion and label app=antrea. |
| Memberlist State | agent, outside |
YAML output of antctl get memberlist |
Only SFTP basic authentication is supported for SupportBundleCollection. Other authentication methods will be added in the future.