Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Insufficient information in logs to debug issues #43

Open
nirs opened this issue Sep 15, 2024 · 0 comments
Open

Insufficient information in logs to debug issues #43

nirs opened this issue Sep 15, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@nirs
Copy link
Member

nirs commented Sep 15, 2024

Current logs lack basic details to be useful for debugging issues:

  • timestamp - allows synchronizing with other logs
  • connect/disconnect: log more info about the connection (mac address?, ip address?)
  • errors: include more context for the failed options for unexpected errors
    • mac address
    • ip address
    • called function and arguments

Detailed error logging may be too slow, need a way to disable it

The following example logs are from 50 runs, each starting 3 lima k8s clusters, provisioning a complex setup, running some tests, and deleting the clusters. 49 of the runs were successful, 1 failed with:

Error building the cluster.Info for the default configuration: error retrieving Submariner: failed to get API group resources: unable to retrieve the complete list of server APIs: submariner.io/v1alpha1: Get "https://192.168.105.11:6443/apis/submariner.io/v1alpha1": dial tcp 192.168.105.11:6443: i/o timeout

Because the logs lack basic details, there is no way to find the related errors.

socket_vmnet was run by launchd, adding KeepAlive to keep it running after errorrs:
nirs@d92f772

I cleared the logs before starting the launchd daemon so the logs should be only from this run. We can see that the daemon was terminated by signal 13 (EACCESS) once, not enough info in the logs to tell why this happened.

Example stdout

Initializing vmnet.framework (mode 1001)
* vmnet_subnet_mask: 255.255.255.0
* vmnet_mtu: 1500
* vmnet_end_address: 192.168.105.254
* vmnet_start_address: 192.168.105.1
* vmnet_interface_id: 71381640-9A80-4CD1-A1EE-AE746AD8E2BB
* vmnet_max_packet_size: 1514
* vmnet_nat66_prefix: fd9b:5a14:ba57:e3d3::
* vmnet_mac_address: 26:73:89:56:71:b6
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)

Received signal 15
Closing a connection (fd 6)
Closing a connection (fd 5)
Initializing vmnet.framework (mode 1001)
* vmnet_subnet_mask: 255.255.255.0
* vmnet_mtu: 1500
* vmnet_end_address: 192.168.105.254
* vmnet_start_address: 192.168.105.1
* vmnet_interface_id: B9124E69-6DD3-415F-A303-F89236C08AA7
* vmnet_max_packet_size: 1514
* vmnet_nat66_prefix: fd9b:5a14:ba57:e3d3::
* vmnet_mac_address: 62:d9:ba:7a:e3:43
Accepted a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)

Received signal 13
Closing a connection (fd 7)
Closing a connection (fd 6)
Initializing vmnet.framework (mode 1001)
* vmnet_subnet_mask: 255.255.255.0
* vmnet_mtu: 1500
* vmnet_end_address: 192.168.105.254
* vmnet_start_address: 192.168.105.1
* vmnet_interface_id: 36D40E7D-1C36-4B0F-9C63-12D7EDB81215
* vmnet_max_packet_size: 1514
* vmnet_nat66_prefix: fd9b:5a14:ba57:e3d3::
* vmnet_mac_address: 66:c1:18:9f:80:62
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)

Received signal 13
Closing a connection (fd 7)

Example stderr

on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0
writev: No buffer space available
writev: Broken pipe
writev: Broken pipe
writev: Bad file descriptor
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0
vmnet_write: Broken pipe
_on_vmnet_packets_available(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_read: Bad file descriptor
_on_vmnet_packets_available(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_read: Bad file descriptor
writev: Broken pipe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nirs @AkihiroSuda