Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

finch fails with "unrecognized system status" #171

Closed
otterley opened this issue Jan 18, 2023 · 7 comments
Closed

finch fails with "unrecognized system status" #171

otterley opened this issue Jan 18, 2023 · 7 comments
Labels
bug Something isn't working

Comments

@otterley
Copy link

Describe the bug
finch can get into a state where, when running any command, it will fail and print

FATA[0000] unrecognized system status

Additional context
Requested by engineers:

❯ LIMA_HOME=/Applications/Finch/lima/data /Applications/Finch/lima/bin/limactl ls
WARN[0000] instance "finch" has errors                   errors="[host agent is running but driver is not]"
NAME     STATUS    SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
finch    Broken    127.0.0.1:53899    qemu      aarch64    3       8GiB      100GiB    /Applications/Finch/lima/data/finch
@otterley otterley added the bug Something isn't working label Jan 18, 2023
@sam-berning
Copy link
Contributor

how did you install finch? via homebrew or the github release?

@otterley
Copy link
Author

otterley commented Jan 18, 2023 •
edited
Loading

Via the GitHub release (pkg file).

@sam-berning
Copy link
Contributor

Had you installed via homebrew previously by any chance?

@otterley
Copy link
Author

No, never.

@sam-berning
Copy link
Contributor

Ok, just wanted to rule out any chance that there were two running instances of Finch using the same disk. That's the only context where we've seen this bug before, but it seems like this might be something else.

In any case, #154 should help prevent this issue from happening. We'll have a release with that PR out soon.

@sam-berning sam-berning mentioned this issue Jan 20, 2023
Merged
1 task
@sam-berning
Copy link
Contributor

Just wanted to add a comment here on how to get back into a good state if you run into this issue. The first thing I would try is to manually delete the Lima instance directory, then re-init the finch VM.

LIMA_HOME=/Applications/Finch/lima/data/finch /Applications/Finch/lima/bin/limactl remove -f finch
finch vm init

If this doesn't fix the issue or if other issues arise with finch images or finch ps, it's possible that the persistent disk that stores containerd user data is corrupted. In this case, you should also delete the persistent disk. Note that this will cause any saved images or containers to be lost.

LIMA_HOME=/Applications/Finch/lima/data/finch /Applications/Finch/lima/bin/limactl remove -f finch
export FINCH_PERSISTENT_DISK=$(readlink /Applications/Finch/lima/data/_disks/finch/datadisk)
LIMA_HOME=/Applications/Finch/lima/data/finch /Applications/Finch/lima/bin/limactl disk delete finch
rm $FINCH_PERSISTENT_DISK
finch vm init

sam-berning added a commit that referenced this issue Jan 27, 2023
@sam-berning
feat: adds a --force flag to vm stop and remove (#178)
d499a7d 
Signed-off-by: Sam Berning <[email protected]>

Issue #, if available: #171

*Description of changes:*

Adds a force flag to `finch vm stop` and `finch vm remove`. This gives
us a native way to recover from "unrecognized system status" as seen in
#171.

*Testing done:*

Unit testing, E2E testing.

Force "unrecognized system status" by messing with the Lima instance
```
$ ./_output/bin/finch vm stop         
FATA[0000] unrecognized system status                   
$ ./_output/bin/finch vm stop --force
INFO[0000] Forcibly stopping Finch virtual machine...   
INFO[0001] Finch virtual machine stopped successfully   
$ ./_output/bin/finch vm start       
INFO[0000] Starting existing Finch virtual machine...   
INFO[0038] Finch virtual machine started successfully
```
```
$ ./_output/bin/finch vm stop         
FATA[0000] unrecognized system status                   
$ ./_output/bin/finch vm remove --force
INFO[0000] Forcibly removing Finch virtual machine...   
INFO[0000] Finch virtual machine removed successfully   
$ ./_output/bin/finch vm status        
Nonexistent
```


- [x] I've reviewed the guidance in CONTRIBUTING.md


#### License Acceptance

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

Signed-off-by: Sam Berning <[email protected]>
@yamatatsu
Copy link

I get same error when I updated and restarted the Mac OS with finch vm running.

Because I had installed with brew, it fixed by uninstalling and re-installing finch with brew.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@otterley @yamatatsu @Shubhranshu153 @sam-berning