Instructions for running tests appear to be wrong for Kata2

[dgibson]

Description of problem

In the 2.0-dev branch README.md gives instructions for running the tests which amount to cloning the tests package, running .ci/setup.sh then running:

$ export RUNTIME=kata-runtime
$ export KATA_DEV_MODE=true
$ sudo -E PATH=$PATH make test

Expected result

Tests run, and pass.

Actual result

# make test
ln -sf . vendor/src
GOPATH=/root/go/src/github.com/kata-containers/tests/vendor go build ./vendor/github.com/onsi/ginkgo/ginkgo
unlink vendor/src
./ginkgo -failFast -v -focus "" -skip "" \
	functional/ -- -runtime=kata-runtime -timeout=120
Running command '/usr/local/bin/kata-runtime [kata-runtime --kata-show-default-config-paths]'
[kata-runtime --kata-show-default-config-paths]
Timeout: 120 seconds
Exit Code: 0
Stdout: /etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Stderr: 
2021/01/14 16:43:59 '-hypervisor' to ginkgo is not set, using 'DefaultHypervisor': 'qemu'
Running Suite: Functional Suite
===============================
Random Seed: 1610603038
Will run 8 of 8 specs

state container 
  with workload [true], timeWait 5
  /root/go/src/github.com/kata-containers/tests/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:43
Running command '/usr/local/bin/kata-runtime [kata-runtime --log=/tmp/bundle597351757/log run --bundle=/tmp/bundle597351757 --console= --pid-file=/tmp/bundle597351757/pid --detach 9Xfpcs28ZCppzdVGSVU1]'
command failed error 'exit status 1'
[kata-runtime --log=/tmp/bundle597351757/log run --bundle=/tmp/bundle597351757 --console= --pid-file=/tmp/bundle597351757/pid --detach 9Xfpcs28ZCppzdVGSVU1]
Timeout: 120 seconds
Exit Code: 1
Stdout: 
Stderr: Invalid command "run"

Running command '/usr/local/bin/kata-runtime [kata-runtime list --quiet]'
command failed error 'exit status 1'
[kata-runtime list --quiet]
Timeout: 120 seconds
Exit Code: 1
Stdout: 
Stderr: Invalid command "list"

2021/01/14 16:43:59 Could not determine if proxy is running: proxy path is empty

Ginkgo ran 1 suite in 1.006629438s
Test Suite Failed
make: *** [Makefile:86: functional] Error 1

Further information

My test environment is a VM based on a Fedora 33 cloud edition image. Qemu command line is:

$(QEMU) -name $(NAME) \
		-machine q35,kernel_irqchip=split \
		-m $(MEMORY) \
		-cpu host \
		-smp 2 \
		-kernel $(KERNEL) \
		-initrd $(INITRAMFS) \
		-append "console=ttyS0,115200n8 systemd.unified_cgroup_hierarchy=0 intel_iommu=on root=/dev/sda1" \
		-hda $(DISK) \
		-netdev user,id=net0,hostfwd=tcp:127.0.0.1:$(SSH_PORT)-:22 \
		-device virtio-net-pci,netdev=net0 \
		-netdev socket,id=net1,listen=:$(SOCKNET_PORT) \
		-device virtio-net-pci,netdev=net1 \
		-device intel-iommu \
		-vga none \
		-nographic

The VM is set up using this cloud-init configuration

#cloud-config
system_info:
  default_user:
    name: root

resize_rootfs: True
timezone: @TZ@
disable_root: False

ssh_authorized_keys:
  - @KEY@

packages:
  - git
  - strace

write_files:
  - path: /root/.bashrc
    append: true
    content: |
      export GOPATH=~/go
      export PATH=$PATH:$GOPATH/bin:/usr/local/go/bin
  - path: /root/kata-tests-setup.sh
    permissions: '0755'
    content: |
      #! /bin/sh
      set -e
      export USER=root
      export HOME=/root
      export GOPATH=~/go
      export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
      KATASRC=$GOPATH/src/github.com/kata-containers
      mkdir -p $KATASRC
      cd $KATASRC
      git clone https://github.com/kata-containers/kata-containers
      (
          cd kata-containers
          git remote add dwg-github https://github.com/dgibson/kata-containers
          git fetch dwg-github
          git checkout -b test dwg-github/main
      )
      git clone https://github.com/kata-containers/tests
      (
          cd tests
          git checkout -b test origin/2.0-dev
          .ci/setup.sh
      )
      systemctl enable docker
      poweroff

runcmd:
  - [ sh, -c, echo "runcmd setup" ]
  - [ /root/kata-tests-setup.sh ]

final_message: "READY!"

The repo/branch I'm checking out for kata-containers/kata-containers is presently just tracking upstream 2.0-dev with no midifications.

[GabyCT]

@dgibson thanks ... we need to modify it to work properly (documentation and scripts)
/cc @chavafg

[dgibson]

@GabyCT so, in the meantime, how would I go about running tests for Kata2? I'm trying to debug kata-containers/kata-containers#1190, and I don't really know how to even start investigating the CI failures.

[dgibson]

The revised scripts and instructions still don't work for me on a Fedora33 cloud image based VM:

# git rev-parse HEAD
923f6f93c5c250466827ea1853532a79f3021a68
# make test
bash .ci/install_bats.sh
/usr/bin/bats
RUNTIME=kata-runtime ./integration/cri-o/cri-o.sh
~/go/src/github.com/cri-o/cri-o ~/go/src/github.com/kata-containers/tests
M	test/ctr.bats
HEAD is now at 6273bea4c Merge pull request #1681 from chavafg/topic/rm-sleep-1.10
~/go/src/github.com/kata-containers/tests
~/go/src/github.com/cri-o/cri-o/test ~/go/src/github.com/kata-containers/tests
Ensure crio service is stopped before running the tests
Ensure docker service is stopped before running the tests
Running cri-o tests with runtime: kata-runtime
cp: cannot stat '/root/go/src/github.com/cri-o/cri-o/bin/conmon': No such file or directory
Failed at 132: ./test_runner.sh ctr.bats
make: *** [Makefile:127: crio] Error 1