[Tracking Issue] Pass command line interfaces tests

[utam0k]

This issue is for tracking PR to pass the command line integration test.
If you are interested, you can comment on this issue and I will assign it to you. This issue is for tracking PR to pass the command line integration test. Some of them may already pass the tests.

• create(assigned @0xdco)
  https://github.com/opencontainers/runtime-tools/tree/master/validation/create
• kill
  https://github.com/opencontainers/runtime-tools/tree/master/validation/kill
• delete(@duduainankai )
  https://github.com/opencontainers/runtime-tools/tree/master/validation/delete
• start(assigned @TinySong)
  https://github.com/opencontainers/runtime-tools/tree/master/validation/start
• state
  https://github.com/opencontainers/runtime-tools/tree/master/validation/state

Goal

Each of them will be targeted to pass integration tests. If possible, it is good to have unit tests as well.
It would be helpful to have a look at the GitHub Actions file to see how to run the integration tests.

[kmpzr]

@utam0k, I'll take create.

[fabjan]

@utam0k all the tasks described (except create) link to https://github.com/opencontainers/runtime-tools/tree/master/validation/kill

[utam0k]

@0xdco Sure, I assigned it to you.

@utam0k, I'll take create.

[utam0k]

Ops, I fixed them. Thanks for pointing that out!

@utam0k all the tasks described (except create) link to https://github.com/opencontainers/runtime-tools/tree/master/validation/kill

[fabjan]

NP! I was actually checking the issues out to see if there was something I could maybe help out with but it seems to me that working on things like this without Linux is not the way to go.

[utam0k]

Thanks for your interest. You can develop by using the devcontainer.
However, I haven't used it recently, so it may take a while to build if it's not well maintained. If you want, you can challenge it.

NP! I was actually checking the issues out to see if there was something I could maybe help out with but it seems to me that working on things like this without Linux is not the way to go.

[bobsongplus]

@utam0k i would like take start, may be will take some time to fix this

[utam0k]

@TinySong Thanks for your interest. I assigned start to you. I'm looking forward to your PR :)

[lizhemingi]

@utam0k I would like to have a try on delete if you don't mind. And it seems that state and kill already pass the test.

[bobsongplus]

@utam0k
there have some problem and don't know how to deal with
according to GitHub Actions file , we use integration_test.sh to integrate test， the script just run runtime-tools to command-line test. but according to the status of the project https://github.com/opencontainers/runtime-tools, the project looks like it hasn't been maintained for a long time， is it necessary to use runtime-tools as youki runtime standard.
for example:
when run start integration test，always failed. After read deeply the runtime-tools code, the line setting process=nil, the create always failed, just return, cannot execute next step, then start cannot pass the integration test

the integration test log:

sudo RUST_BACKTRACE=1 YOUKI_LOG_LEVEL=debug RUNTIME=/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki ./validation/start/start.t
TAP version 13
/tmp/ocitest062540744/config.json
/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki start
ok 1 - start` operation MUST generate an error if it is not provided the container ID
  ---
  {
    "error": "exit status 2",
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.0.2-dev/runtime.md#start",
    "stderr": "error: The following required arguments were not provided:\n    \u003ccontainer-id\u003e\n\nUSAGE:\n    youki start \u003ccontainer-id\u003e\n\nFor more information try --help\n"
  }
  ...
/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki start 0cfaf3a8-94c8-4878-83a0-20eee16388b0
ok 2 - attempting to `start` a container that is not `created` MUST generate an error
  ---
  {
    "error": "exit status 1",
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.0.2-dev/runtime.md#start",
    "stderr": "Error: 0cfaf3a8-94c8-4878-83a0-20eee16388b0 doesn't exists.\n"
  }
  ...
/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki create --bundle /tmp/ocitest062540744 0cfaf3a8-94c8-4878-83a0-20eee16388b0
ok 3 - `process.args` MUST NOT be applied until triggered by the start operation
  ---
  {
    "error": "stat /tmp/ocitest062540744/output: no such file or directory",
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.0.2-dev/runtime.md#create"
  }
  ...
/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki start 0cfaf3a8-94c8-4878-83a0-20eee16388b0
ok 4 - `start` operation MUST run the user-specified program as specified by `process`
  ---
  {
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.0.2-dev/runtime.md#start"
  }
  ...
/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki start 0cfaf3a8-94c8-4878-83a0-20eee16388b0
ok 5 - attempting to `start` a container that is not `created` MUST generate an error
  ---
  {
    "error": "exit status 1",
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.0.2-dev/runtime.md#start",
    "stderr": "[ERROR src/start.rs:28] 2021-06-10T16:35:51.774650795+08:00 0cfaf3a8-94c8-4878-83a0-20eee16388b0 counld not be started because it was Stopped\r\nError: 0cfaf3a8-94c8-4878-83a0-20eee16388b0 counld not be started because it was Stopped\n"
  }
  ...
ok 6 - attempting to `start` a container that is not `created` MUST have no effect on the container
  ---
  {
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.0.2-dev/runtime.md#start"
  }
  ...
 # the below is debug log 
/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki create --bundle /tmp/ocitest062540744 0cfaf3a8-94c8-4878-83a0-20eee16388b0
# create function exit 
create failed exit status 1
not ok 7 - exit status 1

[bobsongplus]

@duduainankai
It seems like that above all command-line can pass except start
can test by this command:
sudo RUST_BACKTRACE=1 YOUKI_LOG_LEVEL=debug RUNTIME=/home/ubuntu/development/rust/youki/target/x86_64-unknown-linux-gnu/debug/youki ./validation/kill/kill.t

[utam0k]

@TinySong runtime-tools is certainly not very active in maintenance.
So I'm thinking of creating an originan integration test.
However, this will take quite some time to complete. So until then, I will use runtime-tools.
#56

If the tests that are failing still fail when using runc, then give up on this test case.

[utam0k]

@utam0k I would like to have a try on delete if you don't mind. And it seems that state and kill already pass the test.

Sure! I assigned you it. For kill and state, can I ask you to give me a PR just to add the test cases?

[bobsongplus]

@TinySong runtime-tools is certainly not very active in maintenance.
So I'm thinking of creating an originan integration test.
However, this will take quite some time to complete. So until then, I will use runtime-tools.
#56

If the tests that are failing still fail when using runc, then give up on this test case.

@utam0k Yeah,I think we can give up the ‘start’ integration test case , testing runc throw out the same error.

[lizhemingi]

@utam0k I would like to have a try on delete if you don't mind. And it seems that state and kill already pass the test.

Sure! I assigned you it. For kill and state, can I ask you to give me a PR just to add the test cases?

Sure.

And I think @TinySong is right, all test cases except start pass the test already.

I think we may add them all.

[utam0k]

@TinySong Thanks for checking the runc case. hmm... Since I have no choice, can I ask you to write down in integration_test.sh that the test does not pass even with runc?

@utam0k Yeah,I think we can give up the ‘start’ integration test case , testing runc throw out the same error.

[utam0k]

@utam0k I would like to have a try on delete if you don't mind. And it seems that state and kill already pass the test.

Sure! I assigned you it. For kill and state, can I ask you to give me a PR just to add the test cases?

Sure.

And I think @TinySong is right, all test cases except start pass the test already.

I think we may add them all.

Thank you for your research. Now, I would like to close this issue by stating that only start does not pass the test.