VMCache: the new function that creates VMs as caches before using it #1166

teawater · 2019-01-23T15:57:02Z

VM cache helps speeding up new container creation.
To use it, need set option "enable_vm_cache" to true and use
"kata-runtime cache" command start the VM cache server that created
some VMs as VM cache. Then each kata-runtime will request VM from
VM cache server.

Currently, VM cache still cannot work with VM templating and vsock.
And just support qemu.

Fixes: #52

Signed-off-by: Hui Zhu [email protected]

cli/cache.go

teawater · 2019-01-30T02:51:19Z

/test

teawater · 2019-01-30T03:08:21Z

/test

teawater · 2019-01-31T07:25:03Z

/test

raravena80 · 2019-02-05T20:21:51Z

@teawater ping, any updates? Thx!

teawater · 2019-02-10T14:55:34Z

/test

teawater · 2019-02-10T15:11:57Z

@raravena80 I am fixing the test issue after Chinese new year.

teawater · 2019-02-11T01:20:47Z

/test

teawater · 2019-02-11T13:11:42Z

jenkins-ci-ARM-ubuntu-18-04 fail relate with #1202.
jenkins-ci-ubuntu-16-04-firecracker fail relate with #1221.
So I think these 2 fails do not relate with this pr.

teawater · 2019-02-11T13:12:57Z

@amshinde @jodh-intel I pushed new comments according to your comments. Please help me review it.

teawater · 2019-02-25T03:57:25Z

/test

teawater · 2019-02-26T06:25:47Z

jenkins-metrics-ubuntu-16-04 failed:

Report Summary:
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| P/F |         NAME         |  FLR  |  MEAN  |  CEIL  |  GAP  |  MIN   |  MAX   | RNG  | COV  | ITS |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| *F* | boot-times           | 94.1% | 127.5% | 105.9% | 11.9% | 126.4% | 129.1% | 2.1% | 0.6% |  20 |
| *F* | memory-footprint     | 95.0% | 110.5% | 105.0% | 10.0% | 110.5% | 110.5% | 0.0% | 0.0% |   1 |
| P   | memory-footprint-ksm | 95.0% | 103.3% | 105.0% | 10.0% | 103.3% | 103.3% | 0.0% | 0.0% |   1 |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
Fails: 2, Passes 1
Failed
checkmetrics FAILED (1)

This pr doesn't change the default behavior of kata-runtime.
So I think this fail does not relate with this fail.

grahamwhaley · 2019-02-26T09:51:14Z

@teawater - ah, yes, since the 4.19 kernel update landed, the metrics will need adjusting. I had to leave it a few days to gather data so I can reset the bounds checks - let me try to update those today...
In the meantime, ignore the metrics CI....

bergwolf

Nice work @teawater !

bergwolf · 2019-03-01T04:20:23Z

pkg/katautils/config.go

+			return errors.New("VM cache just support kata agent")
+		}
+		if config.HypervisorConfig.UseVSock {
+			return errors.New("config vsock conflicts with VM cache, please disable one of them")


It does not really conflict. Need to send the guest vsock cid/port over cache server grpc. Maybe a todo in future PR.

jodh-intel

Thanks @teawater.

lgtm

Is there an issue in 'tests' to create CI tests for this feature?

jodh-intel · 2019-03-01T10:29:08Z

cli/factory.go

+	if err != nil {
+		return nil, err
+	}
+	if err = os.Chmod(path, 0660); err != nil {


Does this need to be 660? Could we use 600 instead?

Updated.

And I opened an issue for needing CI test for vmcache in kata-containers/tests#1259

Thanks @teawater !

teawater · 2019-03-01T15:16:17Z

/test

teawater · 2019-03-02T12:44:57Z

jenkins-ci-ARM-ubuntu-18-04 fail:

INFO: creating OCI bundle in /tmp/kata-runtime-944448708/bundle for tests to use
time="2019-03-02T10:14:34+08:00" level=debug msg="converting /tmp/kata-runtime-944448708/bundle/config.json" source=virtcontainers subsystem=oci
=== RUN   TestConsoleFromFile
--- PASS: TestConsoleFromFile (0.00s)
=== RUN   TestNewConsole
--- FAIL: TestNewConsole (0.00s)
    assertions.go:239: 
                          
	Error Trace:	console_test.go:28
        
	Error:      	Received unexpected error:
        
	            	open /dev/ptmx: no such device
        
	Messages:   	failed to create a new console: open /dev/ptmx: no such device
panic: runtime error: invalid memory address or nil pointer dereference
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xacb5fc]

goroutine 101 [running]:
testing.tRunner.func1(0x40003ce000)
	/usr/local/go/src/testing/testing.go:792 +0x30c
panic(0xc2e900, 0x1560890)
	/usr/local/go/src/runtime/panic.go:513 +0x18c
github.com/kata-containers/runtime/cli.(*Console).Close(0x0, 0x4000344f80, 0x2)
	/home/jenkins/workspace/kata-containers-runtime-ARM-18.04-PR/go/src/github.com/kata-containers/runtime/cli/console.go:88 +0x44
panic(0xc2e900, 0x1560890)
	/usr/local/go/src/runtime/panic.go:513 +0x18c
github.com/kata-containers/runtime/cli.(*Console).Path(...)
	/home/jenkins/workspace/kata-containers-runtime-ARM-18.04-PR/go/src/github.com/kata-containers/runtime/cli/console.go:73
github.com/kata-containers/runtime/cli.TestNewConsole(0x40003ce000)
	/home/jenkins/workspace/kata-containers-runtime-ARM-18.04-PR/go/src/github.com/kata-containers/runtime/cli/console_test.go:31 +0xf0
testing.tRunner(0x40003ce000, 0xd89c98)
	/usr/local/go/src/testing/testing.go:827 +0xa8
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:878 +0x2b0
FAIL	github.com/kata-containers/runtime/cli	1.682s
Makefile:532: recipe for target 'go-test' failed
make: *** [go-test] Error 1
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script  : #!/bin/bash

This PR doesn't change the default behavior of kata-runtime execution. So it has nothing to do with this failure.

teawater · 2019-03-03T05:57:38Z

@sboeuf please review the new version. Thanks.

sboeuf · 2019-03-05T16:54:03Z

@teawater I will review this today!

sboeuf

@teawater I have one comment regarding the vendoring.

Also, the code looks fine to me, but I thought you would add some documentation. Have you created an *.md file somewhere? Maybe the documentation repo?

We need to make sure documentation gets merged along with this PR.

sboeuf · 2019-03-05T19:33:10Z

Gopkg.toml

@@ -74,6 +74,10 @@
  name = "github.com/firecracker-microvm/firecracker-go-sdk"
  revision = "961461227bddf7e40a1d690634e866c343910f86"

+[[constraint]]
+  name = "github.com/gogo/protobuf"
+  revision = "342cbe0a04158f6dcb03ca0079991a51a4248c02"


I would expect that after you add the constraint and run dep ensure -update, you would have a modified Gopkg.lock that also needs to be committed here.

I would expect that after you add the constraint and run dep ensure -update, you would have a modified Gopkg.lock that also needs to be committed here.

It is already in

runtime/Gopkg.lock

Line 341 in ec6a1cc

name = "github.com/gogo/protobuf"

because it's a sub-dependency.

Ok I get that, but I was expecting that since you're modifying the Gopkg.toml, running dep ensure -update would at least modify the digest of the Gopkg.lock:

[[projects]] digest = "1:18108594151654e9e696b27b181b953f9a90b16bf14d253dd1b397b025a1487f"

Ok I get that, but I was expecting that since you're modifying the Gopkg.toml, running dep ensure -update would at least modify the digest of the Gopkg.lock:

[[projects]] digest = "1:18108594151654e9e696b27b181b953f9a90b16bf14d253dd1b397b025a1487f"

Done.

teawater · 2019-03-07T08:19:53Z

@teawater I have one comment regarding the vendoring.

Also, the code looks fine to me, but I thought you would add some documentation. Have you created an *.md file somewhere? Maybe the documentation repo?

We need to make sure documentation gets merged along with this PR.

Add a pr kata-containers/documentation#393 for vmcache.

sboeuf · 2019-03-07T16:20:00Z

Thanks for opening kata-containers/documentation#393 @teawater 👍

VMCache is a new function that creates VMs as caches before using it. It helps speed up new container creation. The function consists of a server and some clients communicating through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. The VMCache server will create some VMs and cache them by factory cache. It will convert the VM to gRPC format and transport it when gets requestion from clients. Factory grpccache is the VMCache client. It will request gRPC format VM and convert it back to a VM. If VMCache function is enabled, kata-runtime will request VM from factory grpccache when it creates a new sandbox. VMCache has two options. vm_cache_number specifies the number of caches of VMCache: unspecified or == 0 --> VMCache is disabled > 0 --> will be set to the specified number vm_cache_endpoint specifies the address of the Unix socket. This commit just includes the core and the client of VMCache. Currently, VM cache still cannot work with VM templating and vsock. And just support qemu. Fixes: kata-containers#52 Signed-off-by: Hui Zhu <[email protected]>

When VMCache is enabled, factory init will run as a VMcache server. Fixes: kata-containers#52 Signed-off-by: Hui Zhu <[email protected]>

teawater · 2019-03-08T03:42:15Z

/test

VMCache code use github.com/gogo/protobuf. Fixes: kata-containers#52 Signed-off-by: Hui Zhu <[email protected]>

teawater · 2019-03-08T13:18:18Z

/test

jodh-intel reviewed Jan 24, 2019

View reviewed changes

cli/cache.go Outdated Show resolved Hide resolved

cli/cache.go Outdated Show resolved Hide resolved

cli/cache.go Outdated Show resolved Hide resolved

cli/cache.go Outdated Show resolved Hide resolved

cli/cache.go Outdated Show resolved Hide resolved

teawater force-pushed the vm_cache branch 5 times, most recently from 2b8d953 to 1e7bb03 Compare January 25, 2019 09:44

amshinde reviewed Jan 26, 2019

View reviewed changes

cli/cache.go Outdated Show resolved Hide resolved

cli/cache.go Outdated Show resolved Hide resolved

cli/cache.go Outdated Show resolved Hide resolved

cli/cache.go Outdated Show resolved Hide resolved

teawater force-pushed the vm_cache branch 3 times, most recently from e8380e3 to 151ab6d Compare January 30, 2019 02:40

teawater force-pushed the vm_cache branch from 151ab6d to 97e6ef8 Compare January 30, 2019 02:59

teawater force-pushed the vm_cache branch 4 times, most recently from 7c9e5a5 to 13ff7e3 Compare January 30, 2019 14:05

teawater mentioned this pull request Feb 1, 2019

jenkins-ci-ARM-ubuntu-18-04 always fail #1206

Closed

teawater force-pushed the vm_cache branch 3 times, most recently from 819a0b3 to 41e2688 Compare February 10, 2019 15:45

teawater force-pushed the vm_cache branch 2 times, most recently from 48df99c to 5b351c5 Compare February 12, 2019 10:34

teawater force-pushed the vm_cache branch from a1b164b to 57d507f Compare February 25, 2019 03:44

teawater requested a review from bergwolf February 25, 2019 07:27

bergwolf approved these changes Mar 1, 2019

View reviewed changes

jodh-intel approved these changes Mar 1, 2019

View reviewed changes

teawater removed the request for review from sboeuf March 1, 2019 15:03

teawater force-pushed the vm_cache branch from 57d507f to e0e781b Compare March 1, 2019 15:11

teawater mentioned this pull request Mar 1, 2019

Need test for vmcache function kata-containers/tests#1259

Closed

teawater mentioned this pull request Mar 2, 2019

jenkins-ci-ARM-ubuntu-18-04 cannot pass unit test #1287

Closed

sboeuf suggested changes Mar 5, 2019

View reviewed changes

gnawux requested a review from sboeuf March 6, 2019 02:43

jodh-intel mentioned this pull request Mar 7, 2019

howto: Add VMCache introduction and guide kata-containers/documentation#393

Merged

sboeuf approved these changes Mar 7, 2019

View reviewed changes

teawater added 2 commits March 8, 2019 10:05

VMCache: Update factory to run as a VMCache server

0f8b2ad

When VMCache is enabled, factory init will run as a VMcache server. Fixes: kata-containers#52 Signed-off-by: Hui Zhu <[email protected]>

teawater force-pushed the vm_cache branch 2 times, most recently from 4e51ca5 to 39272d3 Compare March 8, 2019 03:35

vendor: Add github.com/gogo/protobuf

76d9db3

VMCache code use github.com/gogo/protobuf. Fixes: kata-containers#52 Signed-off-by: Hui Zhu <[email protected]>

teawater force-pushed the vm_cache branch from 39272d3 to 76d9db3 Compare March 8, 2019 12:59

sboeuf merged commit 80cdf89 into kata-containers:master Mar 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VMCache: the new function that creates VMs as caches before using it #1166

VMCache: the new function that creates VMs as caches before using it #1166

teawater commented Jan 23, 2019

teawater commented Jan 30, 2019

teawater commented Jan 30, 2019

teawater commented Jan 31, 2019

raravena80 commented Feb 5, 2019

teawater commented Feb 10, 2019

teawater commented Feb 10, 2019

teawater commented Feb 11, 2019

teawater commented Feb 11, 2019

teawater commented Feb 11, 2019

teawater commented Feb 25, 2019

teawater commented Feb 26, 2019

grahamwhaley commented Feb 26, 2019

bergwolf left a comment

bergwolf Mar 1, 2019

jodh-intel left a comment

jodh-intel Mar 1, 2019

teawater Mar 1, 2019

jodh-intel Mar 1, 2019

teawater commented Mar 1, 2019

teawater commented Mar 2, 2019

teawater commented Mar 3, 2019

sboeuf commented Mar 5, 2019

sboeuf left a comment

sboeuf Mar 5, 2019

teawater Mar 7, 2019

sboeuf Mar 7, 2019

teawater Mar 8, 2019

sboeuf Mar 8, 2019

teawater commented Mar 7, 2019

sboeuf commented Mar 7, 2019

teawater commented Mar 8, 2019

teawater commented Mar 8, 2019

VMCache: the new function that creates VMs as caches before using it #1166

VMCache: the new function that creates VMs as caches before using it #1166

Conversation

teawater commented Jan 23, 2019

teawater commented Jan 30, 2019

teawater commented Jan 30, 2019

teawater commented Jan 31, 2019

raravena80 commented Feb 5, 2019

teawater commented Feb 10, 2019

teawater commented Feb 10, 2019

teawater commented Feb 11, 2019

teawater commented Feb 11, 2019

teawater commented Feb 11, 2019

teawater commented Feb 25, 2019

teawater commented Feb 26, 2019

grahamwhaley commented Feb 26, 2019

bergwolf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teawater commented Mar 1, 2019

teawater commented Mar 2, 2019

teawater commented Mar 3, 2019

sboeuf commented Mar 5, 2019

sboeuf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teawater commented Mar 7, 2019

sboeuf commented Mar 7, 2019

teawater commented Mar 8, 2019

teawater commented Mar 8, 2019