runtime: support memory hotplug via probe interface on aarch64 #1152

Pennyzct · 2019-01-18T06:25:00Z

Description of problem

There are 2 phases in Memory Hotplug. 1) Physical Memory Hotplug phase. 2) Logical Memory Hotplug phase.
The First phase is to communicate hardware/firmware and make/erase environment for hotplugged memory.
If firmware supports notification of connection of new memory to OS, this phase is triggered automatically. ACPI can notify this event. This also what kata supports, memory hotplug via acpi-driven.
But if not, there is another option, probe operation by hand.
And since memory hotplug via acpi is missing on qemu-system-aarch64, we hope to support the other probe solution.
the whole detailed info about probe interface could be found here.
This whole implementation of memory hotplug via probe interface is divided into two phases, the first is to check if guest kernel is capable of memory hot-add via probe interface, which is located at /sys/devices/system/memory/probe.
The second is let kata-agent tell guest kernel the physical address of new memory block.
echo start_address_of_new_memory > /sys/devices/system/memory/probe

Expected result

root@entos-thunderx2-desktop:~# docker run --runtime kata-runtime -it --rm -m 3G ubuntu
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
root@9ebdb553ba9b:/# lsmem
RANGE                                 SIZE  STATE REMOVABLE     BLOCK
0x0000000040000000-0x00000000bfffffff   2G online        no       1-2
0x0000020000000000-0x00000200bfffffff   3G online        no 2048-2050

Memory block size:         1G
Total online memory:       5G
Total offline memory:      0B
root@9ebdb553ba9b:/# ls /sys/devices/system/memory/
auto_online_blocks  block_size_bytes  memory1  memory2  memory2048  memory2049  memory2050  power  probe  uevent

raravena80 · 2019-02-05T20:21:20Z

@Pennyzct ping, any updates? Thx. Is this waiting for review?

ttx · 2019-02-15T08:03:29Z

/zuul-recheck

jodh-intel · 2019-02-26T11:34:25Z

virtcontainers/kata_agent.go

+func (k *kataAgent) memHotplugByProbe(addr uint64, sizeMB uint32, memorySectionSizeMB uint32) error {
+	// hot-added memory device should be sliced into the size of memory section, which is the basic unit for
+	// memory hotplug
+	numSection := uint64(sizeMB / memorySectionSizeMB)


How about returning an error if memorySectionSizeMB is zero? :)

yes,yes,yes.;) definitely need to avoid divided by zero, update asap. ;)

grahamwhaley · 2019-03-05T11:52:10Z

Hi @Pennyzct - is this PR still alive/valid ?

Pennyzct · 2019-03-06T05:48:04Z

Hi~ @grahamwhaley yes,yes,yes. So sorry for the long-time hanging up, I constantly got sidetracked.😭. since related agent/#443 commit has already been merged, i will dep ensure to include the change. update asap. ;)

Pennyzct · 2019-03-06T09:42:50Z

Hi~ all. When I dep ensure -update github.com/kata-conatiners/agent, do i need to git add the files under vendor/? Since Gopkg.lock and Gopkg.toml is not enough, the travis ci is still failing.

jodh-intel · 2019-03-06T09:54:55Z

Hi @Pennyzct - yes, you need to git add vendor/ after a dep ensure. I'll update the contributor guide as that isn't currently specified...

jodh-intel · 2019-03-06T10:04:17Z

PR raised - kata-containers/community#89.

umarcor · 2019-03-06T10:22:21Z

@Pennyzct, @jodh-intel just because of curiosity, why are you using dep instead of go mod?

grahamwhaley · 2019-03-06T10:25:33Z

@umarcor - at this point, it may be mostly due to historical reasons - the kata, and forerunner, code bases have been around a few years now... I think go mod came in with golang 1.11 ?, and we still have our baseline golang supported version as 1.10.4 :-)

Now, if you'd like to propose a move to go mod, and submit a PR ;-) But, first probably best if it was discussed. /cc @sboeuf

jodh-intel · 2019-03-06T11:03:19Z

Yes - at the time we introduced it, dep was the best tool available.

sboeuf · 2019-03-06T16:31:26Z

@umarcor @grahamwhaley @jodh-intel

I recently discussed this with @mcastelino and I think it is time for us to move to go modules. We need to open issues on every Kata repositories for this. And then let's start submit PRs.

@umarcor is it something you're interested in helping with?

jodh-intel

Thanks @Pennyzct !

lgtm

Ping @kata-containers/runtime.

Pennyzct · 2019-03-12T07:34:34Z

/test

jodh-intel · 2019-03-12T09:59:56Z

Hi @Pennyzct - the CI's are failing on this:

virtcontainers/kata_agent_test.go:262:32: cannot use g (type *gRPCProxy) as type "github.com/kata-containers/runtime/vendor/github.com/kata-containers/agent/protocols/grpc".AgentServiceServer in argument to "github.com/kata-containers/runtime/vendor/github.com/kata-containers/agent/protocols/grpc".RegisterAgentServiceServer:
	*gRPCProxy does not implement "github.com/kata-containers/runtime/vendor/github.com/kata-containers/agent/protocols/grpc".AgentServiceServer (missing MemHotplugByProbe method)
FAIL	github.com/kata-containers/runtime/virtcontainers [build failed]

It looks like you need to add a dummy implementation of MemHotplugByProbe() to virtcontainers/kata_agent_test.go.

Pennyzct · 2019-03-13T02:51:42Z

/test

jodh-intel · 2019-03-13T18:14:44Z

/retest

Pennyzct · 2019-03-15T08:23:59Z

02:33:23 time="2019-03-13T18:33:23Z" level=warning msg="Failed Maxval (59001.600000 < 66545.530000) for [memory-footprint-ksm]"
02:33:23 time="2019-03-13T18:33:23Z" level=warning msg="Check for [memory-footprint-ksm] failed [Failed]"
02:33:23 time="2019-03-13T18:33:23Z" level=warning msg=" with [[*F* memory-footprint-ksm 95.0% 118.4% 105.0% 10.0% 118.4% 118.4% 0.0% 0.0% 1]]"
02:33:23 time="2019-03-13T18:33:23Z" level=warning msg="Overall we failed"
02:33:23 
02:33:23 Report Summary:
02:33:23 +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+
02:33:23 | P/F |         NAME         |  FLR  |  MEAN  |  CEIL  |  GAP  |  MIN   |  MAX   |  RNG  | COV  | ITS |
02:33:23 +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+
02:33:23 | P   | boot-times           | 95.0% | 100.8% | 105.0% | 10.0% | 95.3%  | 109.4% | 14.8% | 3.2% |  20 |
02:33:23 | P   | memory-footprint     | 95.0% | 100.3% | 105.0% | 10.0% | 100.3% | 100.3% | 0.0%  | 0.0% |   1 |
02:33:23 | *F* | memory-footprint-ksm | 95.0% | 118.4% | 105.0% | 10.0% | 118.4% | 118.4% | 0.0%  | 0.0% |   1 |
02:33:23 +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+
02:33:23 Fails: 1, Passes 2
02:33:23 Failed
02:33:23 checkmetrics FAILED (1)
02:33:23 restoring KSM settings

Hi~ @grahamwhaley @jodh-intel metrix ci failed. Do we need to re-trigger or just skip it?

jodh-intel · 2019-03-15T08:42:39Z

Hi @Pennyzct - This PR is possibly being affected by the need to "reset" the metrics thresholds. If so, I think we can merge. Hoever, I've re-triggered to see how that compares to the last run...

/cc @chavafg.

raravena80 · 2019-03-25T22:07:02Z

Hi @Pennyzct any updates? Thx!

In order to support memory hotplug via probe interface in kata-runtime, firstly, we need to verify whether guest kernel is capable of that. Fixes: kata-containers#1149 Signed-off-by: Penny Zheng <[email protected]>

If kata-runtime supports memory hotplug via probe interface, we need to reconstruct memoryDevice to store relevant status, which are addr and probe. addr specifies the physical address of the memory device, and probe determines it is hotplugged via acpi-driven or probe interface. Fixes: kata-containers#1149 Signed-off-by: Penny Zheng <[email protected]>

we need to notify guest kernel about memory hot-added event via probe interface. hot-added memory deivce should be sliced into the size of memory section. Fixes: kata-containers#1149 Signed-off-by: Penny Zheng <[email protected]>

We need to change the constraints of kata agent into the memory-hotplug related commit, to keep track of the revisons in kata agent. Fixes: kata-containers#1149 Signed-off-by: Penny Zheng <[email protected]>

Pennyzct · 2019-04-04T09:04:22Z

/test

Pennyzct · 2019-04-04T09:07:33Z

Hi~ guys~ @kata-containers/runtime Do you remember this one? (I indeed totally forget it. 😓) I have already re-based. pls review~thanks. ;)

Pennyzct · 2019-04-08T01:58:57Z

Hi~ @grahamwhaley @jodh-intel @devimc vsock CI is failing.

Running command '/usr/local/bin/kata-runtime [kata-runtime --log=/tmp/bundle160734754/log run --bundle=/tmp/bundle160734754 --console= --pid-file=/tmp/bundle160734754/pid --detach muijkDXHgC48j6oaHysi]'
command failed error 'exit status 1'
[kata-runtime --log=/tmp/bundle160734754/log run --bundle=/tmp/bundle160734754 --console= --pid-file=/tmp/bundle160734754/pid --detach muijkDXHgC48j6oaHysi]
Timeout: 120 seconds
Exit Code: 1
Stdout: 
Stderr: Failed to check if grpc server is working: context deadline exceeded

FWIT, context deadline exceeded is related to this issue #1319, I also addressed a comment there about same error on ARM machine.
I had no clue why initrd CI machine failed. I will let you guys to decide to re-trigger or what~ ;)

grahamwhaley · 2019-04-08T08:53:34Z

@chavafg - that vsock issue is pretty old - did we fix/bypass it? If so, I wonder why we hit it again here?

devimc · 2019-04-08T13:42:54Z

@Pennyzct that error is not related to this pr, I'll take a look later,

Pennyzct mentioned this pull request Jan 18, 2019

guest kernel: add memory hotplug support for arm64 using "probe" kata-containers/packaging#309

Closed

Pennyzct force-pushed the memory_hotplug branch from 5d4319f to 49d52ac Compare January 30, 2019 10:43

ttx added wip and removed wip labels Feb 15, 2019

Pennyzct mentioned this pull request Feb 19, 2019

QMP command failed: 'pc-dimm' is not a valid device model name #1245

Closed

jodh-intel reviewed Feb 26, 2019

View reviewed changes

Pennyzct force-pushed the memory_hotplug branch 2 times, most recently from 9ae365b to b8621a8 Compare March 6, 2019 09:21

jodh-intel mentioned this pull request Mar 6, 2019

Update contributor guide with more re-vendor details kata-containers/community#88

Closed

umarcor mentioned this pull request Mar 6, 2019

Migration from 'go dep' to 'go mod' #1331

Closed

jodh-intel approved these changes Mar 11, 2019

View reviewed changes

Pennyzct force-pushed the memory_hotplug branch from b8621a8 to 994ec50 Compare March 12, 2019 07:19

Pennyzct force-pushed the memory_hotplug branch from 994ec50 to 75c2b78 Compare March 13, 2019 01:50

This was referenced Mar 26, 2019

Add agent tracing support #1369

Closed

Add agent trace support #1442

Merged

Pennyzct added 4 commits April 4, 2019 17:03

agent: acquire memory hotplug probe info via GetGuestDetails

30a6a7d

In order to support memory hotplug via probe interface in kata-runtime, firstly, we need to verify whether guest kernel is capable of that. Fixes: kata-containers#1149 Signed-off-by: Penny Zheng <[email protected]>

agent: add interface memHotplugByProbe

3bfcdf7

we need to notify guest kernel about memory hot-added event via probe interface. hot-added memory deivce should be sliced into the size of memory section. Fixes: kata-containers#1149 Signed-off-by: Penny Zheng <[email protected]>

arm64: dep ensure to keep track of the revisons in kata agent

b50292a

We need to change the constraints of kata agent into the memory-hotplug related commit, to keep track of the revisons in kata agent. Fixes: kata-containers#1149 Signed-off-by: Penny Zheng <[email protected]>

Pennyzct force-pushed the memory_hotplug branch from 75c2b78 to b50292a Compare April 4, 2019 09:03

devimc approved these changes Apr 4, 2019

View reviewed changes

devimc merged commit 34e2064 into kata-containers:master Apr 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: support memory hotplug via probe interface on aarch64 #1152

runtime: support memory hotplug via probe interface on aarch64 #1152

Pennyzct commented Jan 18, 2019 •

edited

Loading

raravena80 commented Feb 5, 2019

ttx commented Feb 15, 2019

jodh-intel Feb 26, 2019

Pennyzct Mar 6, 2019

grahamwhaley commented Mar 5, 2019

Pennyzct commented Mar 6, 2019

Pennyzct commented Mar 6, 2019

jodh-intel commented Mar 6, 2019

jodh-intel commented Mar 6, 2019

umarcor commented Mar 6, 2019

grahamwhaley commented Mar 6, 2019

jodh-intel commented Mar 6, 2019

sboeuf commented Mar 6, 2019

jodh-intel left a comment

Pennyzct commented Mar 12, 2019

jodh-intel commented Mar 12, 2019

Pennyzct commented Mar 13, 2019

jodh-intel commented Mar 13, 2019

Pennyzct commented Mar 15, 2019

jodh-intel commented Mar 15, 2019

raravena80 commented Mar 25, 2019

Pennyzct commented Apr 4, 2019

Pennyzct commented Apr 4, 2019

Pennyzct commented Apr 8, 2019

grahamwhaley commented Apr 8, 2019

devimc commented Apr 8, 2019

runtime: support memory hotplug via probe interface on aarch64 #1152

runtime: support memory hotplug via probe interface on aarch64 #1152

Conversation

Pennyzct commented Jan 18, 2019 • edited Loading

Description of problem

Expected result

raravena80 commented Feb 5, 2019

ttx commented Feb 15, 2019

jodh-intel Feb 26, 2019

Choose a reason for hiding this comment

Pennyzct Mar 6, 2019

Choose a reason for hiding this comment

grahamwhaley commented Mar 5, 2019

Pennyzct commented Mar 6, 2019

Pennyzct commented Mar 6, 2019

jodh-intel commented Mar 6, 2019

jodh-intel commented Mar 6, 2019

umarcor commented Mar 6, 2019

grahamwhaley commented Mar 6, 2019

jodh-intel commented Mar 6, 2019

sboeuf commented Mar 6, 2019

jodh-intel left a comment

Choose a reason for hiding this comment

Pennyzct commented Mar 12, 2019

jodh-intel commented Mar 12, 2019

Pennyzct commented Mar 13, 2019

jodh-intel commented Mar 13, 2019

Pennyzct commented Mar 15, 2019

jodh-intel commented Mar 15, 2019

raravena80 commented Mar 25, 2019

Pennyzct commented Apr 4, 2019

Pennyzct commented Apr 4, 2019

Pennyzct commented Apr 8, 2019

grahamwhaley commented Apr 8, 2019

devimc commented Apr 8, 2019

Pennyzct commented Jan 18, 2019 •

edited

Loading