api: add a CleanupContainer api for VC #1833

lifupan · 2019-06-27T08:47:03Z

When shimv2 was killed by accident, containerd would try to
launch a new shimv2 binarry to cleanup the container. In order
to avoid race condition, the cleanup should be done serialized
in a sandbox. Thus adding a new api to do this by locking the
sandbox.

Fixes:#1832

Signed-off-by: lifupan [email protected]

jodh-intel · 2019-06-27T09:39:13Z

Hi @lifupan - thanks for raising. It looks like you are still changing this so I'll hold off reviewing until it's ready ;)

lifupan · 2019-06-27T09:52:52Z

Hi @jodh-intel it's ready :)

lifupan · 2019-06-27T09:52:59Z

/test

codecov · 2019-06-27T10:42:10Z

Codecov Report

❗ No coverage uploaded for pull request base (master@eb0a3d2). Click here to learn what that means.
The diff coverage is 31.81%.

@@            Coverage Diff            @@
##             master    #1833   +/-   ##
=========================================
  Coverage          ?   51.92%           
=========================================
  Files             ?      107           
  Lines             ?    14446           
  Branches          ?        0           
=========================================
  Hits              ?     7501           
  Misses            ?     6053           
  Partials          ?      892

jodh-intel · 2019-06-27T10:39:53Z

containerd-shim-v2/utils.go

-	if _, err := sandbox.DeleteContainer(cid); err != nil {
-		logrus.WithError(err).WithField("container", cid).Warn("failed to remove container")
-	}
+	rootfs := filepath.Join(bundlePath, "rootfs")


I know this isn't part of your change, but since it seems like it affects cleanup success, why are we not returning the unmount error below?

Aha, maybe missed, I'll add it.

/cc @amshinde

jodh-intel · 2019-06-27T10:40:35Z

virtcontainers/api.go

+// in the sandbox left, do stop the sandbox and delete it. Those serial operations will be done exclusively by
+// locking the sandbox.
+func CleanupContainer(ctx context.Context, sandboxID, containerID string) error {
+	span, ctx := trace(ctx, "ListRoutes")


span, ctx := trace(ctx, "CleanupContainer")

jodh-intel · 2019-06-27T10:45:35Z

virtcontainers/api.go

+	}
+
+	s.stateful = false
+	s.releaseStatelessSandbox()


I think I understand this: releaseStatelessSandbox() only calls Release() for stateless sandboxes. But releaseStatelessSandbox() is a "private" sandbox method (since it starts with a lower-case letter), but bizarrely it calls a public method Release(). So I'm wondering if you should just call s.Release() here? Maybe this is better (safer) in case the implementation of releaseStatelessSandbox() ever changes though.

Yes, you are right.

jodh-intel · 2019-06-27T10:46:54Z

virtcontainers/api.go

+		return err
+	}
+
+	if _, err := s.DeleteContainer(containerID); err != nil {


This will fail if containerID == "" so I wonder if CleanupContainer() itself should check for that scenario as soon as possible to avoid getting to this point and then failing?

jodh-intel · 2019-06-27T10:48:35Z

virtcontainers/api.go

+		return err
+	}
+
+	if len(s.GetAllContainers()) == 0 {


I always prefer to invert the logic to minimise the indent:

if len(s.GetAllContainers()) > 0 { return nil } err = s.Stop() if err != nil { return err } return s.Delete()

gnawux · 2019-06-27T12:49:27Z

I don't agree with this behavior, it prevent us from upgrading the shimv2 easily. Why containerd works this way? Do you have a related issue link, @lifupan?

lifupan · 2019-06-27T12:51:57Z

I don't agree with this behavior, it prevent us from upgrading the shimv2 easily. Why containerd works this way? Do you have a related issue link, @lifupan?

@gnawux
Here is the containerd's PR related with this issue: containerd/containerd#3206

gnawux · 2019-06-27T13:09:31Z

Here is the containerd's PR related with this issue: containerd/containerd#3206

I think we should talk with @Random-Liu for a better solution, this behavior might make our production lives harder.

jcvenegas · 2019-07-26T16:41:25Z

@lifupan ping this PR needs rebase.

Random-Liu · 2019-07-29T06:59:06Z

I think we should talk with @Random-Liu for a better solution, this behavior might make our production lives harder.

When the containerd shim is gone, the container is out of control by containerd, thus containerd kills it.

@gnawux said that you guys want to support live upgrade for containerd shim. I think it is fine to add optional reconnect attempt and a timeout in containerd.

jodh-intel · 2019-07-29T10:57:56Z

Ping @lifupan - branch is conflicted and there are some outstanding comments to address I think.

lifupan · 2019-07-30T01:43:43Z

I think we should talk with @Random-Liu for a better solution, this behavior might make our production lives harder.

When the containerd shim is gone, the container is out of control by containerd, thus containerd kills it.

@gnawux said that you guys want to support live upgrade for containerd shim. I think it is fine to add optional reconnect attempt and a timeout in containerd.

Hi @Random-Liu , It would be great if containerd accept to reconnect attempt when disconnect from shim, but I think for containerd it wouldn't be a small change, except for the reconnect, containerd also needs to reload the task since the connection would be changed, and containerd/cri also needs to deal with the task "wait" expected return. Do you have a good plan
for it?

lifupan · 2019-07-31T02:53:40Z

/test

lifupan · 2019-08-05T11:46:09Z

Hi @chavafg Do you happen to know what's the filesystem on our ci test env, It seems most of the filesystem based flock failed?

grahamwhaley · 2019-08-05T12:34:51Z

@lifupan - is the FS type in the CI matrix table at https://github.com/kata-containers/ci#ci-job-matrix ?
If not, maybe we should add it (either/both host fs type and graph/backing driver).

chavafg · 2019-08-05T13:05:37Z

we use overlay as default storage driver and only use devicemapper for cri-o on ubuntu and for firecracker-docker testing.

I see that the failures are from unit tests:

=== RUN   TestCleanupContainer
time="2019-07-31T03:10:18Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/7f49d00d-1995-4156-8c79-5f5ab24ce138/104/rootfs"
==================
WARNING: DATA RACE
Write at 0x00c00020a390 by goroutine 49:
  runtime.mapassign_faststr()
      /usr/local/go/src/runtime/map_faststr.go:202 +0x0
  github.com/kata-containers/runtime/virtcontainers/store.(*filesystem).lock()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/store/filesystem_backend.go:295 +0x1e2
  github.com/kata-containers/runtime/virtcontainers.rwLockSandbox()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/store/manager.go:274 +0x15f
  github.com/kata-containers/runtime/virtcontainers.CleanupContainer()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api.go:950 +0x167
  github.com/kata-containers/runtime/virtcontainers.TestCleanupContainer.func1()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api_test.go:1739 +0xab

Previous write at 0x00c00020a390 by goroutine 59:
  runtime.mapdelete_faststr()
      /usr/local/go/src/runtime/map_faststr.go:297 +0x0
  github.com/kata-containers/runtime/virtcontainers/store.(*filesystem).unlock()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/store/filesystem_backend.go:311 +0x1ea
  github.com/kata-containers/runtime/virtcontainers.unlockSandbox()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/store/manager.go:279 +0x1b8
  github.com/kata-containers/runtime/virtcontainers.CleanupContainer()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api.go:973 +0x4e8
  github.com/kata-containers/runtime/virtcontainers.TestCleanupContainer.func1()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api_test.go:1739 +0xab

Goroutine 49 (running) created at:
  github.com/kata-containers/runtime/virtcontainers.TestCleanupContainer()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api_test.go:1737 +0x51d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163

Goroutine 59 (finished) created at:
  github.com/kata-containers/runtime/virtcontainers.TestCleanupContainer()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api_test.go:1737 +0x51d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163
==================
==================
WARNING: DATA RACE
Write at 0x00c0003a9818 by goroutine 49:
  github.com/kata-containers/runtime/virtcontainers/store.(*filesystem).lock()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/store/filesystem_backend.go:295 +0x1fa
  github.com/kata-containers/runtime/virtcontainers.rwLockSandbox()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/store/manager.go:274 +0x15f
  github.com/kata-containers/runtime/virtcontainers.CleanupContainer()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api.go:950 +0x167
  github.com/kata-containers/runtime/virtcontainers.TestCleanupContainer.func1()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api_test.go:1739 +0xab

Previous write at 0x00c0003a9818 by goroutine 59:
  [failed to restore the stack]

Goroutine 49 (running) created at:
  github.com/kata-containers/runtime/virtcontainers.TestCleanupContainer()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api_test.go:1737 +0x51d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163

Goroutine 59 (finished) created at:
  github.com/kata-containers/runtime/virtcontainers.TestCleanupContainer()
      /tmp/jenkins/workspace/kata-containers-runtime-cri-containerd-PR/go/src/github.com/kata-containers/runtime/virtcontainers/api_test.go:1737 +0x51d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163
==================
time="2019-07-31T03:10:18Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/7f49d00d-1995-4156-8c79-5f5ab24ce138/1/rootfs"
time="2019-07-31T03:10:18Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/7f49d00d-1995-4156-8c79-5f5ab24ce138/101/rootfs"
time="2019-07-31T03:10:18Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/7f49d00d-1995-4156-8c79-5f5ab24ce138/100/rootfs"
time="2019-07-31T03:10:18Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/7f49d00d-1995-4156-8c79-5f5ab24ce138/103/rootfs"
time="2019-07-31T03:10:18Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/7f49d00d-1995-4156-8c79-5f5ab24ce138/102/rootfs"
--- FAIL: TestCleanupContainer (0.07s)
    testing.go:809: race detected during execution of test

lifupan · 2019-08-16T00:41:44Z

/test

lifupan · 2019-08-22T06:28:52Z

/test

lifupan · 2019-08-22T10:28:17Z

/test

lifupan · 2019-08-23T03:51:04Z

/test

lifupan · 2019-08-23T06:59:06Z

/test

bergwolf · 2019-08-23T11:55:44Z

virtcontainers/api.go

+		return nil
+	}
+
+	if err = s.Stop(true); err != nil && !force {


Since force is passed as parameter, should use it here? Or we can just remove the force parameter and always enable force cleanup.

Hi @bergwolf , Yes, you are right. I had change it using the parameter 'force' instead of 'true'.

When shimv2 was killed by accident, containerd would try to launch a new shimv2 binarry to cleanup the container. In order to avoid race condition, the cleanup should be done serialized in a sandbox. Thus adding a new api to do this by locking the sandbox. Fixes:kata-containers#1832 Signed-off-by: lifupan <[email protected]>

lifupan · 2019-08-24T00:17:34Z

/test

This updates grpc-go vendor package to v1.11.3 release, to fix server.Stop() handling so that server.Serve() does not wait blindly. Full commit list: d11072e (tag: v1.11.3) Change version to 1.11.3 d06e756 clientconn: add support for unix network in DialContext. (kata-containers#1883) 452c2a7 Change version to 1.11.3-dev d89cded (tag: v1.11.2) Change version to 1.11.2 98ac976 server: add grpc.Method function for extracting method from context (kata-containers#1961) 0f5fa28 Change version to 1.11.2-dev 1e2570b (tag: v1.11.1) Change version to 1.11.1 d28faca client: Fix race when using both client-side default CallOptions and per-call CallOptions (kata-containers#1948) 48b7669 Change version to 1.11.1-dev afc05b9 (tag: v1.11.0) Change version to 1.11.0 f2620c3 resolver: keep full unparsed target string if scheme in parsed target is not registered (kata-containers#1943) 9d2250f status: rename Status to GRPCStatus to avoid name conflicts (kata-containers#1944) 2756956 status: Allow external packages to produce status-compatible errors (kata-containers#1927) 0ff1b76 routeguide: reimplement distance calculation dfbefc6 service reflection can lookup enum, enum val, oneof, and field symbols (kata-containers#1910) 32d9ffa Documentation: Fix broken link in rpc-errors.md (kata-containers#1935) d5126f9 Correct Go 1.6 support policy (kata-containers#1934) 5415d18 Add documentation and example of adding details to errors (kata-containers#1915) 57640c0 Allow storing alternate transport.ServerStream implementations in context (kata-containers#1904) 031ee13 Fix Test: Update the deadline since small deadlines are prone to flakes on Travis. (kata-containers#1932) 2249df6 gzip: Add ability to set compression level (kata-containers#1891) 8124abf credentials/alts: Remove the enable_untrusted_alts flag (kata-containers#1931) b96718f metadata: Fix bug where AppendToOutgoingContext could modify another context's metadata (kata-containers#1930) 738eb6b fix minor typos and remove grpc.Codec related code in TestInterceptorCanAccessCallOptions (kata-containers#1929) 211a7b7 credentials/alts: Update ALTS "New" APIs (kata-containers#1921) fa28bef client: export types implementing CallOptions for access by interceptors (kata-containers#1902) ec9275b travis: add Go 1.10 and run vet there instead of 1.9 (kata-containers#1913) 13975c0 stream: split per-attempt data from clientStream (kata-containers#1900) 2c2d834 stats: add BeginTime to stats.End (kata-containers#1907) 3a9e1ba Reset ping strike counter right before sending out data. (kata-containers#1905) 90dca43 resolver: always fall back to default resolver when target does not follow URI scheme (kata-containers#1889) 9aba044 server: Convert all non-status errors to codes.Unknown (kata-containers#1881) efcc755 credentials/alts: change ALTS protos to match the golden version (kata-containers#1908) 0843fd0 credentials/alts: fix infinite recursion bug [in custom error type] (kata-containers#1906) 207e276 Fix test race: Atomically access minConnecTimout in testing environment. (kata-containers#1897) 3ae2a61 interop: Add use_alts flag to client and server binaries (kata-containers#1896) 5190b06 ALTS: Simplify "New" APIs (kata-containers#1895) 7c5299d Fix flaky test: TestCloseConnectionWhenServerPrefaceNotReceived (kata-containers#1870) f0a1202 examples: Replace context.Background with context.WithTimeout (kata-containers#1877) a1de3b2 alts: Change ALTS proto package name (kata-containers#1886) 2e7e633 Add ALTS code (kata-containers#1865) 583a630 Expunge error codes that shouldn't be returned from library (kata-containers#1875) 2759199 Small spelling fixes (unknow -> unknown) (kata-containers#1868) 12da026 clientconn: fix a typo in GetMethodConfig documentation (kata-containers#1867) dfa1834 Change version to 1.11.0-dev (kata-containers#1863) 46fd263 benchmarks: add flag to benchmain to use bufconn instead of network (kata-containers#1837) 3926816 addrConn: Report underlying connection error in RPC error (kata-containers#1855) 445b728 Fix data race in TestServerGoAwayPendingRPC (kata-containers#1862) e014063 addrConn: keep retrying even on non-temporary errors (kata-containers#1856) 484b3eb transport: fix race causing flow control discrepancy when sending messages over server limit (kata-containers#1859) 6c48c7f interop test: Expect io.EOF from stream.Send() (kata-containers#1858) 08d6261 metadata: provide AppendToOutgoingContext interface (kata-containers#1794) d50734d Add status.Convert convenience function (kata-containers#1848) 365770f streams: Stop cleaning up after orphaned streams (kata-containers#1854) 7646b53 transport: support stats.Handler in serverHandlerTransport (kata-containers#1840) 104054a Fix connection drain error message (kata-containers#1844) d09ec43 Implement unary functionality using streams (kata-containers#1835) 37346e3 Revert "Add WithResolverUserOptions for custom resolver build options" (kata-containers#1839) 424e3e9 Stream: do not cancel ctx created with service config timeout (kata-containers#1838) f9628db Fix lint error and typo (kata-containers#1843) 0bd008f stats: Fix bug causing trailers-only responses to be reported as headers (kata-containers#1817) 5769e02 transport: remove unnecessary rstReceived (kata-containers#1834) 0848a09 transport: remove redundant check of stream state in Write (kata-containers#1833) c22018a client: send RST_STREAM on client-side errors to prevent server from blocking (kata-containers#1823) 82e9f61 Use keyed fields for struct initializers (kata-containers#1829) 5ba054b encoding: Introduce new method for registering and choosing codecs (kata-containers#1813) 4f7a2c7 compare atomic and mutex performance in case of contention. (kata-containers#1788) b71aced transport: Fix a data race when headers are received while the stream is being closed (kata-containers#1814) 46bef23 Write should fail when the stream was done but context wasn't cancelled. (kata-containers#1792) 10598f3 Explain target format in DialContext's documentation (kata-containers#1785) 08b7bd3 gzip: add Name const to avoid typos in usage (kata-containers#1804) 8b02d69 remove .please-update (kata-containers#1800) 1cd2346 Documentation: update broken wire.html link in metadata package. (kata-containers#1791) 6913ad5 Document that all errors from RPCs are status errors (kata-containers#1782) 8a8ac82 update const order (kata-containers#1770) e975017 Don't set reconnect parameters when the server has already responded. (kata-containers#1779) 7aea499 credentials: return Unavailable instead of Internal for per-RPC creds errors (kata-containers#1776) c998149 Avoid copying headers/trailers in unary RPCs unless requested by CallOptions (kata-containers#1775) 8246210 Update version to 1.10.0-dev (kata-containers#1777) 17c6e90 compare atomic and mutex performance for incrementing/storing one variable (kata-containers#1757) 65c901e Fix flakey test. (kata-containers#1771) 7f2472b grpclb: Remove duplicate init() (kata-containers#1764) 09fc336 server: fix bug preventing Serve from exiting when Listener is closed (kata-containers#1765) 035eb47 Fix TestGracefulStop flakiness (kata-containers#1767) 2720857 server: fix race between GracefulStop and new incoming connections (kata-containers#1745) 0547980 Notify parent ClientConn to re-resolve in grpclb (kata-containers#1699) e6549e6 Add dial option to set balancer (kata-containers#1697) 6610f9a Fix test: Data race while resetting global var. (kata-containers#1748) f4b5237 status: add Code convenience function (kata-containers#1754) 47bddd7 vet: run golint on _string files (kata-containers#1749) 45088c2 examples: fix concurrent map accesses in route_guide server (kata-containers#1752) 4e393e0 grpc: fix deprecation comments to conform to standard (kata-containers#1691) 0b24825 Adjust keepalive paramenters in the test such that scheduling delays don't cause false failures too often. (kata-containers#1730) f9390a7 fix typo (kata-containers#1746) 6ef45d3 fix stats flaky test (kata-containers#1740) 98b17f2 relocate check for shutdown in ac.tearDown() (kata-containers#1723) 5ff10c3 fix flaky TestPickfirstOneAddressRemoval (kata-containers#1731) 2625f03 bufconn: allow readers to receive data after writers close (kata-containers#1739) b0e0950 After sending second goaway close conn if idle. (kata-containers#1736) b8cf13e Make sure all goroutines have ended before restoring global vars. (kata-containers#1732) 4742c42 client: fix race between server response and stream context cancellation (kata-containers#1729) 8fba5fc In gracefull stop close server transport only after flushing status of the last stream. (kata-containers#1734) d1fc8fa Deflake tests that rely on Stop() then Dial() not reconnecting (kata-containers#1728) dba60db Switch balancer to grpclb when at least one address is grpclb address (kata-containers#1692) ca1b23b Update CONTRIBUTING.md to CNCF CLA 2941ee1 codes: Add UnmarshalJSON support to Code type (kata-containers#1720) ec61302 naming: Fix build constraints for go1.6 and go1.7 (kata-containers#1718) b8191e5 remove stringer and go generate (kata-containers#1715) ff1be3f Add WithResolverUserOptions for custom resolver build options (kata-containers#1711) 580defa Fix grpc basics link in route_guide example (kata-containers#1713) b7dc71e Optimize codes.String() method using a switch instead of a slice of indexes (kata-containers#1712) 1fc873d Disable ccBalancerWrapper when it is closed (kata-containers#1698) bf35f1b Refactor roundrobin to support custom picker (kata-containers#1707) 4308342 Change parseTimeout to not handle non-second durations (kata-containers#1706) be07790 make load balancing policy name string case-insensitive (kata-containers#1708) cd563b8 protoCodec: avoid buffer allocations if proto.Marshaler/Unmarshaler (kata-containers#1689) 61c6740 Add comments to ClientConn/SubConn interfaces to indicate new methods may be added (kata-containers#1680) ddbb27e client: backoff before reconnecting if an HTTP2 server preface was not received (kata-containers#1648) a4bf341 use the request context with net/http handler (kata-containers#1696) c6b4608 transport: fix race sending RPC status that could lead to a panic (kata-containers#1687) 00383af Fix misleading default resolver scheme comments (kata-containers#1703) a62701e Eliminate data race in ccBalancerWrapper (kata-containers#1688) 1e1a47f Re-resolve target when one connection becomes TransientFailure (kata-containers#1679) 2ef021f New grpclb implementation (kata-containers#1558) 10873b3 Fix panics on balancer and resolver updates (kata-containers#1684) 646f701 Change version to 1.9.0-dev (kata-containers#1682) Fixes: kata-containers#307 Signed-off-by: Peng Tao <[email protected]>

lifupan force-pushed the fix_cleanuprace branch 2 times, most recently from 2c35ad4 to 3342fb2 Compare June 27, 2019 09:30

jodh-intel reviewed Jun 27, 2019

View reviewed changes

lifupan force-pushed the fix_cleanuprace branch 2 times, most recently from ed21c2a to a3bfa03 Compare June 27, 2019 12:46

lifupan force-pushed the fix_cleanuprace branch 2 times, most recently from 16b638e to a8e03ed Compare July 31, 2019 02:22

lifupan closed this Aug 6, 2019

lifupan reopened this Aug 6, 2019

lifupan force-pushed the fix_cleanuprace branch 4 times, most recently from 453f086 to f8701d6 Compare August 16, 2019 00:41

lifupan force-pushed the fix_cleanuprace branch from f8701d6 to da685a6 Compare August 16, 2019 04:52

lifupan force-pushed the fix_cleanuprace branch from da685a6 to 962a374 Compare August 16, 2019 06:28

lifupan force-pushed the fix_cleanuprace branch 2 times, most recently from cbebf06 to f6393a6 Compare August 22, 2019 10:05

devimc approved these changes Aug 22, 2019

View reviewed changes

lifupan force-pushed the fix_cleanuprace branch from f6393a6 to 8a8dedc Compare August 23, 2019 03:41

lifupan force-pushed the fix_cleanuprace branch 2 times, most recently from 9452bba to 61accc5 Compare August 23, 2019 06:34

bergwolf reviewed Aug 23, 2019

View reviewed changes

lifupan force-pushed the fix_cleanuprace branch from 61accc5 to c91556a Compare August 24, 2019 00:16

bergwolf approved these changes Aug 24, 2019

View reviewed changes

bergwolf merged commit 3fb8729 into kata-containers:master Aug 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

api: add a CleanupContainer api for VC #1833

api: add a CleanupContainer api for VC #1833

lifupan commented Jun 27, 2019

jodh-intel commented Jun 27, 2019

lifupan commented Jun 27, 2019

lifupan commented Jun 27, 2019

codecov bot commented Jun 27, 2019 •

edited

Loading

jodh-intel Jun 27, 2019

lifupan Jun 27, 2019

egernst Jun 27, 2019

jodh-intel Jun 27, 2019

jodh-intel Jun 27, 2019

lifupan Jun 27, 2019

jodh-intel Jun 27, 2019

jodh-intel Jun 27, 2019

gnawux commented Jun 27, 2019 •

edited

Loading

lifupan commented Jun 27, 2019

gnawux commented Jun 27, 2019

jcvenegas commented Jul 26, 2019

Random-Liu commented Jul 29, 2019

jodh-intel commented Jul 29, 2019

lifupan commented Jul 30, 2019

lifupan commented Jul 31, 2019

lifupan commented Aug 5, 2019 •

edited

Loading

grahamwhaley commented Aug 5, 2019

chavafg commented Aug 5, 2019

lifupan commented Aug 16, 2019

lifupan commented Aug 22, 2019

lifupan commented Aug 22, 2019

lifupan commented Aug 23, 2019

lifupan commented Aug 23, 2019

bergwolf Aug 23, 2019

lifupan Aug 24, 2019

lifupan commented Aug 24, 2019

api: add a CleanupContainer api for VC #1833

api: add a CleanupContainer api for VC #1833

Conversation

lifupan commented Jun 27, 2019

jodh-intel commented Jun 27, 2019

lifupan commented Jun 27, 2019

lifupan commented Jun 27, 2019

codecov bot commented Jun 27, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnawux commented Jun 27, 2019 • edited Loading

lifupan commented Jun 27, 2019

gnawux commented Jun 27, 2019

jcvenegas commented Jul 26, 2019

Random-Liu commented Jul 29, 2019

jodh-intel commented Jul 29, 2019

lifupan commented Jul 30, 2019

lifupan commented Jul 31, 2019

lifupan commented Aug 5, 2019 • edited Loading

grahamwhaley commented Aug 5, 2019

chavafg commented Aug 5, 2019

lifupan commented Aug 16, 2019

lifupan commented Aug 22, 2019

lifupan commented Aug 22, 2019

lifupan commented Aug 23, 2019

lifupan commented Aug 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lifupan commented Aug 24, 2019

codecov bot commented Jun 27, 2019 •

edited

Loading

gnawux commented Jun 27, 2019 •

edited

Loading

lifupan commented Aug 5, 2019 •

edited

Loading