WIP: KEP: Who are we and why are we here? #2363

hh · 2018-07-11T04:39:02Z

Ask some introspective questions of our software interactions
allow cluster wide aggregation of the answers
analyse community wide results for non-obvious patterns

k8s-ci-robot · 2018-07-11T04:39:14Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: jdumars

If they are not already assigned, you can assign the PR to them by writing /assign @jdumars in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

keps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hh · 2018-07-11T04:43:25Z

/cc @kubernetes/sig-architecture-feature-requests @kubernetes/sig-testing-feature-requests
/cc @WilliamDenniss @spiffxp @AishSundar

neolit123

thank you for the read and the cultural background!
found one typo :)

neolit123 · 2018-07-14T19:35:29Z

keps/sig-architecture/0016-20180601-identifying-api-usage-patterns.md

+enable all kubernetes components to begin transmitting identity and purpose.
+
+Setting this variable on all pods could be accomplished with
+an admission or initialization controller allowng all other 


Thanks, fixed.

kubernetes-retired/architecture-tracking#40

hh · 2018-07-15T23:45:42Z

keps/sig-architecture/0016-20180601-identifying-api-usage-patterns.md

+
+## Drawbacks [optional]
+
+Some feel this level of client ‘debugging’ doesn't belong in server-side logs. 


kubernetes-sigs/apisnoop#23 (comment)

DirectXMan12 · 2018-07-16T15:43:10Z

At a glance, some of this looks similar to tracing. can you compare/contrast this a bit with opentracing? It seems like you could inject the user agent and traceback stuff optionally as persistent tracing information, but I'm not super familiar with the details.

hh · 2018-07-18T05:20:04Z

Our goal is to make it simple to submit and gather API usage data, both cluster and community wide. If we want to identify the yet unseen patterns of use, we need the ability to easily instrument all K8s API consumers.

The core of this KEP is to explore easy methods to enable any application to verbosely convey, for each API call, the identity of the code behind a particular interaction. I'd prefer to enable this behavior by setting a variable vs creating special binaries with compile time instrumentation libraries.

Jaeger and Opentracing are great for debugging and development purposes. They do however require instrumentation logic to be added to the applications themselves. They do provide some beautiful insight, if you already know what your looking for. This would also limit the data we would be able to collect.

The initial APISnoop approach used mitm injection via a sidecar, similar to Envoy, and provided a non-kubernetes specific approach.

We explored using Istio / Envoy, but they were not designed to intercept communications destined for the APIServer.

At this point that things were looking a bit Rube Goldberg, hence the switch in focus to audit-logging.

While I'm enjoying the simplicity of audit-logging, it might be worth re-exploring the mitm / Envoy approach.

If we route APIServer traffic to route through a proxy like Envoy we could support APISnooping more than just the K8s API.

dankohn · 2018-07-18T10:53:24Z

@bhs could you please add any comments you have re OpenTracing.

bhs · 2018-07-18T14:39:29Z

There are three pieces here:

Literally adding the necessary instrumentation to pass context / causality information from the clients into k8s, then through k8s while servicing the client API request(s)
Adding information (e.g., "purpose," though also what actually happens (beyond mere context propagation) to decorate the transactions for later analysis
Collection and analysis of all of the above

The third piece almost certainly does require "something new"... general-purpose tracing tools like Jaeger/Zipkin/etc are designed for performance analysis and root cause analysis, and it sounds like @hh intends for more bespoke sorts of insight as an output to the entire effort. It is conceivable to use a general-purpose distributed tracing system to collect the data, then write an analytical script to generate the human-ready output via API calls, but in any case "something new" will still need t obe written.

The second piece is kind of an "it depends"... OpenTracing has the notion of "baggage" which is designed for just this sort of user-level context information that has some user-level interpretation. At first glance it's very similar in spirit, though perhaps I'm missing something. Note that some people are nervous about baggage, though.

The first piece is where I see strong overlap with OpenTracing specifically. The actual context propagation and context management is what OT was designed for, and the API is thin, with no dependencies of its own (i.e., not a Rube Goldberg nightmare per the KEP's language).

I think the determinant would be (a) how much multi-process context propagation we expect to see here, and (b) what k8s wants to do with more debugging-centric context propagation. The main insight with (b) is that "context is context" regardless of the application, and the true "Rube Goldberg" outcome would be to record and propagate it multiple times in parallel.

If there was a decision to proceed with OT, I'd imagine that this effort could be / would be supported via a Tracer implementation that does some sort of passthrough/multiplexing with a "normal" Tracer... i.e., the WhyAreWeHereTracer / WAWHTracer would record the precise info it needed to, then pass things along to a (potentially noop) general-purpose Tracer.

Hope this helps.

dankohn · 2018-07-18T15:31:02Z

@bhs thanks for the quick review. Let's plan to have @hh present this work to the OpenTracing community in the next couple months, once he has some progress to show.

k8s-ci-robot · 2018-07-18T21:14:10Z

@hh: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Automatic merge from submit-queue (batch tested with PRs 67430, 67550). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add e2e Test Description to User-Agent This PR appends detail e2e test line number and test details to the User-Agent. It's needed for kubernetes/community#2363 and Conformance Data gathering. This can be easily tested with any 1.12.0-alpha or later that is configured for audit-logging. Here is an example using the test-infra/kubetest dind from kubernetes/test-infra#9061 ``` cd ~/go/src/k8s.io/kubernetes/ go get -u k8s.io/test-infra/kubetest kubetest --build=dind kubetest --up --deployment=dind export KUBERNETES_CONFORMANCE_TEST=y export KUBECONFIG=$(ls -rt /tmp/k8s-dind-kubecfg-* | tail -1) export DIND_K8S_DATA=$(ls -drt /tmp/dind-k8* | tail -1) make -j 8 GOGCFLAGS="-N -l -v" WHAT=test/e2e/e2e.test ./_output/local/bin/linux/amd64/e2e.test \ --ginkgo.focus=\[Conformance\] \ --ginkgo.seed=1436380640 \ --v=2 \ --ginkgo.skip='\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]|\[HPA\]|Dashboard|Services.*functioning.*NodePort' cp $DIND_KCS_DATA/audit/audit.log . ``` ``` e2e.test/v1.12.0 (linux/amd64) kubernetes/94c2c6c -- k8s.io/kubernetes/test/e2e/kubectl/portforward.go:496 -- [sig-cli] Kubectl Port forwarding [k8s.io] With a server listening on localhost [k8s.io] that expects NO client request should support a client that connects, sends DATA, and disconnects e2e.test/v1.12.0 (linux/amd64) kubernetes/94c2c6c -- k8s.io/kubernetes/test/e2e/network/dns.go:158 -- [sig-network] DNS should provide DNS for ExternalName services e2e.test/v1.12.0 (linux/amd64) kubernetes/94c2c6c -- k8s.io/kubernetes/test/e2e/framework/framework.go:710 -- [sig-cli] Kubectl client [k8s.io] Kubectl cluster-info should check if Kubernetes master services is included in cluster-info [Conformance] e2e.test/v1.12.0 (linux/amd64) kubernetes/94c2c6c -- k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:562 -- [sig-storage] PersistentVolumes-local StatefulSet with pod affinity should use volumes on one node when pod has affinity ``` Sample log file from e2e run: [audit.log.gz](https://github.com/kubernetes/kubernetes/files/2298944/audit.log.gz)

justaugustus · 2018-10-13T03:23:06Z

/kind kep

quinton-hoole · 2018-10-18T19:47:27Z

@hh Very entertaining :-). Can this be closed? If not can it be abbreviated to to the bare essentials to facilitate effective review?

hh · 2018-10-30T21:00:36Z

I'm working on abbreviating it to the bare essentials. I'll set it back to WIP.

justaugustus · 2018-11-20T04:43:42Z

REMINDER: KEPs are moving to k/enhancements on November 30. Please attempt to merge this KEP before then to signal consensus.
For more details on this change, review this thread.

Any questions regarding this move should be directed to that thread and not asked on GitHub.

justaugustus · 2018-12-01T08:05:03Z

KEPs have moved to k/enhancements.
This PR will be closed and any additional changes to this KEP should be submitted to k/enhancements.
For more details on this change, review this thread.

Any questions regarding this move should be directed to that thread and not asked on GitHub.
/close

k8s-ci-robot · 2018-12-01T08:05:04Z

@justaugustus: Closed this PR.

In response to this:

KEPs have moved to k/enhancements.
This PR will be closed and any additional changes to this KEP should be submitted to k/enhancements.
For more details on this change, review this thread.

Any questions regarding this move should be directed to that thread and not asked on GitHub.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hh and others added 21 commits July 11, 2018 15:15

Initial save of metadata / title

99e0c70

Identifying API usage patterns with Applied Anthropology

2979c9e

Increment NEXT_KEP_NUMBER

cd8aa57

Add implementation history

cf8dd91

Clarify Goals

6d464f7

Clean up Proposal intro.

1e69381

Adding emphasis where necessary.

bb53b03

Add a couple user stories

68efde1

Expand identity to include compile time etc

a552c50

Rename file to identifying api usage patterns

b34503b

Set sig-testing as the owning-sig

9aec62e

Summary and Motivation emphasis

e2941d6

More emphasis (more cowbell?)

5bed497

Italic emphasis to draw eyes to questions, atlas, and journey

69f43c8

Generate Table of Contents

9942629

Remove 'Collateral' as it's not the correct word

495836c

Cleaning up ToC and expanding on Identity

fdf4be5

Fixing ToC

4a610d0

Update KEP and final ToC fix

9fc25a8

Set owning sig to sig-architecture

d7f5ed3

Update last-updated

e98e6fe

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 11, 2018

k8s-ci-robot requested review from calebamiles and jbeda July 11, 2018 04:39

k8s-ci-robot added sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 11, 2018

k8s-ci-robot requested review from AishSundar and spiffxp July 11, 2018 04:43

hh added 2 commits July 13, 2018 03:31

Update ToC with User Story titles

0922a40

Restructure first few lines of Proposal

b6e71a7

jdumars mentioned this pull request Jul 12, 2018

KEP #17: Identifying API usage patterns with Applied Anthropology kubernetes-retired/architecture-tracking#40

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 14, 2018

neolit123 reviewed Jul 14, 2018

View reviewed changes

hh added 2 commits July 16, 2018 10:26

bump next KEP to match

d81039a

support every binary run within and on top of the cluster

3fc6da9

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2018

Looks like this might be KEP-17

740b9b2

kubernetes-retired/architecture-tracking#40

hh commented Jul 15, 2018

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 18, 2018

hh mentioned this pull request Aug 17, 2018

Add e2e Test Description to User-Agent kubernetes/kubernetes#67550

Merged

k8s-ci-robot added the kind/kep label Oct 13, 2018

hh changed the title ~~KEP: Who are we and why are we here?~~ WIP: KEP: Who are we and why are we here? Oct 30, 2018

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 30, 2018

k8s-ci-robot closed this Dec 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: KEP: Who are we and why are we here? #2363

WIP: KEP: Who are we and why are we here? #2363

hh commented Jul 11, 2018 •

edited

Loading

k8s-ci-robot commented Jul 11, 2018

hh commented Jul 11, 2018

neolit123 left a comment

neolit123 Jul 14, 2018

hh Jul 15, 2018

hh Jul 15, 2018

DirectXMan12 commented Jul 16, 2018

hh commented Jul 18, 2018 •

edited

Loading

dankohn commented Jul 18, 2018

bhs commented Jul 18, 2018

dankohn commented Jul 18, 2018

k8s-ci-robot commented Jul 18, 2018

justaugustus commented Oct 13, 2018

quinton-hoole commented Oct 18, 2018

hh commented Oct 30, 2018

justaugustus commented Nov 20, 2018

justaugustus commented Dec 1, 2018

k8s-ci-robot commented Dec 1, 2018


		## Drawbacks [optional]

		Some feel this level of client ‘debugging’ doesn't belong in server-side logs.

WIP: KEP: Who are we and why are we here? #2363

WIP: KEP: Who are we and why are we here? #2363

Conversation

hh commented Jul 11, 2018 • edited Loading

k8s-ci-robot commented Jul 11, 2018

hh commented Jul 11, 2018

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 Jul 14, 2018

Choose a reason for hiding this comment

hh Jul 15, 2018

Choose a reason for hiding this comment

hh Jul 15, 2018

Choose a reason for hiding this comment

DirectXMan12 commented Jul 16, 2018

hh commented Jul 18, 2018 • edited Loading

dankohn commented Jul 18, 2018

bhs commented Jul 18, 2018

dankohn commented Jul 18, 2018

k8s-ci-robot commented Jul 18, 2018

justaugustus commented Oct 13, 2018

quinton-hoole commented Oct 18, 2018

hh commented Oct 30, 2018

justaugustus commented Nov 20, 2018

justaugustus commented Dec 1, 2018

k8s-ci-robot commented Dec 1, 2018

hh commented Jul 11, 2018 •

edited

Loading

hh commented Jul 18, 2018 •

edited

Loading