Add CE to Eng handover docs

[dadlerj]

Given the recent discussion, I wanted to get this written down. This is a WIP document, and will be living for some time, especially as more clarity comes to the roles and responsibilities of the CE team. However, I thought it would be beneficial to document the "current state", even if it's not long-lived.

I only added one "new" thing to the process here: a priority tagging system that CE can use to communicate to Eng. Curious to hear your thoughts.

CC @nicksnyder @christinelovett @tistru for visibility

[nicksnyder]

Can we change the wording so it is clear that if someone "informally" takes over the conversation, then they should be formally assigned the ticket? I want to avoid grey areas on who is responsible.

[nicksnyder]

So a tier 2 customer (and below) can never cause a p0?

[dadlerj]

Correct

[dadlerj]

This is just a quick and dirty set of definitions though, and Julia needs to weigh in. The definition of what the eng team is supposed to do when a p0 lands is also not clear here, so Gonza should weigh in on that side! This is just a first attempt at providing guidance to new CE team members based on how I internally think about communicating prioritization currently.

[pecigonzalo]

I believe this list is an extension or update to the one described here https://github.com/sourcegraph/about/blob/main/handbook/ce/support.md#slas.

Here is a draft idea on how we could handle this, I would PR this to incident document:

• p0: All hands on deck, notify on #dev-ops. The incident is the highest priority for all engineering teams and should drop any other tasks to work on the incident as required by the incident owner (not all at once)
• p1: An engineer should prioritize this over all other tasks and transfer to other teams as required.
• p2: We will resolve in a best-efforts basis while we continue to work on our planned release.
• p3: We should consider prioritizing this over other planned work, or schedule it for our next iteration.
• p4: We will evaluate this along with our other features to be planned in a future release.

I would also suggest extending incident response and including some ideas from Google and PagerDuty.

Particularly the different roles (PagerDuty, Google) for p0 incidents and the idea I mentioned about having a /on-call teamX which I found perfectly described here.

The indecent response page by PagerDuty is a great resource for general ideas an guidelines.

[juliasourceress]

This is a great and necessary convo, and the goal of this initial PR is to document how we do this today. So for now I support @dadlerj 's documented process as a start, since this is how he's been operating. A clear next step is to iterate on this based on @pecigonzalo 's suggestions.

[dadlerj]

Thanks Julia! Agreed with that approach.

To be transparent, I'm not directly using this methodology yet, but when I started to document my process, this is roughly how I think about my own method for communicating prioritization of issues. This just adds an official tag onto it (versus my handling in a one-off way). I'll merge this now and get feedback from Tion and Christine once it's live.

@pecigonzalo thanks for the feedback. A few notes:

I believe this list is an extension or update to the one described here https://github.com/sourcegraph/about/blob/main/handbook/ce/support.md#slas.

It's actually quite different... That page reflects the promises we make to customers—i.e., the minimum service level required per our contract—while this list reflects our internal prioritization. There clearly is an intimate connection between the two, but they're not quite the same in terms of what they specify and how we want to describe them (and never quite will be).

Here is a draft idea on how we could handle this, I would PR this to incident document:

This is a great conversation—"what do we do about an issue that was described as a given level of seriousness by CE"—but it is slightly different from what this doc is adding. I'd like to start with our definition of how serious an issue is, and you (and Julia and others of course) can nail down the translation of this into the "so what".

[pecigonzalo]

It's actually quite different... That page reflects the promises we make to customers—i.e., the minimum service level required per our contract—while this list reflects our internal prioritization. There clearly is an intimate connection between the two, but they're not quite the same in terms of what they specify and how we want to describe them (and never quite will be).

I disagree, they are not that different given the agreed response and resolution times are directly related to the communicated prioritization as ce/pX and how we respond to that priority internally. The intended audience is different, but even the description wording on both is quite similar, only more detailed on this document as we have more levels.
We could provide this by having a single matrix, a linked relationship between the two or just the same name across both pages but holding different information to match each intended audience.

[pecigonzalo]

Thanks for taking this initial draft Dan. I will start working on the incident response updates and backlink here from the PR.

[pecigonzalo]

We should add a link to the document that outlines this customers.

[pecigonzalo]

I believe this list is an extension or update to the one described here https://github.com/sourcegraph/about/blob/main/handbook/ce/support.md#slas.

Here is a draft idea on how we could handle this, I would PR this to incident document:

• p0: All hands on deck, notify on #dev-ops. The incident is the highest priority for all engineering teams and should drop any other tasks to work on the incident as required by the incident owner (not all at once)
• p1: An engineer should prioritize this over all other tasks and transfer to other teams as required.
• p2: We will resolve in a best-efforts basis while we continue to work on our planned release.
• p3: We should consider prioritizing this over other planned work, or schedule it for our next iteration.
• p4: We will evaluate this along with our other features to be planned in a future release.

I would also suggest extending incident response and including some ideas from Google and PagerDuty.

Particularly the different roles (PagerDuty, Google) for p0 incidents and the idea I mentioned about having a /on-call teamX which I found perfectly described here.

The indecent response page by PagerDuty is a great resource for general ideas an guidelines.

[juliasourceress]

An engineer should always check in with a CE before replying if it's not a conversation they're already having (but a new thread or ticket). If it's an existing thread the engineer should be the one to triage and either continue responding or let the CE owner know that despite it being the same ticket or thread, it's a new request, and they'd like to hand it back over to the CE for ownership.

[juliasourceress]

Or alternatively they can find someone else to tap in, and let the CE know.

[tistru]

Hi @dadlerj . Great work putting this together and I believe it will really helpful. I didn't have any feedback on the current version of the document. I understand we will iterate on it going forward but I am perfectly fine to operate under these guidelines going forward :)