roachtest: post issues

[tbg]

We run roachtest regularly on both master and release-2.0. A few tests are failing regularly, and with the expected growth in the number of tests this is unlikely to become better anytime soon.

roachtest failures should match the experience from our unit and acceptance testing in that they open issues for failed runs. This makes these failures visible to triage and assignable.

We do have notifications in a slack channel, but that does not create the same sense of responsiblity and is already not working well. These notifications are an auxiliary tool at best.

[vilterp]

How do we create GH issues for unit test runs currently? Is it a script that runs inside of TC, triggered by failed runs?

[benesch]

Yep. See build/teamcity-post-failures.py and pkg/cmd/github-post.

…

On Wed, Apr 18, 2018 at 3:37 PM, Pete Vilter ***@***.***> wrote: How do we create GH issues for unit test runs currently? Is it a script that runs inside of TC, triggered by failed runs? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#24894 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA15IF93tYwbKoEv0ofNOUghacFbfoDFks5tp5XmgaJpZM4TZTc8> .

[tbg]

I'm putting this in CF&S for now, but once the builddevelopment support team is ramping up, feel free to steal this from me (if it's still open -- we need to do this ASAP).

[tbg]

@petermattis bumping this. Triaging is hard without it.

[petermattis]

I wonder if we can just pipe the roachtest output through cmd/github-post, similar to what build/teamcity-stress.sh does.

[petermattis]

github-post works if we're using the normal output, but not if we're using the --teamcity output. Hrmm. I suppose I could extract the logic from github-post into a library and use it directly in roachtest. @benesch probably has a better idea.

[benesch]

I think you want to invoke build/teamcity-post-failures.py. It looks at the structured TeamCity test status API and so should handle (in fact require) the --teamcity output.

…

On Tue, May 15, 2018 at 10:59 PM, Peter Mattis ***@***.***> wrote: github-post works if we're using the normal output, but not if we're using the --teamcity output. Hrmm. I suppose I could extract the logic from github-post into a library and use it directly in roachtest. @benesch <https://github.com/benesch> probably has a better idea. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24894 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA15IIdaCvehhvY8YFXvK_w9klkGAtQdks5tyu1KgaJpZM4TZTc8> .

[petermattis]

@benesch Thanks for the pointer. Looks like it should work, though teamcity-post-failures.py doesn't do the de-duplication that github-post does.

One advantage to having roachtest post issues directly is that we could distinguish between stable and unstable failures.

[benesch]

Makes sense. Sounds like it might be time to merge teamcity-post-failures into github-post and make it a proper library. (I'd really like to avoid a *third* issue posting script.)

…

On Wed, May 16, 2018 at 8:09 AM, Peter Mattis ***@***.***> wrote: @benesch <https://github.com/benesch> Thanks for the pointer. Looks like it should work, though teamcity-post-failures.py doesn't do the de-duplication that github-post does. One advantage to having roachtest post issues directly is that we could distinguish between stable and unstable failures. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24894 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA15IJOwSV0KOnecax7TueE3XeOOVeUMks5ty24fgaJpZM4TZTc8> .

[petermattis]

My thought was to pull the guts of github-post into a library and use that directly from roachtest. Converting teamcity-post-failures would be a natural follow-on step when I rewrite that python script in Go. (The xml parsing in Go looks real easy, so I don't expect that to take much time at all).

[benesch]

👍🏽

…

On Wed, May 16, 2018 at 11:37 PM Peter Mattis ***@***.***> wrote: My thought was to pull the guts of github-post into a library and use that directly from roachtest. Converting teamcity-post-failures would be a natural follow-on step when I rewrite that python script in Go. (The xml parsing in Go looks real easy, so I don't expect that to take much time at all). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24894 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA15IPcdGWXByOyn2NcuO9h4tnTSfhdbks5tzEe8gaJpZM4TZTc8> .