Improve the Release status dashboard #547

encukou · 2024-10-25T12:48:48Z

I've spent some time making the Release Dashboard closer to my daily status reports (and include the info needed for them). Here's the result.

Outdated static demo: https://encukou.github.io/buildbot-status-dashboard/
WIP repo, with caching and hacks needed to iterate on this locally: https://github.com/encukou/buildbot-status-dashboard

Technically it should be drop-in upgrade for the existing dashboard, and that's what this PR does. But I now think it would be better to have both, at least for a while, in case this falls over. But, code is here.

Does this look reasonable?

Quick branch info (status, RM name, release schedule link) for all branches at the top, with detailed per-branch sections after that.
Details can generally be expanded/collapsed.

Show tier-3+ failures: these aren't considered release-blocking but we tend to fix these ASAP. These mark a "green" branch/builder as "yellow".

Show low-priority problems: disconnected builders, build warnings, unstable builder failures. These don't affect the overall status.

Show changes and test results for failing builds (when available).

Show build history overview for builders; this is useful context.

Conceptually, the branch sections list "problems", not builders, in preparation for adding blocker issues from GitHub. Problems are sorted by severity & grouped by type; the highest-severity problem determines the overall branch status.

Pull Request builders are, for now, shown in a no-branch section (hiding at the end with no link to it).

Based on Pablo's existing dashboard and his BuildbotHammer tool.

Quick branch info (status, RM name, release schedule link) for all branches at the top, with detailed per-branch sections after that. Details can generally be expanded/collapsed. Show tier-3+ failures: these aren't considered release-blocking but we tend to fix these ASAP. These mark a branch/builder "yellow". Show low-priority problems: disconnected builders, build warnings, unstable builder failures. These keep a branch marked "green". Show changes and test results for failing builds (when available). Show build history overview for builders; this is useful context. Conceptually, the branch sections list "problems", not "builders", in preparation for adding GitHub blocker issues here. Problems are sorted by severity & grouped by type; the highest-severity problem determines the overall branch status. Based on Pablo's existing dashboard and his BuildbotHammer tool. Co-Authored-By: Pablo Galindo Salgado <[email protected]>

pablogsal

Some comments for a first pass

pablogsal · 2024-10-29T21:28:55Z

master/custom/release_dashboard.py

+        try:
+            return func(*args, **kwargs)
+        except AttributeError as e:
+            raise Exception(f'your error: {e!r}')


Maybe raise something more concrete (maybe TemplateAttributeError or something else).

Sure.
(This is a debug helper -- but one that I find myself using constantly so I'd rather leave it in.)

pablogsal · 2024-10-29T21:30:00Z

master/custom/release_dashboard.py

+        except IndexError:
+            return NoProblem()
+       except AttributeError:
+           raise SystemError


Why SystemError? Shouldn't better be something concrete likeBuilderError or something akin to that?

Oops, I left some debug code in. Thanks for the catch!

pablogsal · 2024-10-29T21:33:16Z

master/custom/release_dashboard.py

+    def junit_results(self):
+        filepath = (
+            self._root._app.test_result_dir
+            / self.builder.branch.tag
+            / self.builder["name"]
+            / f'build_{self["number"]}.xml'
+        )
+        try:
+            file = filepath.open()
+        except OSError:
+            return None
+        with file:
+            etree = ElementTree.parse(file)
+        result = JunitResult(self, {})
+        for element in etree.iterfind('.//error/..'):
+            result.add(element)
        return result


(This assumes there is a logger somewhere, but you can drop that)

Suggested change

def junit_results(self):

filepath = (

self._root._app.test_result_dir

/ self.builder.branch.tag

/ self.builder["name"]

/ f'build_{self["number"]}.xml'

)

try:

file = filepath.open()

except OSError:

return None

with file:

etree = ElementTree.parse(file)

result = JunitResult(self, {})

for element in etree.iterfind('.//error/..'):

result.add(element)

return result

@cached_property

def junit_results(self):

if not self._root._app.test_result_dir:

return None

try:

filepath = (

self._root._app.test_result_dir

/ self.builder.branch.tag

/ self.builder["name"]

/ f'build_{self["number"]}.xml'

).resolve()

# Ensure path doesn't escape test_result_dir

if not str(filepath).startswith(str(self._root._app.test_result_dir)):

logger.warning(f"Attempted path traversal: {filepath}")

return None

if not filepath.is_file():

return None

with filepath.open() as file:

etree = ElementTree.parse(file)

except OSError as e:

logger.error(f"Failed to read JUnit results: {e}")

return None

except ElementTree.ParseError as e:

logger.error(f"Failed to parse JUnit XML: {e}")

raise XMLParseError(f"Malformed JUnit XML: {e}") from e

Thanks!
I'll punt on logging in this PR.

pablogsal · 2024-10-29T21:35:31Z

master/custom/release_dashboard.py

+            if error not in result.errors:
+                # De-duplicate, since failing tests are re-run and often fail
+                # the same way
+                result.errors.extend(errors)


Hummmm aren'y you adding technically all the errors again and again once per error to the list? What am I missing?

Right, this is confusing. I'll add comments.

This is called per <There's usually only one error in errorsanderror_types. (With subTest`, there should IMO be multiple errors per test, but we don't do that yet. I opened libregrtest: junit-xml output doesn't support subTest cpython#126237 for it, so it's not sitting only in my TODO list.)

The types are added to each level of the tree, so that if you don't expand the details you see list of all (unique) exception types under it.

pablogsal · 2024-10-29T22:09:03Z

Wow Petr, this is absolutely fantastic work! 🎉

I'm really excited to see the release dashboard coming together like this! This is going to make everyone's lives so much easier.

I made a first pass but given how big this PR is I probably missed stuff so I may do one more later.

encukou

Thank you for your kind words!

Do you think it's OK to replace the existing dashbuard, or should we leave them side-by-side for a while?

encukou · 2024-10-31T11:45:09Z

master/custom/release_dashboard.py

+        except IndexError:
+            return NoProblem()
+       except AttributeError:
+           raise SystemError


Oops, I left some debug code in. Thanks for the catch!

encukou · 2024-10-31T12:40:26Z

master/custom/release_dashboard.py

+            if error not in result.errors:
+                # De-duplicate, since failing tests are re-run and often fail
+                # the same way
+                result.errors.extend(errors)


Right, this is confusing. I'll add comments.

This is called per <There's usually only one error in errorsanderror_types. (With subTest`, there should IMO be multiple errors per test, but we don't do that yet. I opened libregrtest: junit-xml output doesn't support subTest cpython#126237 for it, so it's not sitting only in my TODO list.)

The types are added to each level of the tree, so that if you don't expand the details you see list of all (unique) exception types under it.

encukou · 2024-10-31T12:52:53Z

master/custom/release_dashboard.py

+    def junit_results(self):
+        filepath = (
+            self._root._app.test_result_dir
+            / self.builder.branch.tag
+            / self.builder["name"]
+            / f'build_{self["number"]}.xml'
+        )
+        try:
+            file = filepath.open()
+        except OSError:
+            return None
+        with file:
+            etree = ElementTree.parse(file)
+        result = JunitResult(self, {})
+        for element in etree.iterfind('.//error/..'):
+            result.add(element)
        return result


Thanks!
I'll punt on logging in this PR.

encukou · 2024-10-31T13:28:22Z

master/custom/release_dashboard.py

+        try:
+            return func(*args, **kwargs)
+        except AttributeError as e:
+            raise Exception(f'your error: {e!r}')


Sure.
(This is a debug helper -- but one that I find myself using constantly so I'd rather leave it in.)

encukou and others added 2 commits October 25, 2024 14:09

This needs Hugo's Humanize

564d62b

pablogsal reviewed Oct 29, 2024

View reviewed changes

Address review + limit each builder to 70 displayed "build dots"

6f8d0b3

encukou commented Oct 31, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the Release status dashboard #547

Improve the Release status dashboard #547

encukou commented Oct 25, 2024

pablogsal left a comment

pablogsal Oct 29, 2024

encukou Oct 31, 2024

pablogsal Oct 29, 2024

encukou Oct 31, 2024

pablogsal Oct 29, 2024

encukou Oct 31, 2024

pablogsal Oct 29, 2024

encukou Oct 31, 2024

pablogsal commented Oct 29, 2024

encukou left a comment

encukou Oct 31, 2024

encukou Oct 31, 2024

encukou Oct 31, 2024

encukou Oct 31, 2024

Improve the Release status dashboard #547

Are you sure you want to change the base?

Improve the Release status dashboard #547

Conversation

encukou commented Oct 25, 2024

pablogsal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pablogsal commented Oct 29, 2024

encukou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment