Measure the accuracy of log classifier ruleset

[huydhn]

Log classifier is the tool that PyTorch CI is using to extract relevant failures from the log. It depends on a rule set defined at https://github.com/pytorch/test-infra/blob/main/aws/lambda/log-classifier/ruleset.toml in which the rules are evaluated from top to bottom in a first-match basis. The first line in the logs that matches one of the rule is consider the relevant error.

The last rule in the list is the catch-all GHA error, when it matches, it means that the log classifier has failed to find the relevant failures. So, the log classifier accuracy is 1 - % of catch-all GHA error.

Notes to bootcampers:

• The data is available on workflow_job ClickHouse table torchci_classification column
• Ping me or @clee2000 if you need access https://clickhouse.com