Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve rustdoc JS tests error output #79443

Merged
merged 1 commit into from
Nov 29, 2020
Merged

Improve rustdoc JS tests error output #79443

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions src/tools/rustdoc-js/tester.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,30 @@ function loadThings(thingsToLoad, kindOfLoad, funcToCall, fileContent) {
return content;
}

function contentToDiffLine(key, value) {
return `"${key}": "${value}",`;
}

// This function is only called when no matching result was found and therefore will only display
// the diff between the two items.
function betterLookingDiff(entry, data) {
let output = ' {\n';
let spaces = ' ';
for (let key in entry) {
if (!entry.hasOwnProperty(key)) {
continue;
}
let value = data[key];
if (value !== entry[key]) {
output += '-' + spaces + contentToDiffLine(key, entry[key]) + '\n';
output += '+' + spaces + contentToDiffLine(key, value) + '\n';
} else {
output += spaces + contentToDiffLine(key, value) + '\n';
}
}
return output + ' }';
}

function lookForEntry(entry, data) {
for (var i = 0; i < data.length; ++i) {
var allGood = true;
Expand Down Expand Up @@ -281,6 +305,13 @@ function runSearch(query, expected, index, loaded, loadedFile, queryName) {
if (entry_pos === null) {
error_text.push(queryName + "==> Result not found in '" + key + "': '" +
JSON.stringify(entry[i]) + "'");
// By default, we just compare the two first items.
let item_to_diff = 0;
if ((ignore_order === false || exact_check === true) && i < results[key].length) {
item_to_diff = i;
}
Comment on lines +308 to +312
Copy link
Member

@jyn514 jyn514 Nov 27, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also seems pretty odd to me - it means that if ignore_order isn't set, this might not even compare the right set of objects!

I think a good way to resolve all the concerns both I and @camelid have mentioned is to unconditionally print all the objects, then show a diff between that and all the expected objects. That means:

  • The order won't matter (assuming you print them sorted)
  • All objects that don't match will be shown
  • It will use diff syntax without having to worry too much about exactly how to print +- . (I'm imagining that you just call the diff CLI tool, but you could also use something like https://docs.rs/patch/0.5.0/patch/).
  • If an object is missing altogether, that will show up in the diff, and you'll be able to tell the difference between that and rustdoc guessing the wrong object to compare to.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with the diff CLI tool is that you need at least one of the arguments to be a file. The problem with the patch crate is this is JavaScript :P

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, it could create a temporary file and then delete it after then.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And like I said: search can return hundreds of results. You don't want to compare them all. We show the failing comparison, I think it provides more than enough information.

Also, I don't want to rely on external tools that might not be installed (try your luck on windows, you'll see how "easy" it is to setup pathes to binaries hehe).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We show the failing comparison, I think it provides more than enough information.

But we don't show the failing comparison if ignore_order isn't set - we show the incorrect object and a random other object that happened to come first. They might have nothing to do with one another!

You don't want to compare them all.

Right, diff will only show the ones that changed. I don't think we should be trying to second guess which changes are important and which aren't when we don't have the information to tell.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We show the failing comparison, I think it provides more than enough information.

But we don't show the failing comparison if ignore_order isn't set - we show the incorrect object and a random other object that happened to come first. They might have nothing to do with one another!

Exactly, but we have literally no way to guess which one was supposed to be the matching one in this case. And you still don't want hundreds of comparisons (all failing).

You don't want to compare them all.

Right, diff will only show the ones that changed. I don't think we should be trying to second guess which changes are important and which aren't when we don't have the information to tell.

I can make a special case for the remaining one and just show the object saying "this object was not found in the data", but then you might miss an information. Generally, when you fail this test, it's simply because you badly set one of the fields (the parent one generally), which you will spot right away with this diff. I think it's really an improvement because of that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I guess in the common case this is an improvement, I'm ok merging it for now. But I definitely think it could be better.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue listing the improvements you have in mind then? That could be a nice first issue for newcomers too I think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure they're clear enough in my mind to suggest concrete improvements. I think we should try this for now and if I find something that bugs me, go back and fix it once I have a better idea what I would like it to be.

error_text.push("Diff of first error:\n" +
betterLookingDiff(entry[i], results[key][item_to_diff]));
} else if (exact_check === true && prev_pos + 1 !== entry_pos) {
error_text.push(queryName + "==> Exact check failed at position " + (prev_pos + 1) +
": expected '" + JSON.stringify(entry[i]) + "' but found '" +
Expand Down