testutil/pkgdata: Create deb programatically

[woky]

Currently we only test with base-files deb which is embedded as base64
encoded binary in testutil/pkgdata.go. If we want to change the contents
of the data files, we have to un-ar it, un-tar it, change the data
files, tar it and ar it. Furthermore, these operations modify attributes
based on the workstation on which this process is run. So we end up with
arbitrary user/group ids/names names and dates.

Fix it by creating debs programatically. The new interface provides way
to create debs with arbitrary content of data tarball.

[niemeyer]

The idea is nice, and the logic looks good too, thank you. In the comments below, I think the main theme is that this seems to be doing a bit too much for something that is supposed to be a tiny helper to produce a trivial deb package. There's some extensibility which is unused and untested, and might as well be dropped. As a suggested approach, I'd try to do the minimum required to get a deb manufactured from short inlined content that is just enough for our purposes.

[niemeyer]

Why not just using inlined content all the time, and using a small amount of data instead? We generally don't care about the data, I think. The base-files use was simply a tiny package that allowed us to not worry about it, but if we're going to create a packge from the ground up, we might as well do the minimum amount of work for that.

[woky]

I'm now inlining the content in package definition. However, I still include base-files and read the content via a helper function (readPkgdataFile), as we have a lot of tests that depend on various files in base-files and their content (via hashes). Changing all the tests for that doesn't sound like minimum amount of work. :-) I propose that from now on, we add new package data as you say, by inlining small amount of data directly, but we keep base-files data. Is that acceptable?

[niemeyer]

As suggested in a different comment, include the new data and new tests, keep the old data and tests as-is, port the old tests to the new inlined data in a separate PR. Let's please not add unwanted logic to avoid unwanted logic, as instead of moving forward that just adds further technical debt to be fixed.

[woky]

The idea is nice, and the logic looks good too, thank you. In the comments below, I think the main theme is that this seems to be doing a bit too much for something that is supposed to be a tiny helper to produce a trivial deb package. There's some extensibility which is unused and untested, and might as well be dropped. As a suggested approach, I'd try to do the minimum required to get a deb manufactured from short inlined content that is just enough for our purposes.

Thanks for detailed review. You are right that it was needlessly complex. The current version is much simpler.

Note that there still some magic in fixupEntry because e.g. specifying tar.Typeflag for each entry is not necessary when it's clear (in our case) from the Name (or Linkname) what kind of file that is. There's tarEntry.noFixup escape hatch for more flexibility.

[cjdcordeiro]

this makes sense and LGTM. thanks! I only have a couple of comments/questions, and one request: could you please add docstrings? It would make the code easier to follow ;)

[cjdcordeiro]

what is this actually used for? I don't think is it used in this base-files test

[cjdcordeiro]

if there a way we could dynamically generate this tarEntry structure from an existing DEB?

my thinking here is: while I agree it is a must to have this default, hardcoded DEB for testing, it would also be useful if we could somehow dynamically add other DEBs to this map, at test time

[niemeyer]

Per earlier comment, and reply to your recent comment just now, let's please not introduce all those files in the repository and more logic to this implementation just to then have to remove them. Instead, it's okay to have a new custom package that is tailored specifically for our tests, with small amounts of inlined data. We don't need to port old tests right away.. just add the specific small amount of data you need for your new test to work, and maybe adapting the tests around it as well if that's useful, and then let's port older tests separately.

[niemeyer]

The standard convention for formatting such lists looks like this:

"base-files": {{
        header: tar.Header{
                Name: "./bin/",
        }, {
                Name: "...",
        }, {
                ...
        }},
}}

This saves lots of empty spacing that makes it harder to go through the list.

[cjdcordeiro]

as a side note, I think this PR could help keep the consistency for formatting

[niemeyer]

Thank you, that's very nice. Just one minor about the testing approach, with an accompanying suggestion:

[niemeyer]

Having the tests comparing against a pre-packaged binary means anyone testing this logic needs to unpack the deb, performs changes, repack the deb, and include the whole binary here. That's exactly what we need to do nowadays for the actual tests we have, so it feels like just moving the goalpost while preserving the same underlying issue.

How about just using our existing "tree-mapping" testing function to verify the output? This is by the end of the day what the actual tests we have will compare the output against, so it feels realistic enough for our purposes, even more when what we have here is really a belt-and-suspender kind of approach in the sense we're testing the testing.

[woky]

@niemeyer I've removed the embedded deb file and now I'm checking tar entries against expected list. I didn't use the tree mapping code, because here, I can just check the list of entries coming out of tar Reader. IOW, I don't need to touch the file system to test this code.

[niemeyer]

Thanks!