Exploring package contents easily

[jakirkham]

One thing that I really enjoy about the current format is how easy it is to unpack a package and inspect its contents. This is really great as a packager to make sure I'm really producing what I think I am or aid in debugging build issues I'm encountering. While I can certainly appreciate the gains made by compressing packages more efficiently and organizing things a bit, I'd be curious to know what recommendations the Conda team has to packagers trying to quickly inspect package contents.

[msarahan]

cph x filename will extract either format. How is that any worse that tar -jxvf filename?

[jakirkham]

Well now I just double click. 😄 That's not really the point though.

I can make some suggestions if you prefer. Was trying to leave it open ended in case you had your own thoughts here.

[msarahan]

sure, if you have suggestions, please post them. Double-clicking is nice, but I'm willing to sacrifice it for 10x decompress/extraction speed.

[msarahan]

I think maybe you're overestimating the timeline for .tar.bz2 files going away. They're not going away any time soon. Right now, things are built as .tar.bz2 and then converted to .conda files later. You can still download .tar.bz2 files for your package. In fact, that's all you'll get unless you do extra work to produce the .conda files.

It would be nice if .conda files were also double-clickable, for sure.

[epruesse]

If they have a useful magic in the beginning, it should be easy to link an application to the file type. You'd probably want an installer to open though, not a content viewer.

[jakirkham]

Well one suggestion would be to have this information accessible on Anaconda.org. That would avoid needing to download the package at all (unless the specific package contents need to be checked more closely).

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[jakirkham]

Would be good to keep this open

[jakirkham]

This came up again in this comment ( conda-forge/conda-forge.github.io#1586 (comment) )

cc @dhirschfeld

[dhirschfeld]

One trick that might help would be to have a separate CI job which unpacked the contents of the just-built package and called tree on it. With GitHub Actions you could even output the result as a job summary - something like:

cph x mypkg.conda
contents=$(tree ./mypkg)
echo "### Package Contents" >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY
echo "${contents}"  >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY

[chrisburr]

One trick that might help would be to have a separate CI job which unpacked the contents of the just-built package and called tree on it. With GitHub Actions you could even output the result as a job summary - something like:

This information is already in https://github.com/regro/libcfgraph

[jakirkham]

Yeah think it depends on what question one is asking. If it is, does this package contain <file> or which pack contains <file>, agree the web interface is the better way to go. If it is, did my build produce the package expected or why is the package produced from this build not behaving, then one wants tools to inspect things locally.

[jakirkham]

FWIW one option might be to use PeaZip. Maybe this is worth packaging ourselves ( conda-forge/staged-recipes#21563 )?

[dholth]

Like tar -tvf somefile?

Could we re-use part of Python's built-in tar command line utility? % python -m tarfile -l iniconfig-1.1.1-pyhd3eb1b0_0.tar.bz2

[jaimergp]

Adding cph list in #236, which should make things super easy to set up a CI step like proposed above (with no extraction cost).

[jaimergp]

While researching for that PR also found out that you can use tar for this too:

# .tar.bz2:
tar tvf path/to/artifact.tar.bz2  
# .conda: package contents
tar xOf path/to/artifact.conda "pkg-*.tar.zst" | tar tv 
# .conda: info/ metadata
tar xOf path/to/artifact.conda "info-*.tar.zst" | tar tv 

In case it varies across platforms, this is on macOS with:

$ tar --version
bsdtar 3.5.3 - libarchive 3.5.3 zlib/1.2.12 liblzma/5.0.5 bz2lib/1.0.8 

[dholth]

Mind blown, extracting .zip with the libarchive tar command! 🤯

[jakirkham]

Adding cph list in #236, which should make things super easy to set up a CI step like proposed above (with no extraction cost).

Nice! Thanks Jaime 🙏

Wonder if it would be worth adding a cph tree (like Linux's tree) to get a better sense of layout

[jaimergp]

Turns out you can use tree for this too!

$ cph list ~/Downloads/conda-22.9.0+93_gb267263a0-py38_0.tar.bz2 | tree --fromfile
.
├── bin
│   ├── activate
│   ├── conda
│   ├── conda-env
│   └── deactivate
├── condabin
│   └── conda
├── etc
│   ├── fish
│   │   └── conf.d
│   │       └── conda.fish
│   └── profile.d
│       ├── conda.csh
│       └── conda.sh
├── info
│   ├── about.json
│   ├── files
│   ├── git
│   ├── has_prefix
│   ├── hash_input.json
│   ├── index.json
│   ├── licenses
│   │   └── LICENSE.txt
│   ├── paths.json
│   ├── recipe
...

[dholth]

@jaimergp fixed?

[jaimergp]

I think so! The only thing missing is some documentation, I'd say? I can open a PR if you agree.