rattler-build-ic way to test the existence of installed files?

[MementoRC]

I wonder if there is a way to implement a for loop with filters?

I have:

  qemu_keymaps: ${{ [
      "ar", "bepo", "cz", "da", "de", "de-ch", "en-gb", "en-us", "es", "et", "fi", "fo", "fr", "fr-be", "fr-ca",
      "fr-ch", "hr", "hu", "is", "it", "ja", "lt", "lv", "mk", "nl", "no", "pl", "pt", "pt-br", "ru", "th", "tr"
      ] | join(' ') }}

(I do not seem to be able to declare an array directly in context section)

I tried:

     tests:
      - package_contents:
          files:
            - include/qemu-plugin.h
            - share/qemu/trace-events-all
            - ${{ qemu_keymaps | split(' ') | list | replace('^', 'share/qemu' }}
            - bin/qemu-user-execve-aarch64

in the hope that it would insert a list in the files list

I also tried to use a for loop, but likely need to do a bash-c '...'?

- for bin in ${{ qemu_bins | split(" ") | list }}; do test -f ${QEMU_LD_PREFIX}/${bin} done;

[wolfv]

Tricky question.

We have one implicit underspecified behavior in hte requirements section where lists are always flattened.

We could potentially do something like this in the files section, too, so that if the Jinja emits something that looks like a valid YAML list, it would be flattened into the list.

E.g.

- ${{ ["foo", "bar"] }}
# becomes
- ["foo", "bar"]
# and then flattened to  
- foo
- bar

However, logic like this could also be easily expressed as a Python script in the tests...

[MementoRC]

I feel it would be a useful feature to add the behavior to the package_contents.{files,includes}. It is often the case that libraries provide lots of includes and shares. I usually organize them in arrays or hashes, in combination with outputs to select exactly what needs to be in the given package. But maybe, I just have a not so optimal way of thinking about it
Note that I seem to be hinting that this behavior would also be beneficial in the outputs.package.files (not sure about the syntax) ;P

[wolfv]

Yes, I can see the usefulness. Note that globs are perfectly OK (e.g. foo/*.sh or foo/{bar,baz}.sh etc. should work fine. Useful for the outputs.files but less useful for testing all with a certain prefix.

The downside of this idea is that the recipe becomes less statically known / validated.

[MementoRC]

Yes, I can see the usefulness. Note that globs are perfectly OK (e.g. foo/*.sh or foo/{bar,baz}.sh etc. should work fine. Useful for the outputs.files but less useful for testing all with a certain prefix.

The downside of this idea is that the recipe becomes less statically known / validated.

Right, if you select the files, then you would less likely need to test their existence in the installed package. In this particular case, I failed to get the common build to work, so I build within each package, so I can't select the files. I should resolve this little nag today with the python call and point you to the recipe for your review/suggestions. Thank you

[MementoRC]

FYI, I made progress, but it is quite a bit trickier then I thought, there seems to be update of " and ', which create some difficult diagnosis of python failure, but this worked:

          python -c "import os; [os.path.exists(os.path.join(os.environ['QEMU_LD_PREFIX'], bin)) for bin in '${{ qemu_bins }}'.split()]"

gives:

 │ │ │ Testing commands:
 │ │ │ + python -c 'import os; [os.path.exists(os.path.join(os.environ['\''QEMU_LD_PREFIX'\''], bin)) for bin in '\''bios.bin bios-256k.bin bios-microvm.bin qboot.rom vgabios.bin vgabios-cirrus.binvgabios-stdvga.bin vgabios-vmware.bin vgabios-qxl.b
 │ │ │ in vgabios-virtio.bin vgabios-ramfb.bin vgabios-bochs-display.bin vgabios-ati.bin openbios-sparc32 openbios-sparc64 openbios-ppc QEMU tcx.bin QEMU cgthree.bin pxe-e1000.rom pxe-eepro100.rom pxe-ne2k_pci.rom pxe-pcnet.rom pxe-rtl8139.rom pxe-
 │ │ │ virtio.rom efi-e1000.rom efi-eepro100.rom efi-ne2k_pci.rom efi-pcnet.rom efi-rtl8139.rom efi-virtio.rom efi-e1000e.rom efi-vmxnet3.rom qemu-nsis.bmp multiboot.bin multiboot_dma.bin linuxboot.bin linuxboot_dma.bin kvmvapic.bin pvh.bin s390-cc
 │ │ │ w.img s390-netboot.img slof.bin skiboot.lid palcode-clipper u-boot.e500 u-boot-sam460-20100605.bin qemu_vga.ndrv edk2-licenses.txt hppa-firmware.img hppa-firmware64.img opensbi-riscv32-generic-fw_dynamic.bin opensbi-riscv64-generic-fw_dynami
 │ │ │ c.bin npcm7xx_bootrom.bin vof.bin vof-nvram.bin bamboo.dtb canyonlands.dtb petalogix-s3adsp1800.dtb petalogix-ml'\''.split()]'

of course, this is not a 'test' but it should only take the addition of all() || exit 1?

[wolfv]

Sorry, I think I wasn't clear enough.

There are two ways this could be done right now:

tests:
  # note that this is a YAML block element
  - script: |
      {% set qemu_libs = ["foo", "bar"] %}
      # or any other bash like code
      {% for lib in qemu_libs %}
        test -f /usr/lib/qemu/${{ lib }}.so
      {% endfor %}

The other way would be with a Python script like this:

tests:
  - script:
      interpreter: python
      content: |
        from pathlib import Path
        qemu_libs = ["foo", "bar"]
        for lib in qemu_libs:
            assert Path(f"/usr/lib/qemu/{lib}.so").exists()
      requirements:
        run:
          - python

[wolfv]

Although it is possible that option 1 is currently broken.

[wolfv]

My initial version had a little bug in the jinja. We configured Jinja comments to look like #{{ I am a comment }}. But we can also just use a bash comment # i am a comment.

[MementoRC]

Oh, I see. Option 1 probably works, but I want to define the array (or string with all the elements) at the top, so that the definition of the list doesn't blob-up the testing actions, plus I would have to repeat it (unless jinja is allowed at any place, but I had some initial syntax error when trying to use jinja)

Option2 is neat, though , in this case I prefer the one-liner. When trying to create the array with ${{ array | split() |list }} injected into the python -c ... the quotes disappeared, which is the tricky part that I had not paid attention to - I surmise, it will work much more naturally for Option 2

With Option 2, can there be several script: with its dedicated content:?

[MementoRC]

FYI, this is the recipe that I am working on. It is a qemu with the execve() patch. This was needed for cross-building ZIG with CMake, since when it rebuilds itself in stage3, it emits an execve() call that would fail.
conda-forge/staged-recipes#27801

Also, I seem to have made some progress on the ZIG PPC64LE, thanks to Alexrp comment on one of the PRs

[wolfv]

You can have as many independent tests as you like (ie. separate script fields).

[MementoRC]

Somehow, there is an odd error when using the interpreter: syntax:

│ │ │ Testing commands:
 │ │ │ + python $SRC_DIR/conda_build_script.py
 │ │ │   File "$SRC_DIR/conda_build_script.py", line 3
 │ │ │     share = f"{os.environ.get(\'PREFIX\')}/share/qemu";
 │ │ │                                                      ^
 │ │ │ SyntaxError: invalid syntax

Also, the requirements entry is not supported in this case. For now, I will go with the python -c ... I kept the interpreter commented-out for your review, if time allows

[MementoRC]

I seem to run into an odd issue with the tests. I create the qemu-user, compile cross-compile for the target then try to run the code. It fails:

 │ │ │ + qemu-user-execve-ppc64le ./hello_from PPC64LE
 │ │ │ × error Script failed with status 1

However, when I manually run it it works fine. One issue was initially that the QEMU_LD_PREFIX set with the activate script was wrong, but now I cannot understand why it fails, there seem to be no information as to why it failed.

[wolfv]

Sometimes, stderr and stdout are interleaved in weird ways, and the output of the process is visible above the printline of the executed command. Could that help?