Piped input from wgrib2 read via /vsistdin/ outputs error; produced output is correct though

[jomey]

Expected behavior and actual behavior.

When reading input via /vsistdin/ from a piped output by the wgrib2 command, gdal detects on more Message than the piped output includes. This produces the following error output:

Warning: Inside GRIB2Inventory, Message # 2
ERROR: Ran out of file reading SECT0

The piped input has only one messages and the produced output file is correct.

Steps to reproduce the problem.

1. Use the attached sample file
2. Execute the following command:
   wgrib2 hrrr.t00z.wrfsfcf01.grib2 -match DSWRF -grib - | gdalwarp -t_srs EPSG:32613 -tr 50 50 /vsistdin/ warped.tif

The above will produce the stated error message above.

Operating system

Tested on:

• Linux version 5.17.5-76051705-generic ([email protected]) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #202204271406165344057622.04~6277a18 SMP PREEMPT Wed May 25 01
• Linux version 4.18.0-348.20.1.el8_5.x86_64 ([email protected]) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-3) (GCC)) NAS fixes #1 SMP Thu Mar 10 20:59:28 UTC 2022

GDAL version and provenance

Tested with two versions Installed via conda-forge:

• libgdal=3.5.0=hd131f80_2
  gdal=3.5.0=py310h8172e47_2
• gdal=3.4.3=py310hce6f0df_0
  libgdal=3.4.3=h56144a5_0

Wgrib2 command version:

• wgrib2=3.1.1=h126ae06_0

hrrr.t00z.wrfsfcf01.zip

[jomey]

As an update, I also tested this with gdal_translate, only saving the piped input to a file, and it has the same behavior. Seems to be related to the GRIB2 input reader implementation.

[rouault]

Due to how the driver works, it is not expected that reading from /vsistdin/ would work since we need to do a first scanning pass to do the inventory of concatened message, before being able to use the file. We should probably just reject attempts at reading GRIB from /vsistdin/ (or have a specific open option to mean that only one message is provided)

[jomey]

I have been accepting the error message for now since the output is correct. Hence, I would vote for an option to indicate expected number of messages.

[rouault]

Hence, I would vote for an option to indicate expected number of messages.

that would rather be a SINGLE_MESSAGE=YES kind of option. We need to parse each GRIB2 message header to build a multi-band GDAL dataset.
You're lucky that it even works with a single message since you process files below 1 MB, which is the limit of caching of /vsistdin/

[jomey]

You're lucky that it even works with a single message since you process files below 1 MB, which is the limit of caching of /vsistdin/

I stumbled upon this while doing a rather different call, where I first concat many wgrib files, then filter with wgrib, and then warp.
Something like this:

cat *.wgrib2 | wgrib2 - -match _MESSAGE_ -grib - | gdal_warp -t_srs EPSG:4326 /vsistdin/ out.tif

Using named pipes instead of /vsistdin/ did not work at all.

[rouault]

There's a usage issue of wgrib2. If you do "-grid -", you'll get a mix in stdout of the GRIB file and the indexing information it also outputs to stdout. Luckily, the GDAL GRIB reader is able to cope with that (with just the annoyance of a warning)
You'll also want to use '-inv /dev/null' to avoid generating the indexing information.
Otherwise, regarding using /vsistdin/ and GRIB files, this meets the issue discussed in #751 , and for which I've a pending enhancement/fix in #5930
So closing that ticket

[jomey]

Makes sense. Thank you for the explanation, Even!