Validation failures are contingent on presence of <file_size> attribute in <File> class

[cgobat]

Checked for duplicates

Seems closely related to #616, but the problem persists, despite closure of that issue.

🐛 Describe the bug

@katecrombie and I have been unable to figure this out. I am not sure if it's a bug with validate, or the IM, or what, but we figured this was a good place to start.

The calibration software for one of the Lucy mission's instruments uses plain-text calibration files that contain a variety of parameters and coefficients used by the processing pipeline. The file contents are organized in a reliable/well-structured way, and can be fully described by a series of Table_Delimited and Stream_Text classes in a PDS4 label.

However, if an overall <file_size> attribute is is included in the <File> class (as is standard practice), the label fails validation due to the following error(s):

ERROR  [error.label.invalid_object_definition]   line 2: Invalid object definition for object #2.  The previously defined object ends at byte 94. Offset defined in label: 24
....

Each individual data object (Table_Delimited or Stream_Text) has a correctly-defined <offset> and <object_length>, and simply omitting the <file_size> enables it to pass validation without complaint.

🕵️ Expected behavior

I expected that inclusion/omission of the <file_size> attribute in the <File> class in the <File_Area_Observational> section would not have an impact on validation failure/passage.

📜 To Reproduce

1. Create an XML label for a product that contains several Table_Delimited and/or Stream_Text objects
2. In the <File> class, include a <file_size> attribute (with unit="byte") with a value equal to the product's file size, in bytes
3. Run validate, and note failure
4. Remove or comment out the <file_size> attribute from the label completely
5. Re-run validate, and note passing result

🖥 Environment Info

Operating System: Issue replicated on both MacOSX and Ubuntu 20.04

📚 Version of Software Used

validate versions 3.1.1 and 3.2.0

🩺 Test Data / Additional context

Attached is a zip file containing a dummy data product of the same format as described, with two corresponding labels whose only differences are the minor version number of their VID (so that both can be validated simultaneously), and whether or not the <file_size> attribute is present on line 61. A validation report showing that one passes while the other fails is also included.

🦄 Related requirements

⚙️ Engineering Details

[cgobat]

Is it expected that NASA-PDS/pds4-jparser#89 would have addressed the case where all three of file_size, offset, and object_length are defined for ≥1 Table_Delimited in a file? @al-niessner

[jordanpadams]

@cgobat looks like a bug. we will have a look.

[cgobat]

@jordanpadams thanks for the speedy resolution to this! Question about validate's release model: do fixes like this get backported to current versions, or do we just wait for the next release/patch version?

[jordanpadams]

@cgobat you can either wait for the next stable release (in 2-4 weeks) or you can try out the "unstable" SNAPSHOT version. the "unstable" version should work fine, but it has not yet been rigorously tested.

[rchenatjpl]

@jordanpadams Has this been fixed? The latest snapshot still has this bug (using File_Area_Observational/File/file_size as the size of the first object) on the attached.

PDS Validate Tool Report
Configuration:
   Version                       3.4.0-SNAPSHOT
   Date                          2023-12-01T08:43:23Z
Parameters:
   Targets                       [file:/Users/rchen/Desktop/RSS001E01_2031066T0241_EURGRVL20XXXXXXGSFC_COV010.xml]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/rchen/Desktop/validate-3.4.0-SNAPSHOT/resources/registered_context_products.json
Product Level Validation Results
  FAIL: file:/Users/rchen/Desktop/RSS001E01_2031066T0241_EURGRVL20XXXXXXGSFC_COV010.xml
      ERROR  [error.label.invalid_object_definition]   line 2: Invalid object definition for object #2.  The previously defined object ends at byte 354958. Offset defined in label: 25
      ERROR  [error.label.invalid_object_definition]   line 3: Invalid object definition for object #3.  The previously defined object ends at byte 354958. Offset defined in label: 7751
        1 product validation(s) completed
Summary:
  2 error(s)
  0 warning(s)
  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped
  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
  Message Types:
    2            error.label.invalid_object_definition
End of Report
Completed execution in 4039 ms

Archive.zip

[jordanpadams]

@rchenatjpl I think the issue you are seeing is tightly coupled to this one, but slightly different. created a new bug here: #781