Still struggling with local "$ref"s

[demitri]

Hi,

I'm still struggling with making local $refs work. I've tried basically every permutation I can think of or have read about (see: #570 and here and here). Here is my minimum example; all files placed in the same directory:

file: gaia_dr2_record_response.json

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "https://trillianverse.org/schemas/datasets/gaia/gaia_dr2_record_response.json",
    "title": "Gaia DR2 Source Catalog Record Schema",
    "type": "object",
    "additionalProperties" : false,
    "properties" : {
	    "$ref" : "gaia_dr2_properties.json#/definitions/GaiaDR2Properties/properties"
	}
}

file: gaia_dr2_properties.json

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "https://trillianverse.org/schemas/datasets/gaia/gaia_dr2_properties.json",
    "type": "object",
    "definitions": {
        "GaiaDR2Properties" : {
            "properties": {
                "solution_id" : {
                    "type": "integer",
                    "enum" : [1635721458409799680]
                },
                "designation" : {
                    "type": "string",
                    "pattern": "^Gaia DR2 [0-9]{19}$"
                },
                "source_id" : {
                    "type": "integer",
                    "minimum" : 4295806720,
                    "maximum" : 6917528997577384320
                }
            }    
        }
    }
}

file: test.py

#!/usr/bin/env python

import os
import json

from jsonschema import RefResolver, Draft7Validator

with open("gaia_dr2_properties.json") as schema_file:
 	properties_schema = json.loads(schema_file.read())
	
with open("gaia_dr2_record_response.json") as schema_file:
	response_schema = json.loads(schema_file.read())

data_string = '''{
  "source_id": 4038848092638268032,
  "solution_id": 1635721458409799680,
  "designation": "Gaia DR2 1635721458409799680"
}'''
data = json.loads(data_string)

file_path = os.path.abspath(__file__)
base_uri = f"file://{os.path.dirname(file_path)}/gaia_dr2_record_response.json"
print(f"\nbase uri = '{base_uri}'\n")

# do I need to load the $ref schema? these don't help
store = {
	"gaia_dr2_properties.json":properties_schema,
	"file://{os.path.dirname(file_path)}/gaia_dr2_record_response.json":properties_schema,
	"file:/{os.path.dirname(file_path)}/gaia_dr2_record_response.json":properties_schema,
}

resolver = RefResolver(base_uri=base_uri, referrer=response_schema, store=store)
validator = Draft7Validator(response_schema, resolver=resolver)

# this throws an error
#validator.check_schema(response_schema)

validator.validate(data)

I've tried many variations on base_uri in the resolver including path to files and URI to the file itself. I get that the RefResolver needs to know how to resolve the reference, but I don't know how it's working to know what to properly pass to it. Seemingly nothing I pass to RefResolver makes a difference (even base_uri='ffff'!).

Also, the I try to check the schema (commented out code above), it gives me this error which I'm not sure what to make of:

jsonschema.exceptions.SchemaError: 'gaia_dr2_properties.json#/definitions/GaiaDR2Properties/properties' is not of type 'object', 'boolean'

When I simply copy and paste the referred to segment into the original schema, the check and validation works.

Any help appreciated!

[Julian]

Hey @demitri -- sorry you're having trouble.

I do apologize -- I wish I had time and energy to be able to give direct support, but unfortunately I just don't at this point. There are some options for how you may get some help -- StackOverflow is unfortunately one, and the JSON Schema (specification) slack is another one, which has lots of folks who may be able to give direct real-time help.

It's also unfortunately possible you're encountering a known $ref issue. There are a few in the issue tracker that you may look over. As far as I know, none are show stoppers, as all can be evaded (mostly by using the store functionality on RefResolvers), but there are at least 2 distinct known issues you may find, and probably more than that number of tickets talking about them which ultimately come down to the same 2 issues.

Going to close this as until/unless you find a specific bug (a new one besides the 2 above), it's just going to likely sit stale here on the issue tracker, but I certainly do hope you figure it out.

[demitri]

Hi @Julian. I spent a non-trivial amount of time working on this; I am aware of the other issues and have read through them several times. I have stepped through the jsonschema code to see what the problem might be.

What is not clear is what the code should look like, regardless of whether it works or not. The $refs simply fail silently, no matter what I specify. If you can't point to any example of a $ref working (I can't find one), can you suggest what the code should look like. It's just been a lot of poking around in the dark.

[mrtj]

Hello, I came up with a small helper function that preloads the local schema fragments to RefResolver cache: https://gist.github.com/mrtj/d59812a981da17fbaa67b7de98ac3d4b