Break optional/ into more granular directories.

README.md

@@ -76,15 +76,13 @@ Here is a single *test case*, containing one or more tests:
 
 ### Subdirectories Within Each Draft
 
-There is currently only one additional subdirectory that may exist within each draft test directory.
+There are currently four additional subdirectories that exist within draft directories.
+They are:
 
-This is:
-
-1. `optional/`: Contains tests that are considered optional.
-
-Note, the `optional/` subdirectory today conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in
-which case they are not truly optional in such a language).
-In the future this directory structure will be made richer to reflect these differences more clearly.
+1. `should/`: Contain tests which the specification designates as recommended behavior, either explicitly or otherwise clearly
+2. `may/`: Contains tests which the specification designates as optional behavior, again explicitly or otherwise clearly
+3. `additional/`: Contains additional tests or subdirectories whose applicability is left for implementations to confirm, either because they test additional vocabularies, apply only to certain languages or environments, or because the strength of their recommendation isn't clear from the specification
+4. `alternatives/`: Contains additional subdirectories of tests which are mutually exclusive to each other. Implementations should often elect a single file within each directory, corresponding to the choice they have made. As a concrete example, the specification in draft 2020 dictates that given `{"$vocabulary": {"foo": false}, ...}`, all implementations regardless of whether they support `"foo"` should proceed with processing. The validation result, however, now depends on whether the implementation understands the vocabulary or not, so the `alternatives` directory has files for each possibility (though not including the third possibility that the implementation disregards the SHOULD and refuses to process the schema entirely).
 
 ## Using the Suite to Test a Validator Implementation
 
@@ -153,7 +151,7 @@ If your implementation supports multiple versions, run the above procedure for e
     ```
 
 2. Test cases found within [special subdirectories](#subdirectories-within-each-draft) may require additional configuration to run.
-   In particular, tests within the `optional/format` subdirectory may require implementations to change the way they treat the `"format"`keyword (particularly on older drafts which did not have a notion of vocabularies).
+   In particular, tests within the `additional/format-assertion` subdirectory may require implementations to change the way they treat the `"format"`keyword (particularly on older drafts which did not have a notion of vocabularies).
 
 ### Invariants & Guarantees
 

tests/draft2020-12/optional/format-assertion.json → tests/draft-next/alternatives/false-vocabulary-format-assertion/is-known.json

@@ -1,8 +1,8 @@
 [
     {
         "description": "schema that uses custom metaschema with format-assertion: false",
+        "comment": "For implementations which support the format assertion vocabulary",
         "schema": {
-            "$id": "https://schema/using/format-assertion/false",
             "$schema": "http://localhost:1234/draft2020-12/format-assertion-false.json",
             "format": "ipv4"
         },
@@ -21,8 +21,8 @@
     },
     {
         "description": "schema that uses custom metaschema with format-assertion: true",
+        "comment": "For implementations which support the format assertion vocabulary",
         "schema": {
-            "$id": "https://schema/using/format-assertion/true",
             "$schema": "http://localhost:1234/draft2020-12/format-assertion-true.json",
             "format": "ipv4"
         },

tests/draft-next/alternatives/false-vocabulary-format-assertion/is-unknown.json

@@ -0,0 +1,22 @@
+[
+    {
+        "description": "schema that uses custom metaschema with format-assertion: false",
+        "comment": "For implementations which do not support the format assertion vocabulary",
+        "schema": {
+            "$schema": "http://localhost:1234/draft2020-12/format-assertion-false.json",
+            "format": "ipv4"
+        },
+        "tests": [
+            {
+                "description": "format-assertion: false: valid string",
+                "data": "127.0.0.1",
+                "valid": true
+            },
+            {
+                "description": "format-assertion: false: invalid string",
+                "data": "not-an-ipv4",
+                "valid": true
+            }
+        ]
+    }
+]

tests/draft-next/optional/format-assertion.json → tests/draft2020-12/alternatives/false-vocabulary-format-assertion/is-known.json

@@ -1,9 +1,9 @@
 [
     {
         "description": "schema that uses custom metaschema with format-assertion: false",
+        "comment": "For implementations which support the format assertion vocabulary",
         "schema": {
-            "$id": "https://schema/using/format-assertion/false",
-            "$schema": "http://localhost:1234/draft-next/format-assertion-false.json",
+            "$schema": "http://localhost:1234/draft2020-12/format-assertion-false.json",
             "format": "ipv4"
         },
         "tests": [
@@ -21,9 +21,9 @@
     },
     {
         "description": "schema that uses custom metaschema with format-assertion: true",
+        "comment": "For implementations which support the format assertion vocabulary",
         "schema": {
-            "$id": "https://schema/using/format-assertion/true",
-            "$schema": "http://localhost:1234/draft-next/format-assertion-true.json",
+            "$schema": "http://localhost:1234/draft2020-12/format-assertion-true.json",
             "format": "ipv4"
         },
         "tests": [

tests/draft2020-12/alternatives/false-vocabulary-format-assertion/is-unknown.json

@@ -0,0 +1,22 @@
+[
+    {
+        "description": "schema that uses custom metaschema with format-assertion: false",
+        "comment": "For implementations which do not support the format assertion vocabulary",
+        "schema": {
+            "$schema": "http://localhost:1234/draft2020-12/format-assertion-false.json",
+            "format": "ipv4"
+        },
+        "tests": [
+            {
+                "description": "format-assertion: false: valid string",
+                "data": "127.0.0.1",
+                "valid": true
+            },
+            {
+                "description": "format-assertion: false: invalid string",
+                "data": "not-an-ipv4",
+                "valid": true
+            }
+        ]
+    }
+]

tests/draft2020-12/optional/ecmascript-regex.json → tests/draft2020-12/should/ecmascript-regex.json

@@ -1,6 +1,7 @@
 [
     {
         "description": "ECMA 262 regex $ does not match trailing newline",
+        "comment": "§6.4: ' These regular expressions SHOULD be valid according to the regular expression dialect described in ECMA-262, section 21.2.1'",
         "schema": {
             "type": "string",
             "pattern": "^abc$"

tests/draft7/alternatives/content/annotation.json

@@ -0,0 +1,77 @@
+[
+    {
+        "description": "validation of string-encoded content based on media type",
+        "schema": {
+            "contentMediaType": "application/json"
+        },
+        "tests": [
+            {
+                "description": "a valid JSON document",
+                "data": "{\"foo\": \"bar\"}",
+                "valid": true
+            },
+            {
+                "description": "an invalid JSON document",
+                "data": "{:}",
+                "valid": true
+            },
+            {
+                "description": "ignores non-strings",
+                "data": 100,
+                "valid": true
+            }
+        ]
+    },
+    {
+        "description": "validation of binary string-encoding",
+        "schema": {
+            "contentEncoding": "base64"
+        },
+        "tests": [
+            {
+                "description": "a valid base64 string",
+                "data": "eyJmb28iOiAiYmFyIn0K",
+                "valid": true
+            },
+            {
+                "description": "an invalid base64 string (% is not a valid character)",
+                "data": "eyJmb28iOi%iYmFyIn0K",
+                "valid": true
+            },
+            {
+                "description": "ignores non-strings",
+                "data": 100,
+                "valid": true
+            }
+        ]
+    },
+    {
+        "description": "validation of binary-encoded media type documents",
+        "schema": {
+            "contentMediaType": "application/json",
+            "contentEncoding": "base64"
+        },
+        "tests": [
+            {
+                "description": "a valid base64-encoded JSON document",
+                "data": "eyJmb28iOiAiYmFyIn0K",
+                "valid": true
+            },
+            {
+                "description": "a validly-encoded invalid JSON document",
+                "data": "ezp9Cg==",
+                "valid": true
+            },
+            {
+                "description": "an invalid base64 string that is valid JSON",
+                "data": "{}",
+                "valid": true
+            },
+            {
+                "description": "ignores non-strings",
+                "data": 100,
+                "valid": true
+            }
+        ]
+    }
+]