From 4d847260825a3711f799eb2d4883dc1ffe2ae5e8 Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 21 Feb 2024 15:08:28 +0000 Subject: [PATCH 01/19] Bootstrapped field order section --- content/docs/specifications/table-schema.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 4f21df08..c41ff9a4 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -69,7 +69,7 @@ For example, `constraints` `SHOULD` be tested on the logical representation of d A Table Schema is represented by a descriptor. The descriptor `MUST` be a JSON `object` (JSON is defined in [RFC 4627](http://www.ietf.org/rfc/rfc4627.txt)). -It `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor (as defined below). The order of elements in `fields` array `SHOULD` be the order of fields in the CSV file. The number of elements in `fields` array `SHOULD` be the same as the number of fields in the CSV file. +It `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor (as defined below). The descriptor `MAY` have the additional properties set out below and `MAY` contain any number of other properties (not defined in this specification). @@ -101,6 +101,10 @@ The following is an illustration of this structure: } ``` +## Field Order + +The order of elements in `fields` array `SHOULD` be the order of fields in the CSV file. The number of elements in `fields` array `SHOULD` be the same as the number of fields in the CSV file. + ## Field Descriptors A field descriptor `MUST` be a JSON `object` that describes a single field. The From fe48c9621f81b0cf8617b8baa3865a03e2507d81 Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 21 Feb 2024 15:47:56 +0000 Subject: [PATCH 02/19] Updted the spec --- content/docs/specifications/table-schema.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index c41ff9a4..2ca54587 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -69,7 +69,9 @@ For example, `constraints` `SHOULD` be tested on the logical representation of d A Table Schema is represented by a descriptor. The descriptor `MUST` be a JSON `object` (JSON is defined in [RFC 4627](http://www.ietf.org/rfc/rfc4627.txt)). -It `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor (as defined below). +### Fields + +A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor (as defined below). The descriptor `MAY` have the additional properties set out below and `MAY` contain any number of other properties (not defined in this specification). @@ -101,9 +103,14 @@ The following is an illustration of this structure: } ``` -## Field Order +### Partial + +A Table Schema descriptor `MAY` contain a property `partial`. This property `MUST` be boolean with default value is `false`. -The order of elements in `fields` array `SHOULD` be the order of fields in the CSV file. The number of elements in `fields` array `SHOULD` be the same as the number of fields in the CSV file. +Depending on the value of the `partial` property, different rules for mapping Table Schema `fields` on data source columns `MUST` be applied: + +- **false** (default): The order of elements in `fields` array `MUST` be the order of columns in the data source. The number of elements in `fields` array `MUST` be the same as the number of columns in data source. Every element in the `fields` array `MUST` be mapped to corresponsing column in data source based on their order. +- **true**: The order of elements in `fields` array `MAY` be arbitrary. The number of elements in `fields` array `MAY` be arbitrary. Every element in the `fields` array `MUST` be mapped to corresponding column in data source based on their names. For example, for a CSV data source the mapping `MUST` be based on the header row values. ## Field Descriptors @@ -132,7 +139,11 @@ The field descriptor `object` `MAY` contain any number of other properties. Some ### `name` -The field descriptor `MUST` contain a `name` property. This property `SHOULD` correspond to the name of field/column in the data file (if it has a name). As such it `SHOULD` be unique (though it is possible, but very bad practice, for the data file to have multiple columns with the same name). `name` `SHOULD NOT` be considered case sensitive in determining uniqueness. However, since it corresponds to the name of the field in the data file it may be important to preserve case. +The field descriptor `MUST` contain a `name` property and it `MUST` be unique amongst other fields in this Table Schema. This property `SHOULD` correspond to the name of a column in the data file if it has a name. + +:::note[Backward Compatibility] +If the `name` properties are not unique amongst a Table Schema a data consumer `MUST NOT` interpret it as an invalid descriptor as duplicate `name` properties were allowed in the `v1.0` of the specification. +::: ### `title` From f57f41185d29a98251214a75da0e3d6438265e15 Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 21 Feb 2024 15:51:54 +0000 Subject: [PATCH 03/19] Updated the profile --- profiles/dictionary/schema.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/profiles/dictionary/schema.yaml b/profiles/dictionary/schema.yaml index ee2ee7a6..910e8fbf 100644 --- a/profiles/dictionary/schema.yaml +++ b/profiles/dictionary/schema.yaml @@ -40,6 +40,8 @@ tableSchema: } ] } + partial: + "$ref": "#/definitions/tableSchemaPartial" primaryKey: "$ref": "#/definitions/tableSchemaPrimaryKey" uniqueKeys: @@ -117,6 +119,9 @@ tableSchemaField: - "$ref": "#/definitions/tableSchemaFieldArray" - "$ref": "#/definitions/tableSchemaFieldDuration" - "$ref": "#/definitions/tableSchemaFieldAny" +tableSchemaPartial: + type: boolean + default: false tableSchemaPrimaryKey: oneOf: - type: array From bb7011a410e19a008cfcd0b0e3013365b2c08e53 Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 21 Feb 2024 16:05:22 +0000 Subject: [PATCH 04/19] Fixed styling --- content/docs/specifications/table-schema.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 2ca54587..3c5b5aa5 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -71,9 +71,9 @@ A Table Schema is represented by a descriptor. The descriptor `MUST` be a JSON ` ### Fields -A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor (as defined below). +A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor as defined below. -The descriptor `MAY` have the additional properties set out below and `MAY` contain any number of other properties (not defined in this specification). +The descriptor `MAY` have the additional properties set out below and `MAY` contain any number of other properties not defined in this specification. The following is an illustration of this structure: @@ -139,7 +139,7 @@ The field descriptor `object` `MAY` contain any number of other properties. Some ### `name` -The field descriptor `MUST` contain a `name` property and it `MUST` be unique amongst other fields in this Table Schema. This property `SHOULD` correspond to the name of a column in the data file if it has a name. +The field descriptor `MUST` contain a `name` property and it `MUST` be unique amongst other field names in this Table Schema. This property `SHOULD` correspond to the name of a column in the data file if it has a name. :::note[Backward Compatibility] If the `name` properties are not unique amongst a Table Schema a data consumer `MUST NOT` interpret it as an invalid descriptor as duplicate `name` properties were allowed in the `v1.0` of the specification. From 919206a8e6c1d63897f6b333f8f9ae815e506dfe Mon Sep 17 00:00:00 2001 From: roll Date: Fri, 23 Feb 2024 14:48:37 +0000 Subject: [PATCH 05/19] Rebased on two properties --- content/docs/specifications/table-schema.md | 29 ++++++++++++++++----- 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 3c5b5aa5..f0f68fcf 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -69,10 +69,6 @@ For example, `constraints` `SHOULD` be tested on the logical representation of d A Table Schema is represented by a descriptor. The descriptor `MUST` be a JSON `object` (JSON is defined in [RFC 4627](http://www.ietf.org/rfc/rfc4627.txt)). -### Fields - -A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor as defined below. - The descriptor `MAY` have the additional properties set out below and `MAY` contain any number of other properties not defined in this specification. The following is an illustration of this structure: @@ -103,6 +99,28 @@ The following is an illustration of this structure: } ``` +## Properties + +### `fields` + +A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor as defined below. + +The way Table Schema `fields` are mapped onto the data source fields are defined by the combination of the `exactFields` and `orderedFields` properties. By default, the least strict approach is applied i.e. fields in the data source `MAY` be unordered and have extra items. Enabling the properties below, a data publisher can enforce additional requirements to the data source. + +### `exactFields` + +A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `false`: + +- **false** (default): The number of fields in the data source `MUST` be equal or more than the number of elements in `fields` array. +- **true**: The number of fields in the data source `MUST` be exactly the same as the number of elements in `fields` array. + +### `orderedFields` + +A Table Schema descriptor `MAY` contain a property `orderedFields` that `MUST` be boolean with default value `false`: + +- **false** (default): Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their names. +- **true**: Each element in the `fields` array `MUST` be mapped to the corresponsing field in the data source based on their order. + ### Partial A Table Schema descriptor `MAY` contain a property `partial`. This property `MUST` be boolean with default value is `false`. @@ -110,9 +128,8 @@ A Table Schema descriptor `MAY` contain a property `partial`. This property `MUS Depending on the value of the `partial` property, different rules for mapping Table Schema `fields` on data source columns `MUST` be applied: - **false** (default): The order of elements in `fields` array `MUST` be the order of columns in the data source. The number of elements in `fields` array `MUST` be the same as the number of columns in data source. Every element in the `fields` array `MUST` be mapped to corresponsing column in data source based on their order. -- **true**: The order of elements in `fields` array `MAY` be arbitrary. The number of elements in `fields` array `MAY` be arbitrary. Every element in the `fields` array `MUST` be mapped to corresponding column in data source based on their names. For example, for a CSV data source the mapping `MUST` be based on the header row values. -## Field Descriptors +## Field Properties A field descriptor `MUST` be a JSON `object` that describes a single field. The descriptor provides additional human-readable documentation for a field, as From f832abf5f83a31748612375b952668b910f2db44 Mon Sep 17 00:00:00 2001 From: roll Date: Fri, 23 Feb 2024 14:49:32 +0000 Subject: [PATCH 06/19] Removed partial --- content/docs/specifications/table-schema.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index f0f68fcf..e71db14a 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -121,14 +121,6 @@ A Table Schema descriptor `MAY` contain a property `orderedFields` that `MUST` b - **false** (default): Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their names. - **true**: Each element in the `fields` array `MUST` be mapped to the corresponsing field in the data source based on their order. -### Partial - -A Table Schema descriptor `MAY` contain a property `partial`. This property `MUST` be boolean with default value is `false`. - -Depending on the value of the `partial` property, different rules for mapping Table Schema `fields` on data source columns `MUST` be applied: - -- **false** (default): The order of elements in `fields` array `MUST` be the order of columns in the data source. The number of elements in `fields` array `MUST` be the same as the number of columns in data source. Every element in the `fields` array `MUST` be mapped to corresponsing column in data source based on their order. - ## Field Properties A field descriptor `MUST` be a JSON `object` that describes a single field. The From 94c9605db964510a624df2ba021c8a2c0e0983b9 Mon Sep 17 00:00:00 2001 From: roll Date: Fri, 23 Feb 2024 15:05:11 +0000 Subject: [PATCH 07/19] Added articles --- content/docs/specifications/table-schema.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index e71db14a..52e406f7 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -105,14 +105,14 @@ The following is an illustration of this structure: A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor as defined below. -The way Table Schema `fields` are mapped onto the data source fields are defined by the combination of the `exactFields` and `orderedFields` properties. By default, the least strict approach is applied i.e. fields in the data source `MAY` be unordered and have extra items. Enabling the properties below, a data publisher can enforce additional requirements to the data source. +The way Table Schema `fields` are mapped onto the data source fields are defined by the combination of the `exactFields` and `orderedFields` properties. By default, the least strict approach is applied i.e. fields in the data source `MAY` be unordered and have extra items. Enabling the properties below, a data producer can enforce additional requirements to the data source. ### `exactFields` A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `false`: -- **false** (default): The number of fields in the data source `MUST` be equal or more than the number of elements in `fields` array. -- **true**: The number of fields in the data source `MUST` be exactly the same as the number of elements in `fields` array. +- **false** (default): The number of fields in the data source `MUST` be equal or more than the number of elements in the `fields` array. +- **true**: The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. ### `orderedFields` From 8657f3b7e59c743e8b79d0c5e31990f26ec19739 Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 14 Mar 2024 15:03:40 +0000 Subject: [PATCH 08/19] Update content/docs/specifications/table-schema.md Co-authored-by: Peter Desmet --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 52e406f7..bd140a5d 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -119,7 +119,7 @@ A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be A Table Schema descriptor `MAY` contain a property `orderedFields` that `MUST` be boolean with default value `false`: - **false** (default): Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their names. -- **true**: Each element in the `fields` array `MUST` be mapped to the corresponsing field in the data source based on their order. +- **true**: Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their order. ## Field Properties From f04b7ede3a8acd253cdb0a84204116118cd701ac Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 14 Mar 2024 15:19:34 +0000 Subject: [PATCH 09/19] Fixed `exactFields` --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 9d023cd9..e8f9e262 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -111,7 +111,7 @@ The way Table Schema `fields` are mapped onto the data source fields are defined A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `false`: -- **false** (default): The number of fields in the data source `MUST` be equal or more than the number of elements in the `fields` array. +- **false** (default): The number of fields in the data source can be arbitrary i.e. less, equal or more than the number of elements in the `fields` array. - **true**: The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. ### `orderedFields` From f6706140be1f2dc8ba39f8e78db10850e9bff3b4 Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 14 Mar 2024 15:21:48 +0000 Subject: [PATCH 10/19] Revert "Fixed `exactFields`" This reverts commit f04b7ede3a8acd253cdb0a84204116118cd701ac. --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index e8f9e262..9d023cd9 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -111,7 +111,7 @@ The way Table Schema `fields` are mapped onto the data source fields are defined A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `false`: -- **false** (default): The number of fields in the data source can be arbitrary i.e. less, equal or more than the number of elements in the `fields` array. +- **false** (default): The number of fields in the data source `MUST` be equal or more than the number of elements in the `fields` array. - **true**: The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. ### `orderedFields` From 64bd8c08c2109cda712a855edad305f1b709da10 Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 14 Mar 2024 15:35:39 +0000 Subject: [PATCH 11/19] Revert "Revert "Fixed `exactFields`"" This reverts commit f6706140be1f2dc8ba39f8e78db10850e9bff3b4. --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 9d023cd9..e8f9e262 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -111,7 +111,7 @@ The way Table Schema `fields` are mapped onto the data source fields are defined A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `false`: -- **false** (default): The number of fields in the data source `MUST` be equal or more than the number of elements in the `fields` array. +- **false** (default): The number of fields in the data source can be arbitrary i.e. less, equal or more than the number of elements in the `fields` array. - **true**: The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. ### `orderedFields` From 528978af9efb537e06316766705cbc86113e22f6 Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 14 Mar 2024 16:29:39 +0000 Subject: [PATCH 12/19] Reverce defaults --- content/docs/specifications/table-schema.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index e8f9e262..1478aa22 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -105,21 +105,21 @@ The following is an illustration of this structure: A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor as defined below. -The way Table Schema `fields` are mapped onto the data source fields are defined by the combination of the `exactFields` and `orderedFields` properties. By default, the least strict approach is applied i.e. fields in the data source `MAY` be unordered and have extra items. Enabling the properties below, a data producer can enforce additional requirements to the data source. +The way Table Schema `fields` are mapped onto the data source fields are defined by the combination of the `exactFields` and `orderedFields` properties. By default, the most strict approach is applied i.e. fields in the data source `MUST` completely match the elements in the `fields` array regarding their amount and order. Enabling the properties below, a data producer can relax requirements to the data source. ### `exactFields` A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `false`: -- **false** (default): The number of fields in the data source can be arbitrary i.e. less, equal or more than the number of elements in the `fields` array. -- **true**: The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. +- **true** (default): The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. +- **false**: The number of fields in the data source can be arbitrary i.e. less, equal or more than the number of elements in the `fields` array. ### `orderedFields` A Table Schema descriptor `MAY` contain a property `orderedFields` that `MUST` be boolean with default value `false`: -- **false** (default): Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their names. -- **true**: Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their order. +- **true** (default): Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their order. +- **false**: Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their names. ## Field Properties From bd742520683f5668a9d9c00065812c6e978af139 Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 14 Mar 2024 16:31:12 +0000 Subject: [PATCH 13/19] Updated the profile --- profiles/dictionary/schema.yaml | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/profiles/dictionary/schema.yaml b/profiles/dictionary/schema.yaml index 23095c42..c273d10c 100644 --- a/profiles/dictionary/schema.yaml +++ b/profiles/dictionary/schema.yaml @@ -40,8 +40,10 @@ tableSchema: } ] } - partial: - "$ref": "#/definitions/tableSchemaPartial" + exactFields: + "$ref": "#/definitions/tableSchemaExactFields" + orderedFields: + "$ref": "#/definitions/tableSchemaOrderedFields" primaryKey: "$ref": "#/definitions/tableSchemaPrimaryKey" uniqueKeys: @@ -119,9 +121,12 @@ tableSchemaField: - "$ref": "#/definitions/tableSchemaFieldArray" - "$ref": "#/definitions/tableSchemaFieldDuration" - "$ref": "#/definitions/tableSchemaFieldAny" -tableSchemaPartial: +tableSchemaExactFields: type: boolean - default: false + default: true +tableSchemaOrderedFields: + type: boolean + default: true tableSchemaPrimaryKey: oneOf: - type: array From 6003b25008b7c11c78398cea855be9f4223930b1 Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 14 Mar 2024 16:32:46 +0000 Subject: [PATCH 14/19] Fixed typo --- content/docs/specifications/table-schema.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 1478aa22..df92983a 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -109,14 +109,14 @@ The way Table Schema `fields` are mapped onto the data source fields are defined ### `exactFields` -A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `false`: +A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `true`: - **true** (default): The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. - **false**: The number of fields in the data source can be arbitrary i.e. less, equal or more than the number of elements in the `fields` array. ### `orderedFields` -A Table Schema descriptor `MAY` contain a property `orderedFields` that `MUST` be boolean with default value `false`: +A Table Schema descriptor `MAY` contain a property `orderedFields` that `MUST` be boolean with default value `true`: - **true** (default): Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their order. - **false**: Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their names. From b6cfbba0e7077498216a85cc754d2ed4aa1840f3 Mon Sep 17 00:00:00 2001 From: roll Date: Mon, 18 Mar 2024 15:21:42 +0000 Subject: [PATCH 15/19] Rebased to `schema.fieldsMatch` --- content/docs/specifications/table-schema.md | 20 +++++++----------- profiles/dictionary/schema.yaml | 23 ++++++++++++--------- 2 files changed, 21 insertions(+), 22 deletions(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index df92983a..11d3ad74 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -105,21 +105,17 @@ The following is an illustration of this structure: A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor as defined below. -The way Table Schema `fields` are mapped onto the data source fields are defined by the combination of the `exactFields` and `orderedFields` properties. By default, the most strict approach is applied i.e. fields in the data source `MUST` completely match the elements in the `fields` array regarding their amount and order. Enabling the properties below, a data producer can relax requirements to the data source. +The way Table Schema `fields` are mapped onto the data source fields are defined by the `fieldsMatch` property. By default, the most strict approach is applied i.e. fields in the data source `MUST` completely match the elements in the `fields` array regarding their amount and order. Using different options below, a data producer can relax requirements to the data source. -### `exactFields` +### `fieldsMatch` -A Table Schema descriptor `MAY` contain a property `exactFields` that `MUST` be boolean with default value `true`: +A Table Schema descriptor `MAY` contain a property `fieldsMatch` that `MUST` be a string with the following possible values and the `exact` value by default: -- **true** (default): The number of fields in the data source `MUST` be exactly the same as the number of elements in the `fields` array. -- **false**: The number of fields in the data source can be arbitrary i.e. less, equal or more than the number of elements in the `fields` array. - -### `orderedFields` - -A Table Schema descriptor `MAY` contain a property `orderedFields` that `MUST` be boolean with default value `true`: - -- **true** (default): Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their order. -- **false**: Each element in the `fields` array `MUST` be mapped to the corresponding field in the data source based on their names. +- **exact** (default): The data source `MUST` have exactly the same fields as defined in the `fields` array. Fields `MUST` be mapped by their order. +- **equal**: The data source `MUST` have exactly the same fields as defined in the `fields` array. Fields `MUST` be mapped by their names. +- **subset**: The data source `MUST` have all the fields defined in the `fields` array. Fields `MUST` be mapped by their names. +- **superset**: The data source `MUST` have only the fields defined in the `fields` array. Fields `MUST` be mapped by their names. +- **partial**: The data source `MUST` have at least 1 field defined in the `fields` array. Fields `MUST` be mapped by their names. ## Field Properties diff --git a/profiles/dictionary/schema.yaml b/profiles/dictionary/schema.yaml index c273d10c..1c5be714 100644 --- a/profiles/dictionary/schema.yaml +++ b/profiles/dictionary/schema.yaml @@ -40,10 +40,8 @@ tableSchema: } ] } - exactFields: - "$ref": "#/definitions/tableSchemaExactFields" - orderedFields: - "$ref": "#/definitions/tableSchemaOrderedFields" + fieldsMatch: + "$ref": "#/definitions/tableSchemaFieldsMatch" primaryKey: "$ref": "#/definitions/tableSchemaPrimaryKey" uniqueKeys: @@ -121,12 +119,17 @@ tableSchemaField: - "$ref": "#/definitions/tableSchemaFieldArray" - "$ref": "#/definitions/tableSchemaFieldDuration" - "$ref": "#/definitions/tableSchemaFieldAny" -tableSchemaExactFields: - type: boolean - default: true -tableSchemaOrderedFields: - type: boolean - default: true +tableSchemaFieldsMatch: + type: array + item: + type: string + enum: + - exact + - equal + - subset + - superset + - partial + default: exact tableSchemaPrimaryKey: oneOf: - type: array From 915867e7c0e8ccb856c5e65f45caeb659480db13 Mon Sep 17 00:00:00 2001 From: roll Date: Mon, 18 Mar 2024 15:25:48 +0000 Subject: [PATCH 16/19] Updated wording --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 11d3ad74..05406d11 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -115,7 +115,7 @@ A Table Schema descriptor `MAY` contain a property `fieldsMatch` that `MUST` be - **equal**: The data source `MUST` have exactly the same fields as defined in the `fields` array. Fields `MUST` be mapped by their names. - **subset**: The data source `MUST` have all the fields defined in the `fields` array. Fields `MUST` be mapped by their names. - **superset**: The data source `MUST` have only the fields defined in the `fields` array. Fields `MUST` be mapped by their names. -- **partial**: The data source `MUST` have at least 1 field defined in the `fields` array. Fields `MUST` be mapped by their names. +- **partial**: The data source `MUST` have at least one field defined in the `fields` array. Fields `MUST` be mapped by their names. ## Field Properties From a901c506c5d0efc9389fec7153e9bdeaa295996d Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 28 Mar 2024 08:28:48 +0000 Subject: [PATCH 17/19] Update content/docs/specifications/table-schema.md Co-authored-by: Peter Desmet --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 05406d11..0bf03c64 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -105,7 +105,7 @@ The following is an illustration of this structure: A Table Schema descriptor `MUST` contain a property `fields`. `fields` `MUST` be an array where each entry in the array is a field descriptor as defined below. -The way Table Schema `fields` are mapped onto the data source fields are defined by the `fieldsMatch` property. By default, the most strict approach is applied i.e. fields in the data source `MUST` completely match the elements in the `fields` array regarding their amount and order. Using different options below, a data producer can relax requirements to the data source. +The way Table Schema `fields` are mapped onto the data source fields are defined by the `fieldsMatch` property. By default, the most strict approach is applied, i.e. fields in the data source `MUST` completely match the elements in the `fields` array, both in number and order. Using different options below, a data producer can relax requirements for the data source. ### `fieldsMatch` From 3bba0571d34b3033b34cf53e205a13cd5c71277b Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 28 Mar 2024 08:29:14 +0000 Subject: [PATCH 18/19] Update content/docs/specifications/table-schema.md Co-authored-by: Peter Desmet --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 0bf03c64..ce2772ff 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -113,7 +113,7 @@ A Table Schema descriptor `MAY` contain a property `fieldsMatch` that `MUST` be - **exact** (default): The data source `MUST` have exactly the same fields as defined in the `fields` array. Fields `MUST` be mapped by their order. - **equal**: The data source `MUST` have exactly the same fields as defined in the `fields` array. Fields `MUST` be mapped by their names. -- **subset**: The data source `MUST` have all the fields defined in the `fields` array. Fields `MUST` be mapped by their names. +- **subset**: The data source `MUST` have all the fields defined in the `fields` array, but `MAY` have more. Fields `MUST` be mapped by their names. - **superset**: The data source `MUST` have only the fields defined in the `fields` array. Fields `MUST` be mapped by their names. - **partial**: The data source `MUST` have at least one field defined in the `fields` array. Fields `MUST` be mapped by their names. From a531108d0661f3cf7093386f11430fd7e5a2f350 Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 28 Mar 2024 08:29:41 +0000 Subject: [PATCH 19/19] Update content/docs/specifications/table-schema.md Co-authored-by: Peter Desmet --- content/docs/specifications/table-schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index ce2772ff..b8cc5f2c 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -114,7 +114,7 @@ A Table Schema descriptor `MAY` contain a property `fieldsMatch` that `MUST` be - **exact** (default): The data source `MUST` have exactly the same fields as defined in the `fields` array. Fields `MUST` be mapped by their order. - **equal**: The data source `MUST` have exactly the same fields as defined in the `fields` array. Fields `MUST` be mapped by their names. - **subset**: The data source `MUST` have all the fields defined in the `fields` array, but `MAY` have more. Fields `MUST` be mapped by their names. -- **superset**: The data source `MUST` have only the fields defined in the `fields` array. Fields `MUST` be mapped by their names. +- **superset**: The data source `MUST` only have fields defined in the `fields` array, but `MAY` have fewer. Fields `MUST` be mapped by their names. - **partial**: The data source `MUST` have at least one field defined in the `fields` array. Fields `MUST` be mapped by their names. ## Field Properties