Add a history of storage format versions et. al. #5205

teo-tsirpanis · 2024-07-19T20:41:58Z

This PR updates the storage format specification to include information about all past versions. The work is divided into two parts. First, a new file was created that lists what changed in each storage format version, going in more detail than the existing table (which will be removed). After that, the fields in the various data structures will be updated to indicate in which version they were introduced.

I sourced the changes in each storage format by searching for the version(\(\)|_)? (>=?|<=?|==) \d+ regex in the code, as well as searching for references of these named constants.

I also made other small fixes to the spec as I found them.

Information about past format versions of groups will be added in another PR.

TYPE: NO_HISTORY
DESC: The storage format specification was updated to include information about all previous versions.

teo-tsirpanis

I linked each change to the pull request that introduced it.

teo-tsirpanis · 2024-07-19T20:43:26Z

format_spec/history.md

+
+## Version 22
+
+Introduced in TileDB 2.25


I did not add the commit or PR that bumped the version, because several of the changes did not happen at the same time the version was bumped, and the purpose of this spec is that you don't need to read the source code to understand it.

teo-tsirpanis · 2024-07-19T20:44:28Z

format_spec/history.md

+
+Introduced in TileDB 2.25
+
+* The _Current domain_ field was added to [array schemas](./array_schema.md).


teo-tsirpanis · 2024-07-19T20:48:41Z

format_spec/history.md

+
+Introduced in TileDB 2.19
+
+* The TileDB implementation has been updated to fix computing [tile metadata](./fragment.md#tile-mins-maxes) for fixed-size strings.


teo-tsirpanis · 2024-07-19T20:48:59Z

format_spec/history.md

+
+Introduced in TileDB 2.17
+
+* Arrays can have [enumerations](./enumeration.md).


teo-tsirpanis · 2024-07-22T18:10:03Z

format_spec/history.md

+
+Introduced in TileDB 2.19
+
+* The TileDB implementation has been updated to fix computing [tile metadata](./fragment.md#tile-mins-maxes) for nullable fixed-size strings on dense arrays.


General feedback: should we be including links to PRs in the Markdown itself? If you don't think that's appropriate, we can skip, but in the past open-source projects I've worked on, it was common practice to publicly include the PRs in all release documentation.

It's definitely not required as I said above, and personally I would prefer to clearly separate the storage format specification from its implementation, but I'm not strongly opposed to it.

@ihnorton what do you think?

teo-tsirpanis · 2024-07-22T18:47:14Z

format_spec/history.md

+
+* Deleting data from arrays is supported. Arrays can have [delete commit files](./delete_commit_file.md).
+* Updating data in arrays is supported. Arrays can have [update commit files](./update_commit_file.md).
+* Fragment metadata contain [tile processed conditions](./fragment.md#tile-processed-conditions).


teo-tsirpanis · 2024-07-22T18:48:07Z

format_spec/history.md

+
+Introduced in TileDB 2.12
+
+* Deleting data from arrays is supported. Arrays can have [delete commit files](./delete_commit_file.md).


format_spec/history.md

teo-tsirpanis · 2024-07-22T19:33:49Z

format_spec/history.md

+Introduced in TileDB 2.14
+
+* The _Order_ field was added to [attributes](./array_schema.md#attribute).
+* Cell offsets in dimensions or attributes of UTF-8 string type are not written in the offset tiles, if the RLE or dictionary filter exists in the filter pipeline. They are instead encoded as part of the data tile.


#3868

I am not entirely sure, from the storage format perspective does this change only mean that the offsets file will be empty?

teo-tsirpanis · 2024-07-22T19:37:10Z

format_spec/history.md

+
+* The [array file hierarchy](./array_file_hierarchy.md) was updated to store fragments, commits and consolidated fragment metadata in separate subdirectories.
+* The extension of commit files was changed to `.wrt`.
+* Cell offsets in dimensions or attributes of ASCII string type are not written in the offset tiles, if the RLE filter exists in the filter pipeline. They are instead encoded as part of the data tile.


#2969

The same with the dictionary filter happened in version 13, but I am not including it because that's when the dictionary filter was added.

teo-tsirpanis · 2024-07-23T16:32:03Z

format_spec/history.md

+    * The [MBR](./fragment.md#mbr) structure has been updated to support variable-sized dimensions.
+    * The _Dimension number_ and _R-Tree datatype_ fields have been removed from [R-Trees](./fragment.md#r-tree).
+* The _Allows dups_ field was added to [array schemas](./array_schema.md#array-schema-file).
+* Committed fragments are indicated by the presence of an `.ok` file in the array's directory, with the same [timestamped name](./timestamped_name.md) as the fragment.


teo-tsirpanis · 2024-07-23T16:33:20Z

format_spec/history.md

+    * The _Domain datatype_ field was removed from [domains](./array_schema.md#domain).
+    * The [MBR](./fragment.md#mbr) structure has been updated to support variable-sized dimensions.
+    * The _Dimension number_ and _R-Tree datatype_ fields have been removed from [R-Trees](./fragment.md#r-tree).
+* The _Allows dups_ field was added to [array schemas](./array_schema.md#array-schema-file).


teo-tsirpanis · 2024-07-23T16:34:03Z

format_spec/history.md

+
+Introduced in TileDB 2.2.3
+
+* [Data files](./fragment.md#data-file) are named by the name of their attribute or dimension, after percent encoding certain characters. These characters are `!#$%&'()*+,/:;=?@[]`, as specified in [RFC 3986](https://tools.ietf.org/html/rfc3986), as well as `"<>\|`, which are not allowed in Windows file names.


teo-tsirpanis · 2024-07-23T16:34:37Z

format_spec/history.md

+
+## Version 8
+
+Introduced in TileDB 2.2.3


Introduced in 2.3.x, backported to 2.2.x in #2054.

teo-tsirpanis · 2024-07-23T16:36:33Z

format_spec/history.md

+
+Introduced in TileDB 2.0
+
+* Dimensions are stored in separate [data files](./fragment.md#data-file).


I would try to mention zipped coordinates here.

My plan of documenting zipped coordinates is to update fragment.md in a subsequent PR to say something like "prior to format version 6 dimension data for sparse arrays are stored in a __coords file…".

What do you think?

Sounds good.

teo-tsirpanis · 2024-07-23T16:47:31Z

format_spec/history.md

+* Attributes can be nullable.
+    * The _Nullable_ and _Fill value validity_ fields were added to [attributes](./array_schema.md#attribute).
+    * The _Validity filters_ field was added to [array schemas](./array_schema.md#array-schema-file).
+    * Fragment metadata contain validity [tile offsets](./fragment.md#tile-offsets).


teo-tsirpanis · 2024-07-23T16:48:43Z

format_spec/history.md

+
+Introduced in TileDB 2.1
+
+* The _Fill value_ field was added to [attributes](./array_schema.md#attribute).


teo-tsirpanis · 2024-07-23T17:08:08Z

format_spec/history.md

+* Sparse arrays can have string dimensions and dimensions with different datatypes.
+    * The _Dimension datatype_, _Cell val num_ and _Filters_ fields were added to [dimensions](./array_schema.md#dimension).
+    * The _Domain size_ field was added to [dimensions](./array_schema.md#dimension). The domain of a dimension can have a variable size.
+    * The _Domain datatype_ field was removed from [domains](./array_schema.md#domain).


#1553 #1558

teo-tsirpanis · 2024-07-23T17:08:21Z

format_spec/history.md

+    * The [MBR](./fragment.md#mbr) structure has been updated to support variable-sized dimensions.
+    * The _Dimension number_ and _R-Tree datatype_ fields have been removed from [R-Trees](./fragment.md#r-tree).


#1514 #1573

format_spec/history.md

teo-tsirpanis · 2024-07-24T16:38:13Z

format_spec/history.md

+* The structure of [fragment metadata files](./fragment.md#fragment-metadata-file) was overhauled.
+    * The [footer](./fragment.md#footer) and [R-Tree](./fragment.md#r-tree) structures were added.
+    * The _Bounding coords_ field was removed.


#1197, specifically this diff

teo-tsirpanis · 2024-07-24T16:38:52Z

format_spec/history.md

+
+Introduced in TileDB 1.5
+
+* Cell coordinate values of each dimension are always stored next to each other, regardless of whether they are filtered with a compression filter or not.


teo-tsirpanis · 2024-07-24T18:01:31Z

The first part of the work is done. For ease of review we can do the second part in a subsequent PR.

nickvigilante

Overall, looks good to me! Only one comment about including the issues directly in line with the release notes.

nickvigilante · 2024-07-29T12:57:56Z

format_spec/history.md

+
+Introduced in TileDB 2.19
+
+* The TileDB implementation has been updated to fix computing [tile metadata](./fragment.md#tile-mins-maxes) for nullable fixed-size strings on dense arrays.


General feedback: should we be including links to PRs in the Markdown itself? If you don't think that's appropriate, we can skip, but in the past open-source projects I've worked on, it was common practice to publicly include the PRs in all release documentation.

format_spec/history.md

KiterLuc · 2024-08-26T13:11:56Z

format_spec/history.md

+
+Introduced in TileDB 2.0
+
+* Dimensions are stored in separate [data files](./fragment.md#data-file).


I would try to mention zipped coordinates here.

KiterLuc · 2024-08-26T13:16:41Z

format_spec/history.md

+
+Introduced in TileDB 2.9
+
+* The [dictionary filter](./filters/dictionary_encoding.md) was added.


Consider mentioning:
Cell offsets in dimensions or attributes of ASCII string type are not written in the offset tiles, if the RLE filter exists in the filter pipeline. They are instead encoded as part of the data tile.

KiterLuc · 2024-08-26T13:17:15Z

format_spec/history.md

+
+Introduced in TileDB 2.10
+
+* Fragments can have timestamp files. The _Includes timestamps_ field was added to the [fragment metadata footer](./fragment.md#footer).


Suggested change

* Fragments can have timestamp files. The _Includes timestamps_ field was added to the [fragment metadata footer](./fragment.md#footer).

* Consolidated fragments can have timestamp files. The _Includes timestamps_ field was added to the [fragment metadata footer](./fragment.md#footer).

KiterLuc · 2024-08-26T14:47:48Z

format_spec/history.md

+
+Introduced in TileDB 2.11
+
+* Fragments can have delete metadata files. The _Includes delete metadata_ field was added to the [fragment metadata footer](./fragment.md#footer).


Suggested change

* Fragments can have delete metadata files. The _Includes delete metadata_ field was added to the [fragment metadata footer](./fragment.md#footer).

* Consolidated fragments can have delete metadata files. The _Includes delete metadata_ field was added to the [fragment metadata footer](./fragment.md#footer).

That wasn't apparent from a read of the source code but sure, it's valuable to mention. Reminder for myself to update the documentation of this flag in a subsequent PR.

KiterLuc · 2024-08-26T14:50:23Z

format_spec/history.md

+* The TileDB implementation has been updated to fix computing [tile metadata](./fragment.md#tile-mins-maxes) for nullable fixed-size strings on dense arrays.
+
+> [!NOTE]
+> This version does not contain any changes to the storage format, but was introduced as an indicator for implementations to not rely on tile metadata under certain circumstances on previous versions.


This only applies to fixed sized strings on dense arrays I think?

I wrote this in the bullet point above and did not want to repeat myself.

I'm mostly worried someone reads this "not rely on tile metadata under certain circumstances on previous versions." and makes the wrong assumption. I think we should repeat for 100% clarity. This is the last issue for me in this PR, once we resolve it's good to go IMO.

Co-authored-by: Theodore Tsirpanis <[email protected]>

[SC-54621](https://app.shortcut.com/tiledb-inc/story/54621/explicitly-mention-what-changed-in-each-storage-format-version-throughout-the-specification) Continuing #5205, this PR goes through each format version and mentions the changes to the main specification document. It also documents structures like `__coords.tdb` and legacy fragment metadata, and fixes any defects encountered in the meantime. --- TYPE: FORMAT DESC: The storage format specification was updated to document format changes of previous versions throughout the main document. --------- Co-authored-by: Nick Vigilante <[email protected]>

teo-tsirpanis added 2 commits July 19, 2024 23:25

Fix typos.

1b45271

Add a history of storage format versions 22 to 16.

1bcd345

teo-tsirpanis force-pushed the format-history branch from 2f6f84a to 1bcd345 Compare July 22, 2024 10:05

teo-tsirpanis added 3 commits July 22, 2024 20:50

Fix wrong comments.

84c7ebf

Add a history of storage format versions 15 to 9.

7c0d4ef

Update historical changes related to skip_offsets_filtering.

e929997

teo-tsirpanis commented Jul 22, 2024

View reviewed changes

teo-tsirpanis force-pushed the format-history branch from c2f4883 to 5465e41 Compare July 23, 2024 16:47

Add a history of storage format versions 9 to 5.

ddfd4d8

teo-tsirpanis force-pushed the format-history branch from 5465e41 to ddfd4d8 Compare July 23, 2024 16:59

teo-tsirpanis commented Jul 23, 2024

View reviewed changes

Add a history of storage format versions 4 to 1.

ebeb70c

teo-tsirpanis force-pushed the format-history branch from 1bf7276 to ebeb70c Compare July 24, 2024 16:30

teo-tsirpanis commented Jul 24, 2024

View reviewed changes

Remove the previous format version history table.

af42f14

teo-tsirpanis marked this pull request as ready for review July 24, 2024 18:01

nickvigilante approved these changes Jul 29, 2024

View reviewed changes

Update per offline feedback.

b74a9c3

teo-tsirpanis requested a review from KiterLuc August 20, 2024 18:40

KiterLuc reviewed Aug 26, 2024

View reviewed changes

teo-tsirpanis and others added 4 commits August 27, 2024 03:30

Merge branch 'dev' into format-history

f4d7137

Address PR feedback.

5004a41

Commit suggestion.

0cc6ba8

Co-authored-by: Theodore Tsirpanis <[email protected]>

Update note.

b92bbea

KiterLuc approved these changes Aug 28, 2024

View reviewed changes

KiterLuc merged commit 6921ebb into TileDB-Inc:dev Aug 28, 2024
62 checks passed

teo-tsirpanis deleted the format-history branch August 28, 2024 12:24

teo-tsirpanis mentioned this pull request Sep 30, 2024

Storage format specification improvements 2/N #5329

Merged


		Introduced in TileDB 2.25

		* The _Current domain_ field was added to [array schemas](./array_schema.md).


		Introduced in TileDB 2.19

		* The TileDB implementation has been updated to fix computing [tile metadata](./fragment.md#tile-mins-maxes) for fixed-size strings.


		Introduced in TileDB 2.17

		* Arrays can have [enumerations](./enumeration.md).


		Introduced in TileDB 2.12

		* Deleting data from arrays is supported. Arrays can have [delete commit files](./delete_commit_file.md).


		Introduced in TileDB 2.2.3

		* [Data files](./fragment.md#data-file) are named by the name of their attribute or dimension, after percent encoding certain characters. These characters are `!#$%&'()*+,/:;=?@[]`, as specified in [RFC 3986](https://tools.ietf.org/html/rfc3986), as well as `"<>\\|`, which are not allowed in Windows file names.


		Introduced in TileDB 2.0

		* Dimensions are stored in separate [data files](./fragment.md#data-file).


		Introduced in TileDB 2.1

		* The _Fill value_ field was added to [attributes](./array_schema.md#attribute).

		* The [MBR](./fragment.md#mbr) structure has been updated to support variable-sized dimensions.
		* The _Dimension number_ and _R-Tree datatype_ fields have been removed from [R-Trees](./fragment.md#r-tree).


		Introduced in TileDB 1.5

		* Cell coordinate values of each dimension are always stored next to each other, regardless of whether they are filtered with a compression filter or not.


		Introduced in TileDB 2.9

		* The [dictionary filter](./filters/dictionary_encoding.md) was added.


		Introduced in TileDB 2.10

		* Fragments can have timestamp files. The _Includes timestamps_ field was added to the [fragment metadata footer](./fragment.md#footer).

	* Fragments can have timestamp files. The _Includes timestamps_ field was added to the [fragment metadata footer](./fragment.md#footer).
	* Consolidated fragments can have timestamp files. The _Includes timestamps_ field was added to the [fragment metadata footer](./fragment.md#footer).


		Introduced in TileDB 2.11

		* Fragments can have delete metadata files. The _Includes delete metadata_ field was added to the [fragment metadata footer](./fragment.md#footer).

	* Fragments can have delete metadata files. The _Includes delete metadata_ field was added to the [fragment metadata footer](./fragment.md#footer).
	* Consolidated fragments can have delete metadata files. The _Includes delete metadata_ field was added to the [fragment metadata footer](./fragment.md#footer).

Add a history of storage format versions et. al. #5205

Add a history of storage format versions et. al. #5205

Conversation

teo-tsirpanis commented Jul 19, 2024 • edited by KiterLuc Loading

teo-tsirpanis left a comment

Choose a reason for hiding this comment

teo-tsirpanis Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teo-tsirpanis commented Jul 24, 2024 • edited Loading

nickvigilante left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teo-tsirpanis commented Jul 19, 2024 •

edited by KiterLuc

Loading

teo-tsirpanis Jul 19, 2024 •

edited

Loading

teo-tsirpanis commented Jul 24, 2024 •

edited

Loading