array_format_history.md

title
Array format version history

Array Format Version History

Version 22

Introduced in TileDB 2.25

• The Current domain field was added to array schemas.

Version 21

Introduced in TileDB 2.19

• The TileDB implementation has been updated to fix computing tile metadata for nullable fixed-size strings on dense arrays.

Note

This version does not contain any changes to the storage format, but was introduced as an indicator for implementations to not rely on tile metadata for nullable fixed-size strings on dense arrays on previous versions.

Version 20

Introduced in TileDB 2.17

• Arrays can have enumerations.
• The bit-width reduction and positive delta encoding filters are supported on data of date or time types.
• The filter pipeline options for the double-delta filter contain the Reinterpret datatype field.

Version 19

Introduced in TileDB 2.16

• Vacuum files contain relative paths to the location of the array.
• The filter pipeline options for the delta filter contain the Reinterpret datatype field.

Version 18

Introduced in TileDB 2.15

• Arrays can have dimension labels.

Version 17

Introduced in TileDB 2.14

• The Order field was added to attributes.
• Cell offsets in dimensions or attributes of UTF-8 string type are not written in the offset tiles, if the RLE or dictionary encoding filter exists in the filter pipeline. They are instead encoded as part of the data tile.

Version 16

Introduced in TileDB 2.12

• Arrays can have delete commit files.
• Arrays can have update commit files.
  • The TileDB implementation currently supports writing update commit files as an experimental feature, but they are not yet considered when performing reads.
• Fragment metadata contain tile processed conditions.

Version 15

Introduced in TileDB 2.11

• Consolidated fragments can have delete metadata files. The Includes delete metadata field was added to the fragment metadata footer.

Version 14

Introduced in TileDB 2.10

• Consolidated fragments can have timestamp files. The Includes timestamps field was added to the fragment metadata footer.

Version 13

Introduced in TileDB 2.9

• Cell offsets in dimensions or attributes of ASCII string type are not written in the offset tiles, if the dictionary encoding filter exists in the filter pipeline. They are instead encoded as part of the data tile.

Version 12

Introduced in TileDB 2.8

• The array file hierarchy was updated to store fragments, commits and consolidated fragment metadata in separate subdirectories.
• The extension of commit files was changed to .wrt.
• Cell offsets in dimensions or attributes of ASCII string type are not written in the offset tiles, if the RLE filter exists in the filter pipeline. They are instead encoded as part of the data tile.

Version 11

Introduced in TileDB 2.7

• Fragment metadata contain metadata (min/max value, sum, null count) for data in the whole fragment and each tile.
• The TileDB implementation has been updated to never split cells when storing them in chunks.

Version 10

Introduced in TileDB 2.4

• Arrays support schema evolution.
  • Array schemas are stored in a __schema subdirectory, and have a timestamped name.
  • The Array schema name field was added to the fragment metadata footer.
• The Footer length field of the fragment metadata footer is always written.

Version 9

Introduced in TileDB 2.3

• Data files are named by the index of their attribute or dimension.
• The URI fields of Consolidated fragment metadata files contain relative paths to the location of fragments in the array.

Version 8

Introduced in TileDB 2.2.3

• Data files are named by the name of their attribute or dimension, after percent encoding certain characters. These characters are !#$%&'()*+,/:;=?@[], as specified in RFC 3986, as well as "<>\|, which are not allowed in Windows file names.

Version 7

Introduced in TileDB 2.2

• Attributes can be nullable.
  • The Nullable and Fill value validity fields were added to attributes.
  • The Validity filters field was added to array schemas.
  • Fragment metadata contain validity tile offsets.

Version 6

Introduced in TileDB 2.1

• The Fill value field was added to attributes.

Version 5

Introduced in TileDB 2.0

• Dimensions are stored in separate data files.
• Sparse arrays can have string dimensions and dimensions with different datatypes.
  • The Dimension datatype, Cell val num and Filters fields were added to dimensions.
  • The Domain size field was added to dimensions. The domain of a dimension can have a variable size.
  • The Domain datatype field was removed from domains.
  • The MBR structure has been updated to support variable-sized dimensions.
  • The Dimension number and R-Tree datatype fields have been removed from R-Trees.
• The Allows dups field was added to array schemas.
• Committed fragments are indicated by the presence of an .ok file in the array's directory, with the same timestamped name as the fragment.

Version 4

Introduced in TileDB 1.7

• Support for the key-value store object type was removed. Key-value stores have been superseded by sparse arrays.

Version 3

Introduced in TileDB 1.6

• The structure of fragment metadata files was overhauled.
  • The footer and R-Tree structures were added.
  • The Bounding coords field was removed.
  • The MBRs field was removed. MBRs are now stored in the R-Tree.
  • Tile offsets and sizes are wrapped in their own generic tiles. This allows loading them lazily and in parallel.

Version 2

Introduced in TileDB 1.5

• Cell coordinate values of each dimension are always stored next to each other, regardless of whether they are filtered with a compression filter or not.

Version 1

Introduced in TileDB 1.4

• Initial version of the TileDB storage format.