Split date time data into smaller data keys?

[zbraniecki]

As I'm implementing Dates in DataProvider and testing them using DateTimeFormat, I have some questions about how should we structure that.

Generally, the data in question looks like this: https://github.com/unicode-cldr/cldr-dates-modern/tree/master/main/en

It has (per locale):

• display names for Months, Weekdays, DayPeriods, Quarters, Eras
• patterns for time, date, and date_time
• list of best patterns for skeletons
• interval patterns
• time zone names
• relative display names

For now, we need:

• display names for months, weekdays, day periods
• patterns for time, date and date_time
• list of best patterns for skeletons

Display names come in different:

• contexts ("format" and "stand_alone")
• widths ("abbreviated", "narrow", "short", "wide")

but they also can be for different calendar systems (I see at least "generic" and "gregorian").

As far as I understand DataProvider we have some flexibility in what do we request and what we get in response.

We could, for example, put "months/format/narrow" as a variant in DataEntry and gregory in DataKey and get just a list of month names in "format" and "narrow" for "gregory" calendar.

Or, we can just ask for "gregory" and set no variant in DataEntry and get all display names for all contexts and all widths.

@sffc - what are your thoughts on that? How should a request/response look like?

[zbraniecki]

Based on @sffc comments in that PR, it also seems like he'd suggest we split the dates data into multiple keys:

• datesym ( month display names, weekday display names)
• timesym ( day period display names)
• patterns (for style, and for skeleton)

I assume that for things like time zone names, interval patterns and relative display names he's suggestion we also create separate keys.

[sffc]

First, please leave skeletons (including available_formats) aside. Those need their own discussion.

Responses to the several sub-questions:

DataProvider Flexibility

As far as I understand DataProvider we have some flexibility in what do we request and what we get in response.

Right. The intended goal of the data provider keys is not to mimic CLDR. It's to return data that is easy and efficient to consume at runtime. Mapping from CLDR format to ICU4X format happens in CldrJsonDataProvider. I want to make sure we are aligned on that goal.

Number of Data Keys

Regarding the breakdown of the data keys. I feel strongly that there should be a minimum of three distinct data keys for DateTimeFormat, which Zibi listed in #257 (comment). Here are my reasons:

1. A component can request the data it needs and nothing more.
2. The same key can be shared by multiple code paths; for example, datesym can be used by both the datetimestyle path and the skeleton path.
3. Smaller data hunks mean that more locales will be able to point to shared hunks and make data smaller on disk.

Format Widths for Display Names

I've been thinking about the format widths (long/short/narrow) for display names and whether they belong in the data requests (key/entry) or not. I'm thinking that no, they don't belong in the data request; long/short/narrow should be together in the same data key and entry, and they should be a leaf of the struct. My reasoning:

1. The width trio (long/short/narrow) tends to be strongly correlated; if one changes, so do the others. Therefore, considering the trio as a single key does not decrease the ability to create shared data hunks.
2. It is not uncommon to fall back between widths; for example, if narrow is not present, then we fall back to short.
3. In order for a split data key to work as a method to reduce code size, we would need to track the requested width through much of the call stack in a way suitable for code slicing. I think this is too fine-grained to track.

Format Widths for Patterns

The situation is a bit different for pattern widths (dateStyle/timeStyle). None of the three aforementioned conditions apply here: the width patterns are not strongly correlated; they do not fall back; and we should be able to slice the data very early in the call stack.

Therefore, I think it makes sense to put format widths into either the data key or the data entry.

I would like to make this judgement after we have the code written and we can sit down and look at the concrete implications to the data bundles.

Calendar Systems

I'm arriving at the conclusion that calendar systems should be in the data key. Reasons:

1. The data for each calendar system is unique. Not all systems are represented by 12 month names. When we need a new struct (data schema), we need a new data key.
2. The code needed to run calendar systems needs to be written for each individual calendar, so the data keys will correspond to those modules of code.

We may at some point want to add an all-in-one calendar data key, but this is not relevant right now. We should cross that bridge later when we add calendar math to ICU4X.

[zbraniecki]

Thank you! This is so helpful!

[sffc]

Because of reasons I discussed in the new doc datetime-input.md, I now believe that we should not have different data keys for different calendar systems. We should pool the essential symbols for calendar systems into the same data key. We could consider filtering the data in the data entry or the offline build tool.

However, I do still feel that we should have different data keys for patterns, date symbols, and time symbols. We should probably go further and split the date symbols down into eras, months, day periods, and time zone names.

[sffc]

Concretely, I envision DateTimeFormat using the following separate, orthogonal keys, which covers all formatting except for time zone (which I want to leave to a separate discussion):

Display Names

1. datetime/era@1 collects era display names covering all desired calendar systems
2. datetime/cycyear@ covers cyclic year names
3. datetime/quarter@1 covers quarter names
4. datetime/month@1 covers month names
5. datetime/weekday@1 covers weekday names
6. datetime/dayperiod@1 covers the a and b day periods (am, noon, pm, and midnight)
7. datetime/flexperiod@1 covers the B day periods (in the morning, in the afternoon, …)

Format Patterns

1. datetime/patterns@1 covers long/medium/short date patterns, time patterns, and glue patterns
2. datetime/skeletons@1 covers availableFormats: the mapping from skeletons to patterns

Why more keys instead of fewer keys? I listed reasons in #257 (comment), but to reiterate:

1. A component can request the data it needs and nothing more.
2. The same key can be shared by multiple code paths; for example, month names can be used by both the datetimestyle path and the skeleton path.
3. Smaller data hunks mean that more locales will be able to point to shared hunks and make data smaller on disk.

[zbraniecki]

I'm convinced. This looks like a great design. One additional benefit of it is that version changes will be less common and more isolated in a more chunked model.

[sffc]

There are still some open questions in my mind about how exactly to provision data across calendar systems, but that question is being tracked in #355.

[sffc]

Shane to implement this along with #355.

[sffc]

Blocked on #409 like #355

[sffc]

This should be one of the last things to do in 1.0 after DTF has stabilized.

We should do this before 1.0 because it impacts data file stability.

[gregtatum]

I think this has two parts, one part is the data keys for the ECMA-402 compatible components bag, and the other is for the ideal components bag. Blocking for 1.0 will be ensuring we have the best split for the ECMA-402 compatible components bag.

[sffc]

Make sure to look at the data representation of the glue pattern and make changes if necessary for future-proofing. See #1131

[sffc]

We have split symbols from patterns and date from time; this is sufficient for the first release. I would still like to explore even more-granular splitting, but there's no time in 1.0 and we should coordinate this with the Ideal Components Bag work.

[Manishearth]

The neo date time format stuff does this

[sffc]

Posting this here because it seems like as good of a place as any:

I'l looking into the minimal set of patterns required for year formatting. A year can take three forms, which we could make dynamically selectable at runtime based on the value of the year:

1. Partial precision, such as: '00
2. Full precision, such as: 2000
3. With era, such as: 2000 AD

I looked into whether the patterns used for case 1 and 2 differ other than the length of the year field. I found that, at least according to the CLDR algorithm and data, the patterns for one are mostly identical to the patterns for the other with the year width swapped out. There are a few exceptions:

Legend: pattern on the left is the possibly-reduced-precision year with a full-precision year substituted. Pattern on the right is the pattern resulting from a full-precision year used during skeleton selection.

DataLocale{ug}/"chinese": "r-M-d" != "r-MM-dd"
DataLocale{qu-BO}/"gregorian": "d-M-y" != "dd-MM-y"
DataLocale{ky}/"gregorian": "y-d-M" != "y-dd-MM"
DataLocale{sat}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{bs-Cyrl}/"gregorian": "d.M.y." != "dd.MM.y."
DataLocale{en-SG}/"gregorian": "d/M/y" != "dd/MM/y"
DataLocale{ne}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{hr-BA}/"gregorian": "d. M. y." != "dd. MM. y."
DataLocale{zu}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{en-MV}/"gregorian": "d/M/y" != "dd/MM/y"
DataLocale{qu}/"gregorian": "d-M-y" != "dd-MM-y"
DataLocale{ha-GH}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{en-AU}/"gregorian": "d/M/y" != "dd/MM/y"
DataLocale{qu-EC}/"gregorian": "d-M-y" != "dd-MM-y"
DataLocale{sah}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{ha}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{lkt}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{ne-IN}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{sat}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{ha-NE}/"gregorian": "y-M-d" != "y-MM-dd"
DataLocale{ug}/"dangi": "r-M-d" != "r-MM-dd"

Spot-checking, in most of these cases, the pattern is inherited from the root locale, which makes me question the quality of the data.

If I were to store the patterns separately, I could use another flag in the data struct. We currently have a single byte reserved for flags in the packed data structure, and 4 bits are used, so this would mean using one more bit. However, we could keep the bit unset if the locale results in equivalent data as observed above.

https://github.com/unicode-org/icu4x/blob/main/components/datetime/src/provider/neo.rs#L560

This could be done as a follow-up so long as the bit remains available.

[sffc]

I believe this issue is fully resolved.