Determining jiff type

[fdncred]

As someone working on a repl with date commands, I may have a little different use case for dealing with datetime. After looking through the jiff source and docs, it seems that if you know what you have to begin with, it's pretty easy to understand and well documented how to deal with it. However, when you're in a repl with commands accepting most any input for date-type information, it's difficult to determine what you've been provided as input. Granted, I understand that "any input" isn't valid. Of course, there are rules we have to abide by for datetime.

I'm not sure this would be very helpful to others or not, but it would be very helpful to me to have some general parse routine that could figure out what was being passed in. Have I missed something like this in jiff?

Example: (just an easy to understand example. since some formats fit within others like, dates in datetimes, we'd have to try parsing in a particular order, if "this string" fits into "this type")

let input_string = "<some date/time/datetime/zoned/timestamp/span here>";
match parse(input_string)?.get_type() {
    Date => println!("Date"),
    DateTime => println!("DateTime"),
    Time => println!("Time"),
    Timestamp => println!("Timestamp"),
    Timezone => println!("Timezone"),
    Span => println!("Span"),
    Zoned => println!("Zoned"),
    _ => println!("Unknown")
}

I have investigated using strptime/strftime/BrokenDownTime to try and hand crank such a function. I've also considered regex. I'm not sure what a good approach would be. Any advice would be appreciated.

[BurntSushi]

Apologies for the ramble. My thoughts on this question are pretty unstructured unfortunately, so I kind of went all over the place.

---

Yeah, I don't think you're the first to bring this up. I think this is a tricky problem in general though and could have some pretty epic footguns if it isn't done right.

With that said, there is some support for this internally, but it isn't exposed. Specifically for the Temporal ISO 8601 format:

jiff/src/fmt/temporal/parser.rs

Lines 16 to 32 in b8ca1f7

	/// The datetime components parsed from a string.
	#[derive(Debug)]
	pub(super) struct ParsedDateTime<'i> {
	/// The original input that the datetime was parsed from.
	input: escape::Bytes<'i>,
	/// An optional civil date.
	date: ParsedDate<'i>,
	/// An optional civil time.
	time: Option<ParsedTime<'i>>,
	/// An optional UTC offset.
	offset: Option<ParsedOffset>,
	/// An optional RFC 9557 annotations parsed.
	///
	/// An empty `ParsedAnnotations` is valid and possible, so this bakes
	/// optionality into the type and doesn't need to be an `Option` itself.
	annotations: ParsedAnnotations<'i>,
	}

And this type is created by this routine:

jiff/src/fmt/temporal/parser.rs

Lines 191 to 240 in b8ca1f7

	// TemporalDateTimeString[Zoned] :::
	// AnnotatedDateTime[?Zoned]
	//
	// AnnotatedDateTime[Zoned] :::
	// [~Zoned] DateTime TimeZoneAnnotation[opt] Annotations[opt]
	// [+Zoned] DateTime TimeZoneAnnotation Annotations[opt]
	//
	// DateTime :::
	// Date
	// Date DateTimeSeparator TimeSpec DateTimeUTCOffset[opt]
	#[inline(always)]
	pub(super) fn parse_temporal_datetime<'i>(
	&self,
	input: &'i [u8],
	) -> Result<Parsed<'i, ParsedDateTime<'i>>, Error> {
	let mkslice = parse::slicer(input);
	let Parsed { value: date, input } = self.parse_date_spec(input)?;
	if input.is_empty() {
	let value = ParsedDateTime {
	input: escape::Bytes(mkslice(input)),
	date,
	time: None,
	offset: None,
	annotations: ParsedAnnotations::none(),
	};
	return Ok(Parsed { value, input });
	}
	let (time, offset, input) = if !matches!(input[0], b' ' | b'T' | b't')
	{
	(None, None, input)
	} else {
	let input = &input[1..];
	// If there's a separator, then we must parse a time and we are
	// *allowed* to parse an offset. But without a separator, we don't
	// support offsets. Just annotations (which are parsed below).
	let Parsed { value: time, input } = self.parse_time_spec(input)?;
	let Parsed { value: offset, input } = self.parse_offset(input)?;
	(Some(time), offset, input)
	};
	let Parsed { value: annotations, input } =
	self.parse_annotations(input)?;
	let value = ParsedDateTime {
	input: escape::Bytes(mkslice(input)),
	date,
	time,
	offset,
	annotations,
	};
	Ok(Parsed { value, input })
	}

Now, this only covers Zoned, Timestamp, civil::DateTime and civil::Date. It doesn't cover Span or civil::Time. For civil::Time, I believe there might be some ambiguity there that is difficult to resolve, and that's why it's treated separately from "datetime types that have a date."

Span though is more of an odd duck to throw in there with datetime. If you need to handle datetime and duration types in the same context, then that feels to me more like that responsibility should be on you as the repl author. Like, that doesn't strike me as a universally good decision. I could imagine some repls (or similar environments) specifically not wanting to ever have a context in which "either a datetime or a duration" is acceptable.

Beyond that, there are some other pretty gnarly questions here. One that comes to mind is: what format is looked for? Is this just the Temporal 8601 format? Or does it also include RFC 2822? What about custom strptime formats? And same thing with durations. (Right now, there is only the ISO 8601 duration format, but I'm planning to add a new "friendly" format as described in #111.) Is this just meant to be maximally flexible and therefore all formats should be accepted?

And if the goal is to be maximally flexible, then there is also the question of ambiguity. Like, do you want to accept things like YYYY/MM/DD? Or MM/DD/YY? Or DD/MM/YY? How does that fit into it? I'm not quite sure.

I basically feel like there is an interesting problem to solve here, but I'm not sure how best to go about it. I actually ran into a similarish problem when integrating Jiff into gitoxide:

https://github.com/BurntSushi/gitoxide.git/blob/25a3f1b0b07c01dd44df254f46caa6f78a4d3014/gix-date/src/parse.rs#L35-L60

But it's not quite the same as your problem. Instead, in gitoxide, it's more like, "there's a bunch of different formats we can accept here, so we just try them each." Is that something that Jiff should make easier? How closely related is it to your problem? I'm... not sure.

In terms of ideas, probably BrokenDownTime won't work for you. The main problem is that the strptime format doesn't provide a way to express "this component is optional." So you would need to do multiple strptime calls to check each format, but this wouldn't really end up being any different than just using the FromStr impls on each of the types.

With all that said, I do wonder if for your specific use case, there is perhaps a simple decision tree you can use here. If you limit yourself to what the FromStr/Display impls for each type support (and if you wanted to support more formats too that would probably be fine, but I'm just saying what I've considered here), then I think something like this would work:

• If the string matches ^[Pp][Tt]?[0-9] then is almost certainly has to be a Span. (The "almost certainly" comes from the fact that, technically, P0 is a valid IANA time zone identifier. (Although it doesn't actually exist.) And this perhaps reveals the folly of the task at hand: some strings can be in more than one category!) You could work around this ambiguity by asking whether the string is an actual time zone identifier in the database, which is... maybe appropriate. Maybe not. Changing the type of something based on a database lookup seems fishy.
• If the string contains [ anywhere, then it must be a Zoned.
• If the string starts with a T, then it must be a Time.
• If the string starts with ^[0-9]{4}, then it must be civil::Date, civil::DateTime or Timestamp. If the string does not contain a T, then it must be a civil::Date. If the string ends with (Z|[-+][0-9:]+)$, then it must be a Timestamp. Otherwise, it has to be a civil::DateTime.

I think that covers all of your types, but it doesn't quite span the full set of supported formats. For example, Jiff will happily parse 01:02:03 as a civil::Time, but it will reject 010203 because it's ambiguous. There are... perhaps other cases I'm missing. For example, Jiff supports -009999-01-01 as a date (and a prefix of a valid datetime or timestamp). But maybe the above cases are good enough for a repl.

Or alternatively, just try parsing in order of least flexible to most flexible:

• Zoned
• Timestamp
• DateTime
• Date
• Time
• Span
• TimeZone

This is the easiest approach. And I believe it will get all cases correct. (Not 100% certain of that.) But it may not necessarily be the fastest.

[fdncred]

Span though is more of an odd duck to throw in there with datetime.

Agreed. I could leave it out maybe or do something different for spans.

One that comes to mind is: what format is looked for?

For me, I want to do date math, so I probably want to promote everything to a Zoned, not matter how it comes in because it seems that it's the best format to use for date math.

Is this just the Temporal 8601 format? Or does it also include RFC 2822?

My tendency is to say both since they're pretty different.

What about custom strptime formats?

I was considering some way of providing the most common formats as strptime formats and just if-letting through that list until a good hit is found. I'm not convinced this would work well though.

Like, do you want to accept things like YYYY/MM/DD? Or MM/DD/YY? Or DD/MM/YY

Understood. With nushell, I could just make a rule that saw we accept it one way, but I could also provide a config point that allows the user to specify their preference. Either way, one way would be supported, either a default or a user contributed format. So, I worry less about this ambiguity for my use case.

gitoxide

I don't have access to the gitoxide link you provided. Not sure if it's private or the url is just bad? I'm a huge fan of gitoxide.

But it's not quite the same as your problem. Instead, in gitoxide, it's more like, "there's a bunch of different formats we can accept here, so we just try them each." Is that something that Jiff should make easier? How closely related is it to your problem? I'm... not sure.

Hard to tell without the link but from the description it sounds very similar since a user can provide anything their little heart desires to nushell. We want the "reasonable" things to be accepted. With our repl we also accept relative times. Not sure if that's covered in your gitoxide stuff. Here's yet another date math example.

❯ "Today 18:30" | into datetime
Thu, 22 Aug 2024 18:30:00 -0500 (in 7 hours)
❯ "This Friday 17:00" | into datetime
Fri, 23 Aug 2024 17:00:00 -0500 (in a day)

Or alternatively, just try parsing in order of least flexible to most flexible:

For me, it was between a bunch of regexes or this direction. I was just hoping it was already baked in and I just missed it. 🤣

Thanks for your input. I'll think on it a bit more. I want to spend more time with your temporal code above too to see if I can "steal" any of it to do my bidding.

[BurntSushi]

I don't have access to the gitoxide link you provided. Not sure if it's private or the url is just bad? I'm a huge fan of gitoxide.

Ah just a bad link. Here's the correct one: https://github.com/BurntSushi/gitoxide/blob/25a3f1b0b07c01dd44df254f46caa6f78a4d3014/gix-date/src/parse.rs#L35-L60

I think if it were me, I'd probably start with the if let chain probably. I think it's the easiest way to get the semantics you want, even if it's not exactly the most efficient.

[BurntSushi]

One thing that might be helpful here, and would probably increase the chances of getting this kind of support baked into Jiff (or perhaps a jiff-flexible crate or something I maintain) is to get your requirements written down as precisely and as completely as you can. I don't necessarily mean a full blown specification, but something that describes the complete picture of what you want Nushell to support. Including things like This Friday and what not. The second most important thing would be to find prior work to see what has been done before.

[fdncred]

I've got my if let chain kind of working but I'm struggling with determining if strings passed in have offsets and/or time zones. Am I right that there isn't really anything like that in jiff publicly available? I see rfc9557.rs, which seems to have most of what I'm looking for, but I can't seem to reach it.

I'd just rather not reinvent the wheel if I don't have to. I've got this below already, but your code seems much more thorough.

fn has_time_zone(input: &str) -> bool {
    input.contains('[') && input.contains(']')
}

fn has_offset(input: &str) -> bool {
    input.contains('T')
        && (input.split('T').last().is_some_and(|f| f.contains('+'))
            || input.split('T').last().is_some_and(|f| f.contains('-')))
}

[BurntSushi]

I forwarded this question to the Temporal folks: tc39/proposal-temporal#2930

[fdncred]

Can you say why that doesn't work for you?

Because the output of a parsed Timestamp doesn't include the offset or timezone. This is from another program that I was writing just to test parsing. All of these below parse as Z versus the provided offset or timezone, right?

[src/main.rs:79:5] parse_as_timestamp("2019-05-10T09:59:12-07:00") = Ok(
    2019-05-10T16:59:12Z,
)
[src/main.rs:80:5] parse_as_timestamp("2019-05-10T09:59:12-07:00[-07]") = Ok(
    2019-05-10T16:59:12Z,
)
[src/main.rs:81:5] parse_as_timestamp("2019-05-10T09:59:12-07:00[-07:00]") = Ok(
    2019-05-10T16:59:12Z,
)
[src/main.rs:82:5] parse_as_timestamp("2019-05-10T09:59:12-07:00[America/Los_Angeles]") = Ok(
    2019-05-10T16:59:12Z,
)

[BurntSushi]

Right, because a timestamp doesn't have an offset with it. Basically, Jiff sees this as just an instant in time.

I think you have to use strptime for now. I'll think about relaxing this restriction when parsing Zoned, but I do want to hear from the Temporal folks first. I would be inclined to allow it personally.

Can you also say more about why you want the offset here? And why you want to preserve it? Like one thing it seems like you could do here is parse it as a timestamp (which does not retain the offset) and then convert it to a Zoned in the system time zone. That might wind up with a different offset then what was given, but it will be the same precise instant in time. And if the user wants a different time zone, then they need to specify it.

I think the general problem here is that a fixed offset alone is kind of a footgun because it doesn't permit DST safe operations. Namely, because an offset in many places in the world shifts at least twice per year.

[fdncred]

Can you also say more about why you want the offset here?

I'm trying to get date math to work as accurately as I can. For my purpose "date math" means adding or subtracting dates or datetimes and getting an output duration.

I thought that would be done by making sure that all datetimes passed in have an offset and/or time zone, but it seems that some conversion still needs to take place first to get them all in the same time zone. If I'm converting to a single time zone, I think one still needs to know which time zone it's coming from to accurately convert the left side and the right side to the same time zone. If there's a better way to do this, please let me know.

Here's my command with debug statements and errors.

❯ '2019-05-10T09:59:12-07:00[-07:00]' | dt diff '2024-08-07T09:36:42.367322100-05:00[-05:00]'
    Input date: "2019-05-10T09:59:12-07:00[-07:00]"
Parameter date: "2024-08-07T09:36:42.367322100-05:00[-05:00]"
Zoned: 2019-05-10T09:59:12-07:00[-07:00]
Zoned: 2024-08-07T09:36:42.3673221-05:00[-05:00]
Error:   × Error calculating difference: computing the span between zoned datetimes, with year units, requires that the time zones are
  │ equivalent, but 2019-05-10T09:59:12-07:00[-07:00] and 2024-08-07T09:36:42.3673221-05:00[-05:00] have distinct time zones

That output is from this rust code.

let span = zoned_input_datetime
    .until(
        ZonedDifference::new(&zoned_parameter_datetime)
            .smallest(smallest_unit)
            .largest(biggest_unit)
            .mode(RoundMode::HalfExpand),
    )
    .map_err(|err| {
        LabeledError::new(format!("Error calculating difference: {}", err))
    })?;

Like one thing it seems like you could do here is parse it as a timestamp (which does not retain the offset) and then convert it to a Zoned in the system time zone.

I'm fine with that if it gives me an accurate duration but I figured it would have to know that time 1 is in pacific time and time 2 is in central time in order to get the calculation correct.

[BurntSushi]

Yeah you can't do ops on datetimes in different time zones because they may have different lengths for the day.

I don't have a good sense of what your top level UX here is so it is very difficult for me to offer more advice.

If you want absolute durations (hours or less) then you can do that easily by computing the duration between timestamps. But if you want non-uniform units, you absolutely must have a consistent time zone. Otherwise the operation simply isn't defined. That your desired UX has led you to try this probably means your UX is wrong somewhere. But I don't have enough context on what you're trying to do at a high enough level to say anything concrete.