[FEATURE] Delayed Connections get removed when departure has passed, when RT Data use "delay" instead of "delay_realtime"

[dafunkydan]

Describe the Bug
For Realtime Updates, instead of "departure_realtime" my Transportation Service uses "delay_realtime".
Delayed Trains get removed from Attributes when the regular scheduled Time has passed.

Expected Behaviour:
Delayed Connection beeing kept in the Attributes as upcoming.
Personally i would prefer having departure_realtime = departure + delay_realtime, as this might make it easier to calculate "Departs in xx Min" in Homeassistant. But any other solution (e.g. internal calculation like include if timestamp_now > timestamp_departure + delay_realtime) would probably work as well.

Steps/data to reproduce the behavior, e.g.
Download the static GTFS: https://rnv-dds-prod-gtfs.azurewebsites.net/latest/gtfs.zip
Register for the API Key for RT-Data: #51 (comment) and use them
Set up a Zone with some Stops
Enable RT for that Stop with *.rt as Source
Reload, wait for the next Departures with a delay
See that it gets kicked out after departure time, not taking the delay into account

Release used
gtfs2: 0.4.4.7
HA: HAOS
Core: 2024.4.3

Additional
No logs for that

[vingerha]

I see the point but no immediate solution.
The local stops sensor has to first query the static data to find out what departures are upcoming/expected and with those trip_id it then searches for realtime. The static data is limited indeed between now() and now() + range, because the sensor is designed to look ahead.
What a user (you) wants is to continue to see vehicles as long as they have not departed but as I cannot limit the static data based on a single trip/vehicle (it does not have the delay) I would need to extend the limits to between: now() - X and now() + range,
There is no simple (very complex) way to query the data including rt-delay, to e.g. compare with previous state....effort vs. value does not make a business case to me (sorry)
What can be done rather easily is collecting historical departures for anything on that stop up to X in the past and then filter if dep+delay < now()...but.... how long to look back...5 mins, 1 hour, a day?

[dafunkydan]

has to first query the static data to find out what departures are upcoming/expected and with those trip_id it then searches for realtime

Ah i see! Didnt think of that. Makes sense. Too bad.

There is no simple (very complex) way to query the data including rt-delay

Fair enough, i see the huge processing effort in the Background necessary. Absolutely understandable.

What can be done rather easily is collecting historical departures for anything on that stop up to X in the past and then filter if dep+delay < now()

Well - that sounds great! Not sure if understood that correctly - but from my perspective the Result would effectly do that Job, wouldnt it? 😍 It would be something like:

1. Get all Departures between now() and now() + range
   2a. Get all Departures between now() - historyrange and now()
   2b. Get the RT Data for Connections found under 2a and discard all found connections dep+delay > now()
   2c. Add those to the Array

If i got that right, and that might be doable with a reasonable effort - i'd be superhappy!
In the End i think of the (Realtime) Departure Monitors we do have at all those stops. "Line X arrives in 3 Minutes". Having that in Homeassistant, and do a quick check before leaving would be awesome 👍

...but.... how long to look back...5 mins, 1 hour, a day?

Spontanious: Leave it up to the User. Just like when setting up the static data ("Checking for departures in future from 'now' (in minutes: between 15 and 120"), in the Dialog for the Realtime Integration, there could be a similar option ("Checking against scheduled departures in the past up to (in minutes: between 15 and 120)".
It is clear that the longer the Timespan, the more computing needs to be done.

[vingerha]

Spontanious: Leave it up to the User. Just like when setting up the static data ("Checking for departures in future from 'now' (in minutes: between 15 and 120"),

That is the offset you asked in the other request
In this case, how long to look 'back' in order to still see departures that have not yet departed. i.e. I have seen trains that had more than 1 hour delay so would you expect to seem them too? The more historical departures are needed to be scanned and compoared with delay ... the slower it gets. For a few stops and departures bo issue but I have a zone in Basel with more than 15 stops, each with departures every 5 mins or less, the total set of departures reaches more than 150 and that is only 'future' departures

[vingerha]

I could also have the user select this but I am also worried about the complexity of adding yet another attribute

[dafunkydan]

Spontanious: Leave it up to the User. Just like when setting up the static data ("Checking for departures in future from 'now' (in minutes: between 15 and 120"),

That is the offset you asked in the other request

Things getting mixed up here, while the two Feature Request sure are related.
The other Feature Request is on setting an offset for Departures (e g.: show Departures in the Range of now() + offset and now() + range).
This here is the other Way around:

in the Dialog for the Realtime Integration, there could be a similar option ("Checking against scheduled departures in the past up to (in minutes: between 15 and 120)".

I could also have the user select this but I am also worried about the complexity of adding yet another attribute

Totally understand your Concerns! I can't estimate the complexity. See below

i.e. I have seen trains that had more than 1 hour delay so would you expect to seem them too? The more historical departures are needed to be scanned and compoared with delay ... the slower it gets.

This would just be a Way for me and probably other affected users, to still catch Delayed connections. I can't estimate how huge the Impact will be - codewise, and also processingwise.
If there would have been such an Option, i would have just given it Shot - "Check for Delayed trains up to: 15 minutes" and see how things go.

In the End, if i got it right, the Basic Workflow at the Moment is:

1. Get all Departures between now() and now() + range
2. Get the RT Data for Connections found under 1 and add it to the Connections
3. Output

This, and the other Feature Request combined would mean a Workflow like:

1. Get all Departures between now() - history and now() + range (where history initally would be set to 0 = current behaviour)
2. Get the RT Data for Connections found under 1 and add it to the Connections
   2b. Set departure_realtime = departure + delay
   2c. Remove all Departures where now() + offset > departure_realtime (This is the other Feature Request)
3. Output

If that is too complex, too much effort, too less outcome.... - it is what it is, it is you who surely can assess that!
From my End-Users Perspective the Benefits would be having a Realtime-Departure Monitor including delayed Connections, but only those i will be able to catch if now leaving the House ;-)

[vingerha]

I all depends on the area where one lives, there may be places where 15min delays is normal... will try wtih default =15 for now then

[vingerha]

And on my above mentioned complexity, the more attributes, the more permutations people need to think of and more complex to reproduce in case of 'errors'

[vingerha]

something alike this then? time is 13:16 but still showing old departures with delay

[dafunkydan]

You are unbelievable!!! 🪄 Yes, this exactly what i thought of!

Whats the approach now for this?
I mean, probably i dont understand or can give advices :-D But just out of Curiosity... ;-)

That is Huge. Really, not kidding. I think to see possible Connections, not filtered by they schedule, but actual Time, is a big Plus imo!! 👍 👍 👍

[vingerha]

resolved in 0.4.4.8

Big challenge with the data is the variety of formats of date/datetime/delay in the static data, realtime and server
Solution extracts additional departures till 15min back in time and only filters them out if there is no delay or dep.time+delay < now()

[dafunkydan]

This is great! For sure this is not so easy, especially with the

variety of formats of date/datetime/delay in the static data, realtime and server

But there seems to be an oversight in 0.4.4.8 in the filters for no delay, or departure time+delay < now():

[vingerha]

That is not clear enough for me...how often does your sensor update?
EDIT....at least not each minute as the last update is 17:10 so this is a logical consequence from where I sit

[dafunkydan]

For every Stop i set the Interval to 1 Minute.
Only missed this one, and didnt check again.
I am so, so sorry 😢

[vingerha]

if the frequency of transport is high...chane the update accordingly to a shorter timespan...yes it will load the system a bit more but at least the data becomes morie usable
Alternative ...if you donot want to run each minute all day long, you can add a button and update the entity when you need it (regular HA sevice call)

[vingerha]

As an example, I tie one of those sensors to me as a person, when I am e.g. in Basel I use two buttons,

1. update location (is often not too fast on &@#&% iPhone)
2. update stops

With that, my map and stop list update to where I am at that time...quite handy :)