Persistent trip_updates when running getStopTimeUpdates

[nukeador]

Hi there,

Is there a way to keep trip_updates data persistent on the SQLite or make them persist/cached for a fixed period of time?

I've observed that my GTFS RT feed provides data in a very intermittent way, and I'm not getting the latest stop times updated when running getStopTimeUpdates().

Is there a recommended way to keep this info stored in the SQLite for at least some time (unless it changes by a new update to the trip)?

Thanks!

[brendannee]

Can you share with me the config.json file you are using so I can try out your feed?

[nukeador]

Note that there are some additional variables I'm using as part of the integration with the rest of my app.

const { environment } = require('../utils');

const { GTFS_DIR } = environment;

const config = {
  sqlitePath: `${GTFS_DIR}/database.sqlite`,
  agencies: [
    {
      agency_key: 'AUVASA',
      path: `${GTFS_DIR}/static`,
      realtimeUrls: [
        'http://212.170.201.204:50080/GTFSRTapi/api/tripupdate',
        'http://212.170.201.204:50080/GTFSRTapi/api/vehicleposition',
        'http://212.170.201.204:50080/GTFSRTapi/api/alert',
      ],
    },
  ],
  staticUrl: 'http://212.170.201.204:50080/GTFSRTapi/api/GTFSFile',
  ignoreDuplicates: true,
};

[brendannee]

Thanks for sharing this.

When you call getStopTimeUpdates() - it doesn't actually fetch the latest stoptimeupdates from GTFS-realtime, it just fetches them from your database.

You need to periodically run the GTFS-Realtime import script or the updateGtfsRealtime function in order to pull data into your database https://github.com/BlinkTagInc/node-gtfs?tab=readme-ov-file#gtfsrealtime-update-script

I realize that this wasn't very clear from the documentation, so just added some notes about this. Let me know if this solves your issue, or if there is something else.

[nukeador]

Yes, I do getStopTimeUpdates() each 20 seconds and I had to setup a caching mechanism because if in the next 20 seconds there was no update for a trip, the row on the SQLite was gone.

Maybe this a behavior from an old node-gtfs version and not the case anymore? (so I don't need caching)

[brendannee]

Let me know if you still see this behavior - there is a mechanism to keep old data around while the new data is fetched and inserted into the DB using the is_updated field https://github.com/BlinkTagInc/node-gtfs/blame/master/lib/import.js#L728 - but I haven't used this feature extensively.

[nukeador]

Would this work even if the agency is sending empty RT updates for a trip? I suspect that's what might be happening.

[brendannee]

No - I think if that is the case they will all be removed from the database.

I haven't heard of this type of issue with GTFS-Realtime before - do you know if this is a common problem, or do you think it just specific to the agency you are working with?

[nukeador]

I suspect this is specific to the agency, but not sure how to deal with it 😕

[brendannee]

You could make your own function to do what updateGtfsRealtime does here https://github.com/BlinkTagInc/node-gtfs/blob/updategtfsrealtime/lib/import.js#L709 - except don't delete old values in the cleanStaleRealtimeData step.

[nukeador]

I've been debugging the RT data and I think the issue is that this agency in each update is pushing only a few stop_sequences:

For example, I'll get an update for a given trip_id with info for stop sequences 14 19 24

Then, once the route advances, they push another update with stop sequences 15 20 25, but only with them. What I see on the SQLite is that only 15 20 and 25 are present and 14 19 and 24 are lost.

Does this has to do with cleanStaleRealtimedata? Is there a way I can manage/config when a RT update should be considered "stale"?

I've been comparing the RT results with a local instance of OpenTripPlanner, and this one is able to catch and keep all RT updates for all stop sequences, using the same RT url to fetch them. So I suspect something is wrong on my node-gtfs project side.

[nukeador]

Additional clarification: The problem is also that there are gaps in the trip updates, providing some stop_sequences and not others, this is a problem at the agency feed level, not sure how gtfs-node can help with that 😕

[nukeador]

Digging more on why OpenTripPlanner is able to provide RT updates even for gaps in stops sequences I read this:

• Interpolation: OTP can estimate the times for stops that are not directly reported in the real-time feed. It does this by interpolating between known times of adjacent stops. For example, if the real-time feed provides arrival times for stops 1 and 3, OTP can estimate the arrival time for stop 2, which lies between them, based on the scheduled times and the real-time deviations at stops 1 and 3.
• Delay Propagation: OTP uses delay information from the real-time feed to adjust the expected arrival times at subsequent stops. If a vehicle is reported to be delayed at a certain stop, OTP will propagate this delay forward to all stops down the line. This means that if a bus is reported to be 5 minutes late at stop 1, OTP will adjust the expected arrival times at stops 2, 3, 4, etc., accordingly.

Maybe those two things should come as a feature request on a new issue then?

[brendannee]

Thanks for looking into this issue of missing times - the #162 feature request should handle solving for interpolation and delay propagation.

As for controlling when data is considered "stale" or persisting data after subsequent GTFS-Realtime data is fetched and stored into the database, it seems like this may not be a needed use case.