Commuter Rail missing stops

[devinmatte]

When looking at the commuter rail data collected by stop, we seem to be missing several inbound and outbound stops

https://github.com/transitmatters/t-performance-dash/blob/d5913dd893bf29bb419868632ce60681f5a2615f/common/constants/cr_constants/cr-fairmount.json

It seems especially true for end of lines, but not exclusively

https://github.com/transitmatters/t-performance-dash/blob/d5913dd893bf29bb419868632ce60681f5a2615f/common/constants/cr_constants/cr-worcester.json

Is the issue on our end or the MBTA side?

(note that these lists are generated based off the list of stops gobble puts in s3)

[idreyn]

I spent a little time monitoring raw API data from some random midday trips into the CR terminals. Here's one inbound trip on the Needham line as it approaches South Station:

IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 89, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34763717651367, 'longitude': -71.07354736328125, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': None, 'updated_at': '2024-02-07T11:38:58-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 89, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34763717651367, 'longitude': -71.07354736328125, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': None, 'updated_at': '2024-02-07T11:39:14-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 89, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.3476448059082, 'longitude': -71.07345581054688, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 1.3, 'updated_at': '2024-02-07T11:39:45-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 89, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34769058227539, 'longitude': -71.0728759765625, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 4.0, 'updated_at': '2024-02-07T11:39:58-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 90, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34765625, 'longitude': -71.07091522216797, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 8.9, 'updated_at': '2024-02-07T11:40:16-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 90, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.347564697265625, 'longitude': -71.06925201416016, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 8.9, 'updated_at': '2024-02-07T11:40:30-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 99, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.347267150878906, 'longitude': -71.06706237792969, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 8.5, 'updated_at': '2024-02-07T11:40:47-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 106, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34698486328125, 'longitude': -71.06558227539062, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 8.0, 'updated_at': '2024-02-07T11:40:59-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 106, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.346588134765625, 'longitude': -71.06351470947266, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 8.0, 'updated_at': '2024-02-07T11:41:18-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 106, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34606170654297, 'longitude': -71.06039428710938, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 4.9, 'updated_at': '2024-02-07T11:41:49-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 52, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.346282958984375, 'longitude': -71.05838012695312, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 4.0, 'updated_at': '2024-02-07T11:42:20-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 40, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34632110595703, 'longitude': -71.05819702148438, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 4.0, 'updated_at': '2024-02-07T11:42:26-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 26, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.34743881225586, 'longitude': -71.05712127685547, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 4.0, 'updated_at': '2024-02-07T11:42:51-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 20, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.348995208740234, 'longitude': -71.05633544921875, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 4.5, 'updated_at': '2024-02-07T11:43:24-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}
IN_TRANSIT_TO DEP CR-Needham South Station AMTKSchCh651178-612 {'attributes': {'bearing': 20, 'carriages': [], 'current_status': 'IN_TRANSIT_TO', 'current_stop_sequence': 110, 'direction_id': 1, 'label': '1834', 'latitude': 42.35052490234375, 'longitude': -71.05581665039062, 'occupancy_status': None, 'revenue': 'REVENUE', 'speed': 4.0, 'updated_at': '2024-02-07T11:43:53-05:00'}, 'id': '1834', 'links': {'self': '/vehicles/1834'}, 'relationships': {'route': {'data': {'id': 'CR-Needham', 'type': 'route'}}, 'stop': {'data': {'id': 'NEC-2287', 'type': 'stop'}}, 'trip': {'data': {'id': 'AMTKSchCh651178-612', 'type': 'trip'}}}, 'type': 'vehicle'}

Then it disappears without ever giving us a STOPPED_AT South Station. So it looks like it's a problem with the API. Now I'm wondering if we could infer an event from raw GPS data. Here are the last few points we get, plotted using a useful web tool:

This trip, at least, gave us GPS data well into the yard and basically up to the platform. Based on this single observation, I think we could fill in arrival events for terminals by looking for updates from the API that are:

1. The last for a given trip
2. Clearly within a small distance from North or South station

The current event loop is structured around events that are there, not missing events, so there's a possibility we'll need to add another kind of interrupt that proactively purges the trips state when a trip hasn't been heard from in a while. As part of that process, we could examine the final handful of events it gave us and see whether we think we can fill in a terminal arrival event.

I'm less confident that we can accurately fill in events at random stations along the line, but a similar approach might work there, too.