optimize agg queries

[OriHoch]

this should reduce the query times without changing the results

[AdlerShay]

🙏

[OriHoch]

@AdlerShay @ShayAdler FYI - fixed a bug (merged directly to main) - 0dbf39a

[ShayAdler]

@OriHoch why was only a day of tolerance problematic?

[OriHoch]

the gtfs date is the date that the data was collected on, it's not necesarily related to the date of the actual ride

I noticed it on the site where it showed less data with the optimization then without..

[ShayAdler]

Alright I see it in the code now, but I was sure that this is the day we downloaded it from the MOT FTP, the therefore we still have reference to the original date.
Can you help me here with understanding the gtfs loading process? (when are we downloading from the ftp? when is the original date is getting lost in the way?).
Also, is the gtfs_ride.start_time also something we can't trust on (regarding the date, not the hour)?

[OriHoch]

yes, gtfs date is the date we downloaded the data from MOT FTP, so it's not directly related to the ride dates/times.. but it is confusing yes..
for example in the data we downloaded on Thursday we might get rides which happened on Sunday.. or something like that

gtfs_ride.start_time is more reliable for both date and time because it should be the date from the actual gtfs data

but I'm not sure, I might be mistaken, it's very confusing..

[ShayAdler]

We actually rely on that when setting the siri_ride.scheduled_time_gtfs_ride_id to optimize the search there as well, maybe we shouldn't?
Maybe best would be adding an ETL that will update the route date based on the most common gtfs_rides.start_time? I definitely feel like I understand nothing again 😅

Basically AFAIK, we only use that date to narrow down searches, as the date that really matters is the ride's.

I think it will help me to look at the raw data, do we store it like we store siri?

[OriHoch]

yes, the raw data is availab,e see here - https://github.com/hasadna/open-bus-pipelines/blob/main/STRIDE_ETL_PROCESSES.md#gtfs-etl-download-upload

[ShayAdler]

@OriHoch can you expand on how did you see this bug?
I wan unable to find such cases in the data (this returns 0 results) -

-- get all rides in which the diff between the gtfs_route.date and the gtfs_ride.start_time is more than 1 day. Sometimes we do have 1 day diff, mostly when the ride hour is late.
select EXtract(epoch from date_trunc('day', gtfs_ride.start_time) - gr.date) , * from gtfs_ride join gtfs_route gr on gtfs_ride.gtfs_route_id = gr.id
where date_trunc('day', gtfs_ride.start_time) != gr.date
  and EXtract(epoch from date_trunc('day', gtfs_ride.start_time) - gr.date) not in (-86400, 86400)
and date_trunc('day', gtfs_ride.start_time) between '2023-10-01' and '2023-10-30'

[OriHoch]

This is the original complaint - https://hasadna.slack.com/archives/C1M8YT5AN/p1701675411278589
this change fixed it