Update timestamp desc to unix timestamp milliseconds #138

asadowns · 2018-10-13T17:37:02Z

Per #104 and #133 all timestamp references are unix timestamp in milliseconds.

Provide link with different language implementations.

Also, update all provider sample data to be in unix timestamps in milliseconds.

No changes made to agency.

Addresses: #104 and #133

Per openmobilityfoundation#104 and openmobilityfoundation#133 all timestamp references are unix timestamp in milliseconds. Provide link with different language implementations. Also, update all provider sample data to be in unix timestamps in milliseconds. No changes made to agency. Addresses: openmobilityfoundation#104 and openmobilityfoundation#133

thekaveman

Great clarifications, thank you @asadowns!

I'm pinning this to 0.2.1, a downcast from float -> int doesn't seem like a breaking change to me. @hunterowens @ian-r-rose ?

hunterowens · 2018-10-14T20:41:19Z

in postgres

CREATE TABLE IF NOT EXISTS test (

float_field float, 
int_field int
);

INSERT INTO test (float_field, int_field) VALUES (2.0, 2.0);

runs successfully and if you pass 2.0,2.3 it will automatically round floats -> ints.

So at least in the python/postgres DB world this in non-breaking. I don't think that will be universal but right now that is most provider ingest stacks so I am okay making the change.

We should also update the JSONSchema, right @ian-r-rose?

ian-r-rose · 2018-10-14T21:55:58Z

Yes, I think it should be updated.

I think that, strictly speaking, changing the type from a float to an integer would be considered a breaking change. Python is pretty lax in this regard, and Javascript doesn't even have the concept of integers, but if anybody is developing tooling in JVM-based or C-allied languages there could be some updates required.

As long as we are discussing changing data types, is there a reason the trip_duration, trip_distance, and accuracy are defined as integers rather than floats?

hunterowens · 2018-10-14T22:10:53Z

@ian-r-rose can you move that discussion to #128. I'm tracking all JSON schema errors over there.

thekaveman · 2018-10-15T20:46:18Z

So we're saying this is breaking then, and should be in 0.3.0?

…ta-standard into dev * 'dev' of https://github.com/CityOfLosAngeles/mobility-data-standard: Fixing newlines again (openmobilityfoundation#135) typo fix. Update README to show associated python libraries (openmobilityfoundation#132) Add Lyft Add Razor to providers.csv update update_vehicle_status to use standard post clarification on how we extend GBFS Elevating Status of Realtime Feed

hunterowens · 2018-10-18T23:19:48Z

@thekaveman

I've been wrestling with this issue for a few days, and learning a whole bunch about how weird python and the concept of time can get.

That being said, this seems like an important change, and I'm unaware of anybody building MDS ingest tooling in a language that doesn't silently downcast floats->ints, so I think we should tag this in 0.2.1.

I'll update the milestone and the JSON schema to get this PR ready to merge.

…://github.com/asadowns/mobility-data-specification into asadowns-adowns/133-104-provider-timestamp-unix-millis * 'adowns/133-104-provider-timestamp-unix-millis' of https://github.com/asadowns/mobility-data-specification: Update timestamp desc to unix timestamp milliseconds

hunterowens · 2018-10-19T02:37:45Z

@asadowns can you give me permission to push to this branch? there are some changes to the JSON schema needed to support this PR. Follow these steps

asadowns · 2018-10-19T15:03:08Z

@hunterowens . Looks like it is already clicked but you should be good to go now.

hunterowens · 2018-10-19T17:45:04Z

found the issue, was trying to push the @adownsbird to the @asadowns branch. fixed now.

thekaveman · 2018-10-19T17:51:46Z

generate_schema/common.json

@@ -97,9 +97,10 @@
    },
    "timestamp": {
      "$id": "#/definitions/timestamp",
-      "title": "Floating-point seconds since Unix epoch",
+      "title": "Integer seconds since Unix epoch",


seconds -> milliseconds

thekaveman · 2018-10-19T17:54:08Z

generate_schema/common.json

      "type": "number",
-      "minimum": 0.0
+      "multiple_of": 1.0, 


I think the keyword is multipleOf - but I like this, nice approach!

thekaveman · 2018-10-19T17:57:14Z

provider/status_changes.json

            "Feature"
          ]
        },
        "properties": {
-          "type": "object",


type can be an array, see #142 for how I implemented this to allow null for the optional fields like parking_verification_url.

thekaveman · 2018-10-19T17:57:45Z

provider/status_changes.json

      "type": "number",
-      "minimum": 0.0
+      "multiple_of": 1.0,


multipleOf

thekaveman · 2018-10-19T17:59:59Z

generate_schema/common.json

      "type": "number",
-      "minimum": 0.0
+      "multiple_of": 1.0, 
+      "minimum": 1483228800 


Is this a good idea to enforce via the schema?

What about a process that generates fake data, shouldn't it be allowed to generate data across whatever time series it wants?

If MDS gets back-ported to TNCs and other related types of shared mobility, this could hamstring the ability to get historic data.

that fair, I'm gonna say let's not enforce via schema but some sort of "truth-yness" validator.

thekaveman · 2018-10-19T18:00:29Z

provider/status_changes.json

      "type": "number",
-      "minimum": 0.0
+      "multiple_of": 1.0,
+      "minimum": 1483228800


See comment above.

thekaveman · 2018-10-19T18:00:43Z

provider/trips.json

            "Feature"
          ]
        },
        "properties": {
-          "type": "object",


Use array for type, see comment above

provider/trips.json

Per openmobilityfoundation#104 and openmobilityfoundation#133 all timestamp references are unix timestamp in milliseconds. Provide link with different language implementations. Also, update all provider sample data to be in unix timestamps in milliseconds. No changes made to agency. Addresses: openmobilityfoundation#104 and openmobilityfoundation#133

hunterowens · 2018-10-19T21:12:41Z

@thekaveman changes made.

thekaveman · 2018-10-19T22:34:48Z

Closing in favor of #146

hunterowens and others added 2 commits October 6, 2018 13:02

update update_vehicle_status to use standard post

5e8392e

thekaveman added this to the 0.2.1 milestone Oct 14, 2018

thekaveman approved these changes Oct 14, 2018

View reviewed changes

thekaveman added the Schema Implications for JSON Schema or OpenAPI label Oct 15, 2018

thekaveman modified the milestones: 0.2.1, 0.3.0 Oct 16, 2018

hunterowens modified the milestones: 0.3.0, 0.2.1 Oct 18, 2018

hunterowens added 2 commits October 18, 2018 19:24

update timestamp to be integer using.

14eebc2

thekaveman suggested changes Oct 19, 2018

View reviewed changes

adownsbird and others added 4 commits October 19, 2018 11:18

update timestamp to be integer using.

cf952fc

remove minimum time and typo fix

c503199

rebase

5d738b4

thekaveman mentioned this pull request Oct 19, 2018

Timestamp milliseconds #146

Merged

thekaveman closed this Oct 19, 2018

thekaveman removed this from the 0.2.1 milestone Nov 25, 2018

thekaveman added this to the 0.3.0 milestone Nov 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update timestamp desc to unix timestamp milliseconds #138

Update timestamp desc to unix timestamp milliseconds #138

asadowns commented Oct 13, 2018

thekaveman left a comment

hunterowens commented Oct 14, 2018

ian-r-rose commented Oct 14, 2018

hunterowens commented Oct 14, 2018

thekaveman commented Oct 15, 2018

hunterowens commented Oct 18, 2018

hunterowens commented Oct 19, 2018

asadowns commented Oct 19, 2018

hunterowens commented Oct 19, 2018

thekaveman Oct 19, 2018

thekaveman Oct 19, 2018

thekaveman Oct 19, 2018

thekaveman Oct 19, 2018

thekaveman Oct 19, 2018

hunterowens Oct 19, 2018

thekaveman Oct 19, 2018

thekaveman Oct 19, 2018

hunterowens commented Oct 19, 2018

thekaveman commented Oct 19, 2018

Update timestamp desc to unix timestamp milliseconds #138

Update timestamp desc to unix timestamp milliseconds #138

Conversation

asadowns commented Oct 13, 2018

thekaveman left a comment

Choose a reason for hiding this comment

hunterowens commented Oct 14, 2018

ian-r-rose commented Oct 14, 2018

hunterowens commented Oct 14, 2018

thekaveman commented Oct 15, 2018

hunterowens commented Oct 18, 2018

hunterowens commented Oct 19, 2018

asadowns commented Oct 19, 2018

hunterowens commented Oct 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hunterowens commented Oct 19, 2018

thekaveman commented Oct 19, 2018