Profiles/specs versioning

[roll]

Libraries like tableschema and datapackage are shipped with specs profiles. But for now there is no API to know which profile version is used.

Blocked by frictionlessdata/datapackage#444

---

From @simleo
Is there a way to find out which version of the specs a given datapackage-py release implements? For instance, if I pip install datapackage==0.8.9 and use it to validate a data package, which version of the specs am I validating against?

Thanks!

[roll]

I think this problem is really important

[rufuspollock]

@roll can you elaborate more on why it is important both right now and potentially in the future. (I'm generally in agreement - i just want more info on specific job stories).

---

And this is the update i've just posted in frictionlessdata/datapackage#444

There's renewed interest in this and we have now hit v1 and making improvements beyond that. In terms of structure what about following classic package dependencies e.g. npm package.json and having:

implements: {
  "data-package": "1.4.3",
  "@datopian/geodata": "1.0.0"
}

I assume this would imply at each level e.g. data package, data resource and table schema - would we want some kind of cascading (e.g. the same spec applys to all resources in a package)

[roll]

@rufuspollock
For me the main question is compatibility.

For example, we would like to make our Python libs 100% compatible/implementing the specs. TBH at the moment, I don't really understand what does it mean. Whether there is a frozen v1 of the specs to be compatible with and where all the current spec changes go v1.1/v2 branch of the specs etc

In my opinion, the versions of the specs should be very discrete (no [email protected] 😃) and well-emphasized in the docs. For example:

• only v1 / v2 as in jsonschema - one release per a few years
• something like ecmascript v2019 / v2020 - one release per year

cc @akariv @lwinfree

[rufuspollock]

@roll afaik (and i've been trying to track) we have merged no change to the spec since v1 that would be breaking. I agree you need some way to know compatibility.

In my opinion, the versions of the specs should be very discrete (no [email protected] 😃) and well-emphasized in the docs. For example:

OK, that's a separate point - can you open a specific issue in the specs repo about how you think versioning of the specs should work along these lines and we can discuss.

[roll]

@rufuspollock
Aside from no breaking changes are there new features since v1?

What we've learned lately with @lwinfree that if some software doesn't support something that users can see in the specs, the users consider it as a bug.

PS.
frictionlessdata/datapackage#671

[rufuspollock]

@roll i don't believe so but we could double check - i think pretty everything new were in patterns.

Any examples from you and @lwinfree about users finding these "bugs" - that would be useful to record.

[roll]

@rufuspollock
No-no, I'm not saying we've run into problems because of the latest changes. It was earlier when some tableschema drivers hadn't supported some v1 features.

So I'm saying in general that it would be great to understand the strategy behind the evolution of the spec. For example, the approach that for now, everything goes to the patterns I think is not stated anywhere. So I just didn't know it.

Another thing, for example, if there will be v1.1. release of the specs (I guess it was a plan though I'm not sure it's a good idea vs longer release cycle and v2 - frictionlessdata/datapackage#671) - how to handle it in the libs. It's clear that for some time there will be a gap between the libs and the specs. Some libs will be v1-only for years I guess (like Ruby). For example, if we had profiles versioning we would provide information about compatibility level in the libs much easier etc. Actually the way we store JSON Schemas, for now, doesn't allow us to have multiple versions of the specs' profiles. So it's again back to versioning.

[rufuspollock]

FIXED / MOVED. Agreed to do something about this and follow up is in specs frictionlessdata/datapackage#444 (Adding a version to profile attribute)