Include pandoc version in json abstract syntax tree

[tomduck]

Given the evolving pandoc API, it would be very helpful for filter writers if the pandoc version number could be included in the json abstract syntax tree. For example, it could be stored in the unMeta field along with the other metadata. Filter writers could then retrieve the pandoc version and adjust the behavior of their filters accordingly.

Note that a simple call like "pandoc -v" from the filter won't work if users have multiple versions of pandoc installed (as I do, for testing purposes).

As an example of how not being able to detect the pandoc version causes a problem, please see this comment on pull request #2637 which has just been submitted. The changing behavior of the LaTeX writer will likely cause a conflict with my pandoc-fignos filter. If I could detect the pandoc version, then I would be able to make adjustments allowing the filter to operate in a backward-compatible manner.

[jgm]

I think that something like this is needed.

I'm not sure adding it to the metadata is the right thing to
do. (Imagine you're filtering Markdown, for example; you'll
get a new metadata field with the pandoc version.)

Maybe it should be a separate field in the JSON?

Another issue is that pandoc and pandoc-types have separate
versions, in general. I think it's the pandoc-types version
that is important here.

[tomduck]

OK, great. As long as the info is somewhere in the JSON, then filter authors will be able to get at it.

For my use case, it is the pandoc version that is important. Perhaps both the pandoc and pandoc-types versions could be included.

[jgm]

For one possible approach see

jgm/pandoc-types@a8b75de

though I've had second thoughts and reverted it for now.

Mainly the second thoughts are because this only gives you the pandoc-types version, not also pandoc.

But I don't see how we can give pandoc version, since toJSONFilter is entirely defined in pandoc-types.

[emwap]

That field would provide a version of the Pandoc data structure (and siblings).
But as you say, it will not provide the version of any library/program that has participated in the creation of the data structure.
Unless the data structure has a field to capture these versions there is no way to capture the information.

A stringly typed variant would be to have libraries/programs insert their version into the Meta field.

A slightly more controlled variant is to add an abstract Versions field to the Pandoc constructor and let the ToJSON instance render it as a JSON object under the tool-versions tag.

"tool-versions":
{
  "pandoc": [1, 17],
  "pandoc-types": [1, 17, 1],
  "pandoc-citeproc": [0,10]
}

Of course this requires a change to the Pandoc type which affects all readers, writers and filters.

With a few helper functions the Meta variant could maintain the same versions structure.

[tomduck]

On Jan 12, 2016, at 4:52 AM, Anders Persson [email protected] wrote:

A slightly more controlled variant is to add an abstract Versions field to the Pandoc constructor and let the ToJSON instance render it as a JSON object under the tool-versions tag.

"tool-versions":
{
"pandoc": [1, 17],
"pandoc-types": [1, 17, 1],
"pandoc-citeproc": [0,10]
}

Of course this requires a change to the Pandoc type which affects all readers, writers and filters.

Is there a way to minimize the impact? e.g., Add “tools-versions" as a new field to the same JSON dict as “unMeta”? Perhaps append a new item to the JSON structure?

In either case I know that my filters would continue to work without modification. I suspect that this would be the case for (many? all?) other python filters. I don’t know enough about the expectations of Haskell to assess the rest.

[jgm]

Passing all these versions as strings every time we
serialize to JSON seems a bit overkill to me.

What if we added an option to pandoc to spit out all
the dependent versions? Then your filter could simply
run pandoc to get this information:

pandoc --library-versions

+++ Thomas J. Duck [Jan 12 16 06:37 ]:

On Jan 12, 2016, at 4:52 AM, Anders Persson
[email protected] wrote:

A slightly more controlled variant is to add an abstract Versions
field to the Pandoc constructor and let the ToJSON instance render it
as a JSON object under the tool-versions tag.

"tool-versions":
{
"pandoc": [1, 17],
"pandoc-types": [1, 17, 1],
"pandoc-citeproc": [0,10]
}

Of course this requires a change to the Pandoc type which affects all
readers, writers and filters.
Is there a way to minimize the impact? e.g., Add “tools-versions" as a
new field to the same JSON dict as “unMeta”? Perhaps append a new
item to the JSON structure?
In either case I know that my filters would continue to work without
modification. I suspect that this would be the case for (many? all?)
other python filters. I don’t know enough about the expectations of
Haskell to assess the rest.

—
Reply to this email directly or [1]view it on GitHub.

References

1. Include pandoc version in json abstract syntax tree #2640 (comment)

[tomduck]

On Jan 12, 2016, at 2:40 PM, John MacFarlane [email protected] wrote:

Passing all these versions as strings every time we
serialize to JSON seems a bit overkill to me.

True, although compared to the amount of data serialized to JSON it seems rather small.

What if we added an option to pandoc to spit out all
the dependent versions? Then your filter could simply
run pandoc to get this information:

I can’t count on accessing the correct version of pandoc using a shell call. It is possible that people will have multiple pandoc versions installed on their machine (I do). It would also mean the filter would not work with any renamed pandoc forks. I think version information needs to somehow be passed along by pandoc to its filters.

[tomduck]

Any further thoughts on this request? I'm concerned that if pull request #2637 gets accepted that my filters won't be able to reliably detect the changed LaTeX/pdf writer. Having pandoc pass along version information in its JSON would allow for more robust filters. Cheers. :o)

[jgm]

I still don't know how to deal with this.

Currently the JSON is generated by a module in pandoc-types,
which wouldn't have access to all the versions for pandoc
and its dependent libraries.

Pandoc could add something to the metadata before calling
the filter. But the problem then is that this metadata will
be passed through to the output format, which is usually
undesirable. (You don't want your markdown output, for
example, to have all this in a YAML header.)

Perhaps implementing a convention for writers to ignore
certain metadata fields would do the trick.

+++ Thomas J. Duck [Jan 24 16 14:34 ]:

Any further thoughts on this request? I'm concerned that if [1]pull
request #2637 gets accepted that my filters won't be able to reliably
detect the changed LaTeX/pdf writer. Having pandoc pass along version
information in its JSON would allow for more robust filters. Cheers.
:o)

—
Reply to this email directly or [2]view it on GitHub.

References

1. LaTeX writer: figure label #2637
2. Include pandoc version in json abstract syntax tree #2640 (comment)

[tomduck]

I see the new "pandoc-api-version" in the JSON output and have adapted the pandoc-fignos filter (and friends) accordingly. Thank you!

It would still be useful to have the pandoc version in the JSON in order to accommodate changes in pandoc's writers (which I assume won't be reflected in "pandoc-api-version"). So I will leave this issue open if that is OK.

Cheers.

[jgm]

I'd rather set an environment variable, PANDOC_VERSION. That would work too, right?

[tomduck]

@jgm: Yes, that should work.

[tomduck]

This works great. Thanks, @jgm.