Generating docspec with pydoctor

[tristanlatr]

Hi @NiklasRosenstein ,

I've been working on generating docspec classes from pydoctor's internals.
I think it could replace the current docspec-python parser. This would avoid you to rewrite a new parser as described in #3.

I've written the serializer with the visitor pattern, you can see the code here: https://github.com/twisted/pydoctor/blob/json-docspec/pydoctor/jsonbuilder.py#L213

It's currently using a modified version of docspec, but it's working great.

The main changes that I've done are the following:

• Added the ApiObject.kind attribute which is a list of all possibles python objects types. See https://github.com/twisted/pydoctor/blob/json-docspec/pydoctor/docspec/__init__.py#L90. This could be removed if the Function.modifiers attribute is moved to ApiObject.modifiers such that we can store this information there.
• Added zope interface information support. This does not have to live in the docspec repository, but if you like it I can submit it as well. See https://github.com/twisted/pydoctor/blob/json-docspec/pydoctor/zopeinterface.py#L330
• Dump indirections (imports and aliases) as Data such that the links can be correctly linked to objects documentation when using the local name.
• Replaced the decorations: List[Decoration] by decorators: List[str]. I found the Decoration object not really optimal, it's harder to dump from AST because the name is separated from the arguments. I'm currently using astor.to_source to dump the ast.Call object to string. This is the only thing i'm not sure how to do, any help appreciated.

I could send a few patches to introduces the required changes, then we can introduce a new module docspec_pydoctor of something, that will offer very similar interface as docspec_python, but using the pydoctor parser, such that both implementation can live of a period of time.

Tell me what you think

Thanks

[NiklasRosenstein]

Hey @tristanlatr , haven't had a lot of time to think about this yet, but I'll respond tomorrow.

[NiklasRosenstein]

Hi @tristanlatr ,

Thanks for your thoughts! I'll try to cover each point and give my perspective.

I think it could replace the current docspec-python parser. This would avoid you to rewrite a new parser as described in #3.

The main reason that docspec is using lib2to3 now, and why I was planning to use libCST in the future, is that both options preserve single-line comments in the syntax tree, which are discarded when parsing Python code with the ast module (which seems to be what pydoctor is using now).

Docspec is intended as only a slightly higher-level API compared to an abstract syntax tree. It leaves the interpretation of that tree to the client (i.e. any information derived from content rather than structure). Pydoctor seems to be doing a lot of that interpretation, and with that perspective in mind I would see how pydoctor could rely on docspec to build it's interpreted documentable tree, but not the other way around as you are suggesting.

It's currently using a modified version of docspec, but it's working great.

Can you elaborate what you are currently getting out of the docspec <-> pydoctor integration? Is it only the JSON de/serialization or where you aiming to feed the results into Pydoc-Markdown?

Added the ApiObject.kind attribute which is a list of all possibles python objects types. See https://github.com/twisted/pydoctor/blob/json-docspec/pydoctor/docspec/__init__.py#L90. This could be removed if the Function.modifiers attribute is moved to ApiObject.modifiers such that we can store this information there.

This "kind" information would be derivable from the other information in the docspec tree (e.g. module is a package if it has modules under it/if it's filename is __init__.py, function is a property if @property is in the decorations, etc.). As I pointed out above, I want to avoid this interpretation in docspec and leave it up to the client.

Added zope interface information support. This does not have to live in the docspec repository, but if you like it I can submit it as well. See https://github.com/twisted/pydoctor/blob/json-docspec/pydoctor/zopeinterface.py#L330

Same as above.

Dump indirections (imports and aliases) as Data such that the links can be correctly linked to objects documentation when using the local name.

I like the idea of recording external references because right now the information is simply not present and prevents clients to properly resolve the full qualified name of referenced names (for example in Class.bases or in Decoration.name). think I would maybe model it slightly differently, for example by introducing a new Indirection class.

Replaced the decorations: List[Decoration] by decorators: List[str].

The name decoration is chosen to distinguish between an instance of a decorator being applied (ie. using the @ decoration syntax) and the decorator itself (like property, classmethod or a user defined decorator). I would like to keep it in a form where you can introspect some information about the decoration (e.g. the name and whether it is being called with arguments).

I found the Decoration object not really optimal, it's harder to dump from AST because the name is separated from the arguments. I'm currently using astor.to_source to dump the ast.Call object to string. This is the only thing i'm not sure how to do, any help appreciated.

As I mentioned above I don't see pydoctor as a potential drop in replacement for two main reasons, but if you still want to build a docspec_pydoctor module (which you are welcome to do!), could simply split the result from astor.to_source() by the parenthesis? 🤔 However, that approach, and the current Decorator class are not capable of properly handling/representing all possible decorations starting with Python 3.9 and the relaxed decorator grammar introduced via PEP 614.

Sorry for the overall negative vibe. Maybe there is some clarification to be added in the README/docs to describe exactly what docspec is intended to be. I hope I was able to show you my point of view on your points, the purpose I see for docspec in this context and why I don't quite think it matches with pydoctor as you suggest.

Kind regards,
Niklas

[NiklasRosenstein]

For your awareness, I migrated this to the Discussions section. :)

[tristanlatr]

Hi,

The main reason that docspec is using lib2to3 now, and why I was planning to use libCST in the future, is that both options preserve single-line comments

Single lines comments are the comments starting by #:, on the same line of the object or the line directly before, right ? If so, I think it would be possible to add support for that kind of comments to pydoctor parser, by using the tokenize module and lookup the COMMENT tokens at the required lines.

Docspec is intended as only a slightly higher-level API compared to an abstract syntax tree. It leaves the interpretation of that tree to the client (i.e. any information derived from content rather than structure). Pydoctor seems to be doing a lot of that interpretation, and with that perspective in mind I would see how pydoctor could rely on docspec to build it's interpreted documentable tree, but not the other way around as you are suggesting.

I'm OK with dumping less interpreted data and doing more of the interpretation in pydoctor json builder, it will probably create some code duplication with our ast builder, tough.

In terms of interpretation, are you talking about the modules hierarchy ? Pydoctor generates nested modules inside modules.

I understand the ApiObject.kind modification should be undone, and replaced by more processing.

I'm not sure regarding the Decoration object. As you pointed out, it will not suffice from python3.9. We could add Decoration.raw_expression: str attribute and make Decoration.name and Decoration.args properties that would compute the result with the ast module and return None if the decoration is using a more complex expression than ast.Name/Attribute or ast.Call. This way, we keep the compatibility when decorators are not using the new syntax.

where you aiming to feed the results into Pydoc-Markdown?

That is definitively the goal. This is also why I want to make this work.

I like the idea of the creation of a new class Indirection(ApiObject) to represent imports and aliases, I'll try to sumbit a PR with that.

Tell me what you think,

Thanks

[NiklasRosenstein]

Single lines comments are the comments starting by #:, on the same line of the object or the line directly before, right ? If so, I think it would be possible to add support for that kind of comments to pydoctor parser, by using the tokenize module and lookup the COMMENT tokens at the required lines.

That is correct. If you feel like this would work, feel free to give it a try. Still I believe using libCST would probably make the loader code more maintainable and reliable.

In terms of interpretation, are you talking about the modules hierarchy ? Pydoctor generates nested modules inside modules.

Docspec supports nested modules as of recently, I need to update the docspec-python loader though.

I'm not sure regarding the Decoration object. As you pointed out, it will not suffice from python3.9. We could add Decoration.raw_expression: str attribute and make Decoration.name and Decoration.args properties that would compute the result with the ast module and return None if the decoration is using a more complex expression than ast.Name/Attribute or ast.Call. This way, we keep the compatibility when decorators are not using the new syntax.

Something like this I, yes. I would keep name and args as normal members (instead of properties) so they are serialized as well if the loader can determine them, and if the loader can't, it can populate the raw_expression instead.

Best

[NiklasRosenstein]

@tristanlatr The next version of docspec will have a Decoration.name_expression field. See https://niklasrosenstein.github.io/docspec/specification/#struct-decoration

[tristanlatr]

Thanks.
I don't understand why it's a list of str and not just a string, though.

[NiklasRosenstein]

Thanks for pointing that out, it's a mistake in the spec. It's a str in the docspec module. 😄

[tristanlatr]

Ok, so that should not represent the whole expression, just the name ?

[NiklasRosenstein]

A decorator has the form @name and @name(args...), and the name piece can be an expression. I wanted to differentiate between the case where name is a pure name or when name is an expression in which case name_expression should be used. Actually thinking about it again, maybe we don't even need the name_expression field and just make name be a piece of code if it needs to be. 🤔

[NiklasRosenstein]

#44

[tristanlatr]

Yes that's also what I'de thought.