Q: How to document CallExpression/CallSignature with TypeDoc?

[tristanzander]

Search terms

CallSignature, CallExpression, CucumberJS, Function Calls, Custom Reflections

Question

TL;DR

1. Is there a generic way to convert a ts.JSDoc into a list of TypeDoc tokens that I can put into a reflection?
2. Is there a way to register a custom reflection explicitly dedicated to function CallExpressions that is placed in it's own folder?

Related to a thread on Discord.

Details

I'm using the CucumberJS framework for some integration tests, and we distribute some framework-specific code to other people in the form of step definitions. These step definitions are function calls that are registered at runtime to provide BDD assertions in the form of Gherkins. The following is an example of a step definition (taken straight from the docs with an added TSDoc):

import { Given } from '@cucumber/cucumber';

/**
 * Prepare the actor with a specific number of cucumbers having already being eaten.
 *
 * @param cucumberCount The number of cucumbers to eat
 * 
 * @example
 * ```
 * ...
 * ```
 */
Given('I have {int} cucumbers in my belly', function (cucumberCount: number) {
  assert.equal(this.responseStatus, cucumberCount)
});

I want to write a plugin that is able to convert those function calls into readable documentation. So far, it's been going quite nicely in the conversion stage of things, but I'm getting stuck after having extracted the function details and JSDoc from the function. Is there a generic method that can convert a ts.JSDoc to a typedoc.Comment or some form of tokens that I can place directly into a reflection? I was looking at lexBlockComment as that solution, but it's not publicly available through the exports. It might be a little bit too low-level for my needs anyway, but I'm not seeing much else available for my inputs.

Additionally, I haven't quite figured out how to register a reflection that goes into it's own folder. I notice that each part of a reflection can get its own folder (ex. "classes", "enums", "functions", "types", "modules") based on the ReflectionKind that's used in the DeclarationReflection. There doesn't seem to be a way to create a custom ReflectionKind and have it create a unique folder. This is a very niche use-case, so I might be the first person to try something like this.

Any help I can get with these questions would be greatly appreciated! 🙏

Current Demo Plugin

Here's what I have implemented so far. It's fairly basic, but you can see the part in readSourceFile() where it extracts the functions that I'm attempting to convert to docs.

Demo Plugin Code

import { glob } from 'glob';
import {
	Application,
	Context,
	Converter,
	DeclarationReflection,
	ParameterType,
	ReflectionKind,
	Comment,
	TypeScript as ts,
} from 'typedoc';

const CUCUMBER_FUNCTION_NAMES: CucumberFunctionNames[] = [
	'Given',
	'When',
	'Then',
	'Before',
	'After',
	'AfterAll',
	'BeforeAll',
	'BeforeEach',
	'AfterEach',
];

type CucumberFunctionNames =
	| 'Given'
	| 'When'
	| 'Then'
	| 'Before'
	| 'After'
	| 'BeforeAll'
	| 'AfterAll'
	| 'BeforeEach'
	| 'AfterEach';

function getUniquePrograms(programs: readonly ts.Program[]): readonly ts.Program[] {
	const uniqueMap: Record<string, ts.Program[]> = programs.reduce((acc, p) => {
		const root = p.getCurrentDirectory();
		acc[root] = acc[root] ?? [];
		acc[root].push(p);
		return acc;
	}, {});
	return Object.values(uniqueMap).map((pArray) => pArray[0]);
}

/**
 * Reads a TypeScript file and extracts any documentation on Step Definitions.
 * @param sourceFile The TypeScript source file
 */
function readSourceFile(sourceFile: ts.SourceFile) {
	// These are what store the JSDocs
	const toplevelExpressions = sourceFile.statements.filter(
		(f) => f.kind == ts.SyntaxKind.ExpressionStatement
	) as ts.ExpressionStatement[];

	const sources: {
		sourceFile: ts.SourceFile;
		jsDoc: ts.JSDoc;
		functionName: CucumberFunctionNames;
	}[] = [];

	for (const expression of toplevelExpressions) {
		const cucumberFunctionName = isCucumberFunctionCall(expression);
		if (cucumberFunctionName === false) {
			// Not a cucumber function
			continue;
		}

		if (expression['jsDoc'] === undefined) {
			continue;
		}

		const jsDoc = expression['jsDoc'] as [ts.JSDoc];
		// const docComment = sourceFile.getFullText().slice(jsDoc[0].pos, jsDoc[0].end);

		sources.push({
			functionName: cucumberFunctionName,
			sourceFile,
			jsDoc: jsDoc[0],
		});
	}
}

/**
 * Determines if the expression is a cucumber support code function call.
 * @param expression
 * @returns the name of the cucumber function that was called, or false in a situation where the expression is not a function call or a cucumber function.
 */
function isCucumberFunctionCall(expression: ts.ExpressionStatement): CucumberFunctionNames | false {
	const innerExpression = expression.expression as ts.CallExpression;
	if (innerExpression.kind !== ts.SyntaxKind.CallExpression) {
		return false;
	}

	if (innerExpression.expression.kind !== ts.SyntaxKind.Identifier) {
		return false;
	}

	if (
		!CUCUMBER_FUNCTION_NAMES.includes(
			(innerExpression.expression as ts.Identifier).escapedText.toString() as any
		)
	) {
		return false;
	}

	return (
		innerExpression.expression as ts.Identifier
	).escapedText.toString() as CucumberFunctionNames;
}

export function load(app: Application) {
	app.options.addDeclaration({
		name: 'stepDefinitions',
		help:
			'Extract TSDocs from cucumber step definitions listed at the specified paths. ' +
			'Should be a list of files relative to the project root, represented as a relative path or glob.',
		type: ParameterType.GlobArray,
		defaultValue: [],
	});

	app.converter.on(Converter.EVENT_RESOLVE_BEGIN, async (ctx: Context) => {
		const paths = ctx.converter.application.options.getValue('stepDefinitions') as string[];

		if (paths.length == 0) {
			return;
		}

		const resolvedPaths = (await Promise.all(paths.map((p) => glob(p)))).flat();

		for (const program of getUniquePrograms(ctx.programs)) {
			for (const path of resolvedPaths) {
				const sourceFile = program.getSourceFile(path);

				if (!sourceFile) {
					continue;
				}

				readSourceFile(sourceFile);
			}
		}

		const reflection = new DeclarationReflection('customThingy', ReflectionKind.Module);
		reflection.comment = new Comment(
			[
				{
					kind: 'text',
					text: 'this is a custom comment',
				},
			],
			[]
		);

		reflection.children = [
                         // attempting to see if I can register a custom module with a "function" declaration, where the function
                         // is just the information I extract from the step definitions
			new DeclarationReflection('fakeCallSignature', ReflectionKind.Function, reflection),
		];

		ctx.project.children?.push(reflection);
	});
}

[tristanzander]

I think it might be necessary to monkey-patch the typedoc.DefaultTheme.getUrls() and typedoc-plugin-markdown.MarkdownTheme.getUrls() in order to get the structure we want. I wanted to avoid having dedicated themes to make this work if possible.

[Gerrit0]

There doesn't seem to be a way to create a custom ReflectionKind and have it create a unique folder

There is not, kinds are intended to map to TS types of symbols, and there's logic in TypeDoc that depends on having covered all of the kinds to be sure things work correctly. Surprisingly, to me, you're at least the fourth person to have tried something like this.

I want to eventually make it possible for people to create custom folder structures in output, but have yet to come up with something I'm happy with... not concretely tied to any release target yet. You're going to be stuck with patching or a custom theme for now.

It's worth considering if adding a @group tag to show the organization on a website might be enough.

Is there a generic method that can convert a ts.JSDoc to a typedoc.Comment or some form of tokens that I can place directly into a reflection?

For this specific use case, no. There are a couple comments on Context which let you get a comment for a symbol, but you don't have a symbol for the Given call, just a Node. TypeDoc doesn't need this, but exposing a method for getting a Node's comment seems reasonable, I'll expose a Context.getNodeComment method in the next release. (Edit: now released)

Using that, I made a couple tweaks to your plugin to do this:

Plugin source

import { globSync as glob } from "glob";
import { Application, Context, Converter, DeclarationReflection, ParameterType, TypeScript as ts } from "typedoc";

declare module "typescript" {
    interface Node {
        symbol?: Symbol;
    }
}

const CUCUMBER_FUNCTION_NAMES = [
    "Given",
    "When",
    "Then",
    "Before",
    "After",
    "AfterAll",
    "BeforeAll",
    "BeforeEach",
    "AfterEach",
] as const;

type CucumberFunctionNames = (typeof CUCUMBER_FUNCTION_NAMES)[number];

/**
 * Reads a TypeScript file and extracts any documentation on Step Definitions.
 * @param sourceFile The TypeScript source file
 */
function readSourceFile(sourceFile: ts.SourceFile, context: Context) {
    // These are what store the JSDocs
    const toplevelExpressions = sourceFile.statements.filter(ts.isExpressionStatement);

    for (const expression of toplevelExpressions) {
        const cucumberFunctionName = isCucumberFunctionCall(expression);
        if (cucumberFunctionName === false) {
            // Not a cucumber function
            continue;
        }

        const expr = expression.expression as ts.CallExpression;
        const template = expr.arguments[0] as ts.Identifier;
        const implementation = expr.arguments[1];

        const symbol = implementation.symbol;
        if (!symbol) return; // Something broke. Should always have one.

        context.converter.convertSymbol(context, symbol);
        const reflection = context.project.getReflectionFromSymbol(symbol) as DeclarationReflection | undefined;
        if (!reflection) continue; // Excluded, probably because of an @ignore

        reflection.name = `${cucumberFunctionName} ${template.text}`;
        for (const sig of reflection.signatures || []) {
            sig.comment ||= context.getNodeComment(expression, sig.kind);
            sig.name = cucumberFunctionName;
        }
    }
}

/**
 * Determines if the expression is a cucumber support code function call.
 * @param expression
 * @returns the name of the cucumber function that was called, or false in a situation where the expression is not a function call or a cucumber function.
 */
function isCucumberFunctionCall(expression: ts.ExpressionStatement): CucumberFunctionNames | false {
    const innerExpression = expression.expression as ts.CallExpression;
    if (!ts.isCallExpression(innerExpression)) {
        return false;
    }

    if (!ts.isIdentifier(innerExpression.expression)) {
        return false;
    }

    if (!CUCUMBER_FUNCTION_NAMES.includes(innerExpression.expression.text as any)) {
        return false;
    }

    return innerExpression.expression.text as CucumberFunctionNames;
}

export function load(app: Application) {
    app.options.addDeclaration({
        name: "stepDefinitions",
        help:
            "Extract TSDocs from cucumber step definitions listed at the specified paths. " +
            "Should be a list of files relative to the project root, represented as a relative path or glob.",
        type: ParameterType.GlobArray,
        defaultValue: [],
    });

    app.converter.on(
        Converter.EVENT_RESOLVE_BEGIN,
        (ctx: Context) => {
            let paths = app.options.getValue("stepDefinitions") as string[];

            if (paths.length == 0) {
                paths = app.options.getValue("entryPoints");
            }

            const resolvedPaths = paths.map((p) => glob(p)).flat();

            for (const program of ctx.programs) {
                for (const path of resolvedPaths) {
                    const sourceFile = program.getSourceFile(path);

                    if (!sourceFile) {
                        continue;
                    }

                    const context = ctx.withScope(ctx.project);
                    context.setActiveProgram(program);
                    readSourceFile(sourceFile, context);
                }
            }
        },
        100,
    );
}

A couple notes on the changes:

• Event handlers may not be async.
• context.programs containing duplicates was a bug, fixed now.
• Adding additional reflections to the project should be done with a high priority so that if the user includes @hidden in a comment, the comment plugin can remove it. A priority of 1 is sufficient, but 100 makes it possible to insert additional listeners in between more easily.
• (FYI, not doing manual creation above) If creating reflections manually, they should be created with a reference to their parent, and context.project.registerReflection should be called. If it isn't then some things like link resolution will mysteriously break. context.createDeclarationReflection will do this for you.

[tristanzander]

@Gerrit0 This is a FANTASTIC example that you provided. This is absolutely the piece of information that I was missing to make this plugin work. I do believe I can use custom reflections/themes with that generated function signature in order to build out the plugin the way I was hoping. I cannot express how much I appreciate this

Adding additional reflections to the project should be done with a high priority so that if the user includes @hidden in a comment, the comment plugin can remove it. A priority of 1 is sufficient, but 100 makes it possible to insert additional listeners in between more easily.

Could you elaborate on the priority a bit further? Are you saying that we should recommend that the plugin is loaded early to prevent changes from this plugin being ignored by other plugins? Nevermind, I see what you mean

[tristanzander]

Surprisingly, to me, you're at least the fourth person to have tried something like this.

Also regarding this, I can imagine it being useful for more than just Cucumber to be able to document framework-level reflections for specific things. Mainly, these would only really apply to things that are governed by the following properties:

• Generated/registered at runtime (which is the case for Cucumber)
• Governed by folder structure (not totally sure if/why you'd want to document routes with NextJS through TypeDoc)
• Not traditionally exported by the program through any listed entry points, but still expected to be required/imported for any such reason (which is also the case for Cucumber)

I think it's probably important to consider the scope of TypeDoc when making decisions on how well to support these use-cases, because it really makes your life infinitely harder to deal with the edge-cases that arise from that. Like you mentioned, TypeDoc is bound to TS symbols, so it maybe shouldn't do more than that anyway. Just thinking out loud here.

I want to eventually make it possible for people to create custom folder structures in output, but have yet to come up with something I'm happy with... not concretely tied to any release target yet.

Both this repo and typedoc-plugin-markdown do their folder/type mapping almost the exact same way. You've already provided the types to be able to make a generic interface for the mappings. It's certainly possible to make a standardized equivalent to DefaultTheme.mappings extensible for plugins, depending on how easily you can abstract renderer implementations over the mapping template. I can see how you'd have much trouble creating a solution for this, because there are downsides either way. I might look into it this weekend to see if I could come up with a decent solution.

[tristanzander]

My POC was successful. I will open any issues if I come across any other problems. Thank you for your help @Gerrit0!

[samwatts98]

Thanks very much for this plugin, it's exactly what I was looking for too!

Just a quick note though, the plugin above doesn't work on Windows, the values of options entryPoints and stepDefinitions resolve to use windows path separators '\', even though the options themselves are using relatives paths in POSIX format '/'. Just changing the resolvedPaths definition line to the following line resolves that though:

const resolvedPaths = paths.map((p) => glob(p.replace(/\\/g, '/'))).flat();