Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[api-extractor] Fix incorrect declaration references for symbols not exported from the package's entry point #3584

Merged
merged 4 commits into from
Sep 2, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions apps/api-extractor/src/generators/ApiModelGenerator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,13 +54,7 @@ export class ApiModelGenerator {
public constructor(collector: Collector) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran your build of API Extractor on some monorepo projects and compared the resulting .api.json files with the baseline output. Here's some differences:

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I've deleted some replies, since I was comparing in the wrong direction 😆 )

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1. Interesting case with inline type destructuring

This is NOT good API practices. 😄 But it is an interesting edge case:

forgotten-exports.d.ts

export declare type HeadersInitializer = Record<string, string>;
export interface IOptions {
    headers?: HeadersInitializer;
}

index.d.ts

import { IOptions } from './forgotten-exports';

export declare const example1: ({ headers: addHeaders }: IOptions) => Promise<void>;

Before this PR, the excerpt for example1 includes:
"canonicalReference": "the-package!IOptions#headers"

After this PR:
"canonicalReference": "the-package!IOptions.headers"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2. Case involving "typeof"

// Resolves to [email protected]\node_modules\webpack\types.d.ts
import webpack from 'webpack';

export declare type Example2 = typeof webpack.prototype.inputFileSystem;

Before this PR:

          "excerptTokens": [
            {
              "kind": "Content",
              "text": "export declare type FileSystem = "
            },
            {
              "kind": "Content",
              "text": "typeof "
            },
            {
              "kind": "Reference",
              "text": "webpack.prototype",
              "canonicalReference": "!Function#prototype:member"
            },
            {
              "kind": "Content",
              "text": ".inputFileSystem"
            },
            {
              "kind": "Content",
              "text": ";"
            }
          ],

After this PR:

"canonicalReference": "!Function.prototype:member"

I'm not sure either string is really correct. 😋

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These examples are really useful! I couldn't find any examples of code generating Navigation.Members navigation steps. I'll take a look at these. 👍

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3. Case involving preact

This is not a bug really, more of a curiosity:

import { Context } from 'preact';

export declare const Example3: Context<{}>;

Before this PR:

            {
              "kind": "Reference",
              "text": "Context",
              "canonicalReference": "preact!~preact.Context:interface"
            },

After this PR:

            {
              "kind": "Reference",
              "text": "Context",
              "canonicalReference": "preact!preact.Context:interface"
            },

It is an improvement, but I think this canonical reference maybe should not have the extra preact part?

The Context is declared like this:

[email protected]\node_modules\preact\src\index.d.ts

export = preact;
export as namespace preact;

. . .

declare namespace preact {
	. . .
	interface Context<T> {
		Consumer: Consumer<T>;
		Provider: Provider<T>;
		displayName?: string;
	}
}

So it is a member of namespace preact but the intended way of importing is really import { Context } from 'preact';. 🤷‍♂️

Copy link
Collaborator

@octogonz octogonz Aug 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Across >30 libraries, these were the only weird cases. There were a number of canonical references that are fixed by your PR. Otherwise I didn't see any obvious regressions. 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. I think cases 1 and 2 should now have the same behavior as before this PR. Previously, I didn't think it was possible for a members navigation step to be present in an identifier passed to this function, but the cases you shared demonstrated it is possible.

I didn't tackle case 3 because this PR is an improvement, and it seemed like potentially a larger fix. But I agree that the currently behavior isn't necessarily ideal.

this._collector = collector;
this._apiModel = new ApiModel();
this._referenceGenerator = new DeclarationReferenceGenerator(
collector.packageJsonLookup,
collector.workingPackage.name,
collector.program,
collector.typeChecker,
collector.bundledPackageNames
);
this._referenceGenerator = new DeclarationReferenceGenerator(collector);
}

public get apiModel(): ApiModel {
Expand Down
154 changes: 60 additions & 94 deletions apps/api-extractor/src/generators/DeclarationReferenceGenerator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,38 +10,26 @@ import {
Navigation,
Meaning
} from '@microsoft/tsdoc/lib-commonjs/beta/DeclarationReference';
import { PackageJsonLookup, INodePackageJson, InternalError } from '@rushstack/node-core-library';
import { INodePackageJson, InternalError } from '@rushstack/node-core-library';
import { TypeScriptHelpers } from '../analyzer/TypeScriptHelpers';
import { TypeScriptInternals } from '../analyzer/TypeScriptInternals';
import { Collector } from '../collector/Collector';
import { CollectorEntity } from '../collector/CollectorEntity';

export class DeclarationReferenceGenerator {
public static readonly unknownReference: string = '?';

private _packageJsonLookup: PackageJsonLookup;
private _workingPackageName: string;
private _program: ts.Program;
private _typeChecker: ts.TypeChecker;
private _bundledPackageNames: ReadonlySet<string>;

public constructor(
packageJsonLookup: PackageJsonLookup,
workingPackageName: string,
program: ts.Program,
typeChecker: ts.TypeChecker,
bundledPackageNames: ReadonlySet<string>
) {
this._packageJsonLookup = packageJsonLookup;
this._workingPackageName = workingPackageName;
this._program = program;
this._typeChecker = typeChecker;
this._bundledPackageNames = bundledPackageNames;
private _collector: Collector;

public constructor(collector: Collector) {
this._collector = collector;
}

/**
* Gets the UID for a TypeScript Identifier that references a type.
*/
public getDeclarationReferenceForIdentifier(node: ts.Identifier): DeclarationReference | undefined {
const symbol: ts.Symbol | undefined = this._typeChecker.getSymbolAtLocation(node);
const symbol: ts.Symbol | undefined = this._collector.typeChecker.getSymbolAtLocation(node);
if (symbol !== undefined) {
const isExpression: boolean = DeclarationReferenceGenerator._isInExpressionContext(node);
return (
Expand Down Expand Up @@ -99,68 +87,48 @@ export class DeclarationReferenceGenerator {
);
}

private static _getNavigationToSymbol(symbol: ts.Symbol): Navigation | 'global' {
private _getNavigationToSymbol(symbol: ts.Symbol): Navigation {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an interesting canonical reference question: Consider the following example:

// index.ts (entry point of "my package")
import * as n from './internal';
export function someFunction(): n.SomeType { return 5; }

// internal.ts
export type SomeType = number;

What should the canonical reference of SomeType be? Today...

  • DeclarationReferenceGenerator produces the following: my-package!~SomeType:type (after this PR)
  • whereas api-extractor-model produces the following: my-package!~n.SomeType

api-extractor-model produces the latter because the "synthetic" AstNamespaceImport for n makes its way into the .api.json.

I feel like the correct canonical reference in this case is my-package!~SomeType:type, because the namespace n is really an implementation detail of how index.ts imports from internal.ts, and not part of SomeType itself. There could be the following code in internal.ts:

export function someOtherFunction(): SomeType { return 6; }

and it would be odd for someOtherFunction's reference token for SomeType to include the synthetic namespace n. I'd expect for it to be my-package!~SomeType:type.

Regardless, I don't think we should solve this in this PR, just something to think about.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm.... A central principle of API Extractor is that it interprets declarations, imposing a semantics that is not technically present in the source code. This is what justifies rolling up declarations into a different file from where they were declared, discarding some .d.ts files entirely (because they aren't reachable from the entry point), and renaming API items to avoid naming conflicts.

Your question is really about what semantics should be applied to an unexported declaration.

  1. I'm a little sketchy on the grammar, but I think the physical location would be described as my-package/lib/internal!~SomeType:type.

  2. But generally API Extractor's model is that of the .d.ts rollup. Physically in the .d.ts rollup, the location would be my-package!~SomeType:type.

  3. However if there are naming collisions, this could become something arbitrary like my-package!~SomeType_3:type.

  4. Logically, the human intent of import * as n is to move this declaration inside n which would better be described as my-package!n~SomeType:type. The reason it does not get placed there was technical: if they also did import * as n2 then its ambiguous which one is the "real" definition and which one is the alias. With more sophisticated rollup logic, we could solve this problem in the future, and then my-package!n~SomeType:type maybe would be correct.

I guess the choice boils down to requirements:

Is our priority that the canonical reference should help a tool to find the declaration in the .d.ts files? If so then it should probably be a physical location in the .d.ts rollup. If the .d.ts rollup feature is disabled, then maybe all canonical references should actually be physical locations such as (1).

Alternatively, is our priority that the canonical reference should be a stable identifier, i.e. resistant to getting shuffled around by trivial changes? I think this is the design choice that API Extractor has pursued. In this case I guess we would go with (2) or (3).

const declaration: ts.Declaration | undefined = TypeScriptHelpers.tryGetADeclaration(symbol);
const sourceFile: ts.SourceFile | undefined = declaration?.getSourceFile();
const parent: ts.Symbol | undefined = TypeScriptInternals.getSymbolParent(symbol);
// First, try to determine navigation to symbol via its parent.
if (parent) {
if (
parent.exports &&
DeclarationReferenceGenerator._isSameSymbol(parent.exports.get(symbol.escapedName), symbol)
) {
return Navigation.Exports;
}
if (
parent.members &&
DeclarationReferenceGenerator._isSameSymbol(parent.members.get(symbol.escapedName), symbol)
) {
return Navigation.Members;
}
if (
parent.globalExports &&
DeclarationReferenceGenerator._isSameSymbol(parent.globalExports.get(symbol.escapedName), symbol)
) {
return 'global';
}

// If it's a global, then use an Exports navigation.
if (sourceFile && !ts.isExternalModule(sourceFile)) {
return Navigation.Exports;
}

// Next, try determining navigation to symbol by its node
if (symbol.valueDeclaration) {
const declaration: ts.Declaration = ts.isBindingElement(symbol.valueDeclaration)
? ts.walkUpBindingElementsAndPatterns(symbol.valueDeclaration)
: symbol.valueDeclaration;
if (ts.isClassElement(declaration) && ts.isClassLike(declaration.parent)) {
// class members are an "export" if they have the static modifier.
return ts.getCombinedModifierFlags(declaration) & ts.ModifierFlags.Static
? Navigation.Exports
: Navigation.Members;
}
if (ts.isTypeElement(declaration) || ts.isObjectLiteralElement(declaration)) {
// type and object literal element members are just members
return Navigation.Members;
}
if (ts.isEnumMember(declaration)) {
// enum members are exports
return Navigation.Exports;
}
if (
ts.isExportSpecifier(declaration) ||
ts.isExportAssignment(declaration) ||
ts.isExportSpecifier(declaration) ||
ts.isExportDeclaration(declaration) ||
ts.isNamedExports(declaration)
) {
return Navigation.Exports;
// If it's from an external library, then use an Exports navigation.
if (sourceFile && this._collector.program.isSourceFileFromExternalLibrary(sourceFile)) {
return Navigation.Exports;
}

// Otherwise, this symbol is from the current package.
if (parent) {
// If we've found an exported CollectorEntity, then it's exported from the package entry point, so
// use an Exports navigation.
const namedDeclaration: ts.DeclarationName | undefined = (
declaration as ts.NamedDeclaration | undefined
)?.name;
if (namedDeclaration && ts.isIdentifier(namedDeclaration)) {
const collectorEntity: CollectorEntity | undefined =
this._collector.tryGetEntityForNode(namedDeclaration);
octogonz marked this conversation as resolved.
Show resolved Hide resolved
if (collectorEntity && collectorEntity.exported) {
return Navigation.Exports;
}
}
// declarations are exports if they have an `export` modifier.
if (ts.getCombinedModifierFlags(declaration) & ts.ModifierFlags.Export) {

// If its parent symbol is not a source file, then use an Exports navigation. If the parent symbol is
// a source file, but it wasn't exported from the package entry point (in the check above), then the symbol
// is a local, so fall through below.
if (!DeclarationReferenceGenerator._isExternalModuleSymbol(parent)) {
return Navigation.Exports;
}
Copy link
Contributor Author

@zelliott zelliott Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my testing, I could not find a scenario in which any line from 112 to 157 was hit. I could also not find a situation where Members navigation was ever returned. I'm worried I may be missing an entire class of cases here... the original logic was added in the mega-PR #1337.

Maybe this logic was here because previously this method was used to generate the canonical references for API items in the .api.json, but now these references are generated from the .api.json structure itself.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rbuckton any idea what kind of type declaration input would cover these these code paths?

Copy link
Contributor Author

@zelliott zelliott Aug 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@octogonz shared some instances where Members navigation was returned, and I'm covering those now. But I still can not find a scenario where any line from 118-157 was hit.

if (ts.isSourceFile(declaration.parent) && !ts.isExternalModule(declaration.parent)) {
// declarations in a source file are global if the source file is not a module.
return 'global';
}
}
// all other declarations are locals

// Otherwise, we have a local symbol, so use a Locals navigation. These are either:
//
// 1. Symbols that are exported from a file module but not the package entry point.
// 2. Symbols that are not exported from their parent module.
return Navigation.Locals;
}

Expand Down Expand Up @@ -218,20 +186,21 @@ export class DeclarationReferenceGenerator {
meaning: ts.SymbolFlags,
includeModuleSymbols: boolean
): DeclarationReference | undefined {
const declaration: ts.Node | undefined = TypeScriptHelpers.tryGetADeclaration(symbol);
const sourceFile: ts.SourceFile | undefined = declaration?.getSourceFile();

let followedSymbol: ts.Symbol = symbol;
if (followedSymbol.flags & ts.SymbolFlags.ExportValue) {
followedSymbol = this._typeChecker.getExportSymbolOfSymbol(followedSymbol);
followedSymbol = this._collector.typeChecker.getExportSymbolOfSymbol(followedSymbol);
}
if (followedSymbol.flags & ts.SymbolFlags.Alias) {
followedSymbol = this._typeChecker.getAliasedSymbol(followedSymbol);
followedSymbol = this._collector.typeChecker.getAliasedSymbol(followedSymbol);
}

if (DeclarationReferenceGenerator._isExternalModuleSymbol(followedSymbol)) {
if (!includeModuleSymbols) {
return undefined;
}
const declaration: ts.Node | undefined = TypeScriptHelpers.tryGetADeclaration(symbol);
const sourceFile: ts.SourceFile | undefined = declaration?.getSourceFile();
return new DeclarationReference(this._sourceFileToModuleSource(sourceFile));
}

Expand Down Expand Up @@ -270,13 +239,11 @@ export class DeclarationReferenceGenerator {
}
}

let navigation: Navigation | 'global' =
DeclarationReferenceGenerator._getNavigationToSymbol(followedSymbol);
if (navigation === 'global') {
if (parentRef.source !== GlobalSource.instance) {
parentRef = new DeclarationReference(GlobalSource.instance);
}
navigation = Navigation.Exports;
const navigation: Navigation = this._getNavigationToSymbol(followedSymbol);

// If the symbol is a global, ensure the source is global.
if (sourceFile && !ts.isExternalModule(sourceFile) && parentRef.source !== GlobalSource.instance) {
parentRef = new DeclarationReference(GlobalSource.instance);
}

return parentRef
Expand Down Expand Up @@ -313,7 +280,7 @@ export class DeclarationReferenceGenerator {
if (grandParent && ts.isModuleDeclaration(grandParent)) {
const grandParentSymbol: ts.Symbol | undefined = TypeScriptInternals.tryGetSymbolForDeclaration(
grandParent,
this._typeChecker
this._collector.typeChecker
);
if (grandParentSymbol) {
return this._symbolToDeclarationReference(
Expand All @@ -334,28 +301,27 @@ export class DeclarationReferenceGenerator {
}

private _getPackageName(sourceFile: ts.SourceFile): string {
if (this._program.isSourceFileFromExternalLibrary(sourceFile)) {
const packageJson: INodePackageJson | undefined = this._packageJsonLookup.tryLoadNodePackageJsonFor(
sourceFile.fileName
);
if (this._collector.program.isSourceFileFromExternalLibrary(sourceFile)) {
const packageJson: INodePackageJson | undefined =
this._collector.packageJsonLookup.tryLoadNodePackageJsonFor(sourceFile.fileName);

if (packageJson && packageJson.name) {
return packageJson.name;
}
return DeclarationReferenceGenerator.unknownReference;
}
return this._workingPackageName;
return this._collector.workingPackage.name;
}

private _sourceFileToModuleSource(sourceFile: ts.SourceFile | undefined): GlobalSource | ModuleSource {
if (sourceFile && ts.isExternalModule(sourceFile)) {
const packageName: string = this._getPackageName(sourceFile);

if (this._bundledPackageNames.has(packageName)) {
if (this._collector.bundledPackageNames.has(packageName)) {
// The api-extractor.json config file has a "bundledPackages" setting, which causes imports from
// certain NPM packages to be treated as part of the working project. In this case, we need to
// substitute the working package name.
return new ModuleSource(this._workingPackageName);
return new ModuleSource(this._collector.workingPackage.name);
} else {
return new ModuleSource(packageName);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@
{
"kind": "Reference",
"text": "MyPromise",
"canonicalReference": "api-extractor-scenarios!Promise:class"
"canonicalReference": "api-extractor-scenarios!~Promise:class"
},
{
"kind": "Content",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@
{
"kind": "Reference",
"text": "Options",
"canonicalReference": "api-extractor-scenarios!Options:interface"
"canonicalReference": "api-extractor-scenarios!~Options:interface"
},
{
"kind": "Content",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@
{
"kind": "Reference",
"text": "ForgottenClass",
"canonicalReference": "api-extractor-scenarios!ForgottenClass:class"
"canonicalReference": "api-extractor-scenarios!~ForgottenClass:class"
},
{
"kind": "Content",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@
{
"kind": "Reference",
"text": "Base",
"canonicalReference": "api-extractor-lib2-test!~DefaultClass:class"
"canonicalReference": "api-extractor-lib2-test!DefaultClass:class"
},
{
"kind": "Content",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@
{
"kind": "Reference",
"text": "default",
"canonicalReference": "api-extractor-lib2-test!~DefaultClass:class"
"canonicalReference": "api-extractor-lib2-test!DefaultClass:class"
},
{
"kind": "Content",
Expand Down Expand Up @@ -230,7 +230,7 @@
{
"kind": "Reference",
"text": "DefaultClass_namedImport",
"canonicalReference": "api-extractor-lib2-test!~DefaultClass:class"
"canonicalReference": "api-extractor-lib2-test!DefaultClass:class"
},
{
"kind": "Content",
Expand Down Expand Up @@ -258,7 +258,7 @@
{
"kind": "Reference",
"text": "DefaultClass_reExport",
"canonicalReference": "api-extractor-lib2-test!~DefaultClass:class"
"canonicalReference": "api-extractor-lib2-test!DefaultClass:class"
},
{
"kind": "Content",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,35 @@
},
"implementsTokenRanges": []
},
{
"kind": "Class",
"canonicalReference": "api-extractor-scenarios!SomeClass4:class",
"docComment": "/**\n * Reference to a symbol exported from another file, but not exported from the package.\n *\n * @public\n */\n",
"excerptTokens": [
{
"kind": "Content",
"text": "export declare class SomeClass4 extends "
},
{
"kind": "Reference",
"text": "SomeClass5",
"canonicalReference": "api-extractor-scenarios!~SomeClass5:class"
},
{
"kind": "Content",
"text": " "
}
],
"releaseTag": "Public",
"name": "SomeClass4",
"preserveMemberOrder": false,
"members": [],
"extendsTokenRange": {
"startIndex": 1,
"endIndex": 2
},
"implementsTokenRanges": []
},
{
"kind": "Enum",
"canonicalReference": "api-extractor-scenarios!SomeEnum:enum",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ export class SomeClass1 {
export class SomeClass3 extends SomeClass2 {
}

// Warning: (ae-forgotten-export) The symbol "SomeClass5" needs to be exported by the entry point index.d.ts
//
// @public
export class SomeClass4 extends SomeClass5 {
}

// @public (undocumented)
export enum SomeEnum {
// (undocumented)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,16 @@ declare class SomeClass2 {
export declare class SomeClass3 extends SomeClass2 {
}

/**
* Reference to a symbol exported from another file, but not exported from the package.
* @public
*/
export declare class SomeClass4 extends SomeClass5 {
}

declare class SomeClass5 {
}

/** @public */
export declare enum SomeEnum {
A = "A",
Expand Down
Loading