feat(Parser): just-in-time YTNode generation

[Wykerd]

Sometimes YouTube adds new renderers to their responses. Our current approach requires new nodes to be added to the codebase before users can interact with these new nodes. This is a problem because it requires a new release of the library to be published before users can interact with these new nodes. This PR adds a new feature to the library that allows users to interact with these new nodes without requiring a new release of the library. This is done by generating the missing nodes on-demand at runtime.

Using our existing YTNode class, users can interact with these new nodes in a type-safe way, however will not be able to cast them to the node's specific type, as this requires the node to be defined in the codebase.

This new implementation outputs the generated nodes as TypeScript classes to the console, so that we may implement new nodes easier.

The current implementation recognises the following values:

• Renderers
• Renderer arrays
• Text
• Navigation endpoints
• Author (does not currently detect the author thumbnails)
• Thumbnails

This may be expanded in the future.

At runtime, these JIT generated nodes will revalidate themselves when constructed, so that when the types changes, the node will be re-generated.

Basic Example

Given an unknown node's classdata (in this case SearchSubMenu), we can generate a new node class on-demand:

const unknown_node_classdata = {
  "title": {
    "runs": [
      {
        "text": "Search options"
      }
    ]
  },
  "groups": [
    {
      "searchFilterGroupRenderer": {
        "title": {
          "simpleText": "Upload date"
        },
        "filters": [
          {
            "searchFilterRenderer": {
              "label": {
                "simpleText": "Last hour"
              },
              "navigationEndpoint": {
                "clickTrackingParams": "CDIQk3UYACITCM6W-5DpgP0CFY2OFgodJbYGdQ==",
                "commandMetadata": {
                  "webCommandMetadata": {
                    "url": "/results?search_query=Space+DOES+NOT+expand+everywhere&sp=EgIIAQ%253D%253D",
                    "webPageType": "WEB_PAGE_TYPE_SEARCH",
                    "rootVe": 4724
                  }
                },
                "searchEndpoint": {
                  "query": "Space DOES NOT expand everywhere",
                  "params": "EgIIAQ%3D%3D"
                }
              },
              "tooltip": "Search for Last hour",
              "trackingParams": "CDIQk3UYACITCM6W-5DpgP0CFY2OFgodJbYGdQ=="
            }
          },
          // ...
        ],
        "trackingParams": "CC0QknUYACITCM6W-5DpgP0CFY2OFgodJbYGdQ=="
      }
    },
    // ...
  ],
  "trackingParams": "CBEQkXUiEwjOlvuQ6YD9AhWNjhYKHSW2BnU=",
  "button": {
    "toggleButtonRenderer": {
      "style": {
        "styleType": "STYLE_TEXT"
      },
      "isToggled": false,
      "isDisabled": false,
      "defaultIcon": {
        "iconType": "TUNE"
      },
      "defaultText": {
        "runs": [
          {
            "text": "Filters"
          }
        ]
      },
      "accessibility": {
        "label": "Search filters"
      },
      "trackingParams": "CBIQmE0iEwjOlvuQ6YD9AhWNjhYKHSW2BnU=",
      "defaultTooltip": "Open search filters",
      "toggledTooltip": "Close search filters",
      "toggledStyle": {
        "styleType": "STYLE_DEFAULT_ACTIVE"
      },
      "accessibilityData": {
        "accessibilityData": {
          "label": "Search filters"
        }
      }
    }
  }
}

Generates the following TypeScript code and outputs it as warnings to the console:

// SearchSubMenu not found!
// This is a bug, want to help us fix it? Follow the instructions at https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md or report it at https://github.com/LuanRT/YouTube.js/issues!
// Introspected and JIT compiled this class in the meantime:
class SearchSubMenu extends YTNode {
  static type = 'SearchSubMenu';

  title: Text;
  groups: ObservedArray<YTNodes.SearchFilterGroup>;
  button: YTNodes.ToggleButton;

  constructor(data: any) {
    super();
    this.title = new Text(data.title);
    this.groups = Parser.parse(data.groups, true, [YTNodes.SearchFilterGroup]);
    this.button = Parser.parseItem(data.button, YTNodes.ToggleButton);
  }
}

// SearchFilterGroup not found!
// This is a bug, want to help us fix it? Follow the instructions at https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md or report it at https://github.com/LuanRT/YouTube.js/issues!
// Introspected and JIT compiled this class in the meantime:
class SearchFilterGroup extends YTNode {
  static type = 'SearchFilterGroup';

  title: Text;
  filters: ObservedArray<YTNodes.SearchFilter>;

  constructor(data: any) {
    super();
    this.title = new Text(data.title);
    this.filters = Parser.parse(data.filters, true, [YTNodes.SearchFilter]);
  }
}

// SearchFilter not found!
// This is a bug, want to help us fix it? Follow the instructions at https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md or report it at https://github.com/LuanRT/YouTube.js/issues!
// Introspected and JIT compiled this class in the meantime:
class SearchFilter extends YTNode {
  static type = 'SearchFilter';

  label: Text;
  endpoint: NavigationEndpoint;
  tooltip: /* TODO: determine correct type */ unknown;

  constructor(data: any) {
    super();
    this.label = new Text(data.label);
    this.endpoint = new NavigationEndpoint(data.navigationEndpoint);
    this.tooltip = data.tooltip;
  }
}

Revalidation Example

Sometimes keys are optional, we cannot determine this given only one example, so we need to figure this out as new examples of the node is encountered. This example showcases how this works.

// The first encounter of the node
const classdata1 = {
  "badges": [
    {
      "metadataBadgeRenderer": {
        "style": "BADGE_STYLE_TYPE_SIMPLE",
        "label": "New",
        "trackingParams": "CPcCEKQwGAAiEwiyou7goIP9AhVHxEkHHVHJDAc="
      }
    }
  ],
};

// The second encounter of the node
const classdata2 = {
  "badges": [
    {
      "metadataBadgeRenderer": {
        "style": "BADGE_STYLE_TYPE_SIMPLE",
        "label": "New",
        "trackingParams": "CPcCEKQwGAAiEwiyou7goIP9AhVHxEkHHVHJDAc="
      }
    },
    {
      "thumbnailOverlayTimeStatusRenderer": {
        "text": {
          "accessibility": {
            "accessibilityData": {
              "label": "9 minutes, 37 seconds"
            }
          },
          "simpleText": "9:37"
        },
        "style": "DEFAULT"
      }
    }
  ],
}

// The third encounter of the node
const classdata3 = {};

// Generate the new class
// This will inspect the initial data we have about the node and generate a new class
// We'll find an renderer array of MetadataBadges
const Node = YTNodeGenerator.generateRuntimeClass('Example', classdata1);

// We may also access these runtime generated classes using the Parser.getParserByName method
// const Node = Parser.getParserByName('Example');

// Create the first instance
// nothing special happens, the data does not invalidate any of our assumptions about the node
const node1 = new Node(classdata1); 

// Create the second instance
// during the revalidation, we prove one of the assumptions wrong
// so we need to update the class such that the renderer array now may contain both MetadataBadges and ThumbnailOverlayTimeStatuses
const node2 = new Node(classdata2);

// Create the third instance
// during the revalidation, we prove another assumption wrong
// we find that the renderer array is optional, so we need to update the class such that the renderer array may be undefined
const node3 = new Node(classdata3);

This will output the following warnings to the console:

// Example not found!
// This is a bug, want to help us fix it? Follow the instructions at https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md or report it at https://github.com/LuanRT/YouTube.js/issues!
// Introspected and JIT generated this class in the meantime:
class CompactVideo extends YTNode {
  static type = 'CompactVideo';

  badges: ObservedArray<YTNodes.MetadataBadge>;

  constructor(data: any) {
    super();
    this.badges = Parser.parse(data.badges, true, [YTNodes.MetadataBadge]);
  }
}

// Example changed!
// The following keys where altered: badges
// The class has changed to:
class CompactVideo extends YTNode {
  static type = 'CompactVideo';

  badges: ObservedArray<YTNodes.MetadataBadge | YTNodes.ThumbnailOverlayTimeStatus>;

  constructor(data: any) {
    super();
    this.badges = Parser.parse(data.badges, true, [YTNodes.MetadataBadge, YTNodes.ThumbnailOverlayTimeStatus]);
  }
}

// Example changed!
// The following keys where altered: badges
// The class has changed to:
class CompactVideo extends YTNode {
  static type = 'CompactVideo';

  badges?: ObservedArray<YTNodes.MetadataBadge | YTNodes.ThumbnailOverlayTimeStatus>;

  constructor(data: any) {
    super();
    this.badges = Reflect.has(data, 'badges') ? Parser.parse(data.badges, true, [YTNodes.MetadataBadge, YTNodes.ThumbnailOverlayTimeStatus]) : undefined;
  }
}

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have checked my code and corrected any misspellings

[LuanRT]

Awesome work. I've submitted a review. Thanks!

[Wykerd]

The test cases for music is failing because 'MusicCardShelf' is not found. Don't think I should fix it here since it is out of the scope of this PR. However, you can see how the new generator tried to figure out this node 😆

MusicCardShelf not found!
This is a bug, want to help us fix it? Follow the instructions at https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md or report it at https://github.com/LuanRT/YouTube.js/issues!
Introspected and JIT generated this class in the meantime:
class MusicCardShelf extends YTNode {
  static type = 'MusicCardShelf';

  thumbnail: YTNodes.MusicThumbnail | null;
  title: Text;
  subtitle: Text;
  contents: ObservedArray<YTNodes.Message | YTNodes.MusicResponsiveListItem> | null;
  buttons: ObservedArray<YTNodes.Button> | null;
  menu: YTNodes.Menu | null;
  on_tap: {
    click_tracking_params: string,
    watch_endpoint: NavigationEndpoint
  };
  header: YTNodes.MusicCardShelfHeaderBasic | null;
  thumbnail_overlay: YTNodes.MusicItemThumbnailOverlay | null;

  constructor(data: RawNode) {
    super();
    this.thumbnail = Parser.parseItem(data.thumbnail, [ YTNodes.MusicThumbnail ]);
    this.title = new Text(data.title);
    this.subtitle = new Text(data.subtitle);
    this.contents = Parser.parse(data.contents, true, [ YTNodes.Message, YTNodes.MusicResponsiveListItem ]);
    this.buttons = Parser.parse(data.buttons, true, [ YTNodes.Button ]);
    this.menu = Parser.parseItem(data.menu, [ YTNodes.Menu ]);
    this.on_tap = {
      click_tracking_params: data.onTap.clickTrackingParams,
      watch_endpoint: new NavigationEndpoint(data.onTap.watchEndpoint)
    };
    this.header = Parser.parseItem(data.header, [ YTNodes.MusicCardShelfHeaderBasic ]);
    this.thumbnail_overlay = Parser.parseItem(data.thumbnailOverlay, [ YTNodes.MusicItemThumbnailOverlay ]);
  }
}

[absidue]

Introspected and JIT generated this class in the meantime:

The wording of this message sounds a bit weird

Possible alternatives:
"Introspected and JIT generated class in the meantime"
"JIT generated this class in the meantime"

Also is there a reason you went with Parser.parse and setting the requireArray arg to true instead of using Parser.parseArray?

Other than that it's quite impressive

[absidue]

Also might have missed it in the docs in this pull request, but you seem to have only documented how to manually generate and use nodes but now how the auto-generated ones can be used or accessed.

[Wykerd]

Also is there a reason you went with Parser.parse and setting the requireArray arg to true instead of using Parser.parseArray?

Parse will return null if no data is passed in while parseArray will return an empty array if there's no data. Not sure which if preferred in this case.

[LuanRT]

Looks good. I'll probably add the new YouTube Music node in the main branch today so this doesn't error out upon merging.