Skip to content

Commit

Permalink
refactor(parser): type YTNodes' data arg as RawNode (wip) (#339)
Browse files Browse the repository at this point in the history
* replaced YTNode's data arg as RawNode

* updated documentation

* removed unused import

---- Note that there are still many nodes that need to be updated, hence the WIP status.
  • Loading branch information
chinmay021 authored Mar 7, 2023
1 parent 95033e7 commit cfc1a18
Show file tree
Hide file tree
Showing 90 changed files with 190 additions and 132 deletions.
14 changes: 9 additions & 5 deletions docs/updating-the-parser.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Updating the parser

YouTube is constantly changing, so it is not rare to see YouTube crawlers/scrapers breaking every now and then.
YouTube is constantly changing, so it is not rare to see YouTube crawlers/scrapers breaking every now and then.

Our parser, on the other hand, was written so that it behaves similarly to an official client, parsing and mapping renderers (a.k.a YTNodes) dynamically without hard-coding their path in the response. This way, whenever a new renderer pops up (e.g; YouTube adds a new feature / minor UI changes) the library will print a warning similar to this:

Our parser, on the other hand, was written so that it behaves similarly to an official client, parsing and mapping renderers (a.k.a YTNodes) dynamically without hard-coding their path in the response. This way, whenever a new renderer pops up (e.g; YouTube adds a new feature / minor UI changes) the library will print a warning similar to this:
```
InnertubeError: SomeRenderer not found!
This is a bug, want to help us fix it? Follow the instructions at https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md or report it at https://github.com/LuanRT/YouTube.js/issues!
Expand All @@ -26,17 +27,19 @@ Thanks to the modularity of the parser, a renderer can be implemented by simply
For example, say we found a new renderer named `verticalListRenderer`, to let the parser know it exists we would have to create a file with the following structure:

> `../classes/VerticalList.ts`
```ts
import Parser from '..';
import { YTNode } from '../helpers';
import type { RawNode } from '../index.js';

class VerticalList extends YTNode {
static type = 'VerticalList';

header;
contents;

constructor(data: any) {
constructor(data: RawNode) {
// parse the data here, ex;
this.header = Parser.parseItem(data.header);
this.contents = Parser.parseArray(data.contents);
Expand All @@ -47,8 +50,9 @@ export default VerticalList;
```

Then update the parser map:

```bash
npm run build:parser-map
```

And that's it!
And that's it!
3 changes: 2 additions & 1 deletion src/parser/classes/AccountChannel.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
import Text from './misc/Text.js';
import NavigationEndpoint from './NavigationEndpoint.js';
import { YTNode } from '../helpers.js';
import type { RawNode } from '../index.js';

class AccountChannel extends YTNode {
static type = 'AccountChannel';

title: Text;
endpoint: NavigationEndpoint;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.title = new Text(data.title);
this.endpoint = new NavigationEndpoint(data.navigationEndpoint);
Expand Down
5 changes: 3 additions & 2 deletions src/parser/classes/AccountItemSection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import NavigationEndpoint from './NavigationEndpoint.js';
import AccountItemSectionHeader from './AccountItemSectionHeader.js';

import { YTNode } from '../helpers.js';
import type { RawNode } from '../index.js';

class AccountItem {
static type = 'AccountItem';
Expand All @@ -18,7 +19,7 @@ class AccountItem {
endpoint: NavigationEndpoint;
account_byline: Text;

constructor(data: any) {
constructor(data: RawNode) {
this.account_name = new Text(data.accountName);
this.account_photo = Thumbnail.fromResponse(data.accountPhoto);
this.is_selected = data.isSelected;
Expand All @@ -35,7 +36,7 @@ class AccountItemSection extends YTNode {
contents;
header;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.contents = data.contents.map((ac: any) => new AccountItem(ac.accountItem));
this.header = Parser.parseItem<AccountItemSectionHeader>(data.header, AccountItemSectionHeader);
Expand Down
4 changes: 2 additions & 2 deletions src/parser/classes/AccountItemSectionHeader.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
import Text from './misc/Text.js';
import { YTNode } from '../helpers.js';

import type { RawNode } from '../index.js';
class AccountItemSectionHeader extends YTNode {
static type = 'AccountItemSectionHeader';

title: Text;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.title = new Text(data.title);
}
Expand Down
4 changes: 2 additions & 2 deletions src/parser/classes/AccountSectionList.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ import AccountChannel from './AccountChannel.js';
import AccountItemSection from './AccountItemSection.js';

import { YTNode } from '../helpers.js';

import type { RawNode } from '../index.js';
class AccountSectionList extends YTNode {
static type = 'AccountSectionList';

contents;
footers;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.contents = Parser.parseItem<AccountItemSection>(data.contents[0], AccountItemSection);
this.footers = Parser.parseItem<AccountChannel>(data.footers[0], AccountChannel);
Expand Down
4 changes: 2 additions & 2 deletions src/parser/classes/Alert.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
import Text from './misc/Text.js';
import { YTNode } from '../helpers.js';

import type { RawNode } from '../index.js';
class Alert extends YTNode {
static type = 'Alert';

text: Text;
alert_type: string;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.text = new Text(data.text);
this.alert_type = data.type;
Expand Down
4 changes: 2 additions & 2 deletions src/parser/classes/AudioOnlyPlayability.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import { YTNode } from '../helpers.js';

import type { RawNode } from '../index.js';
class AudioOnlyPlayability extends YTNode {
static type = 'AudioOnlyPlayability';

audio_only_availability: string;

constructor (data: any) {
constructor (data: RawNode) {
super();
this.audio_only_availability = data.audioOnlyAvailability;
}
Expand Down
4 changes: 2 additions & 2 deletions src/parser/classes/AutomixPreviewVideo.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
import { YTNode } from '../helpers.js';
import NavigationEndpoint from './NavigationEndpoint.js';

import type { RawNode } from '../index.js';
class AutomixPreviewVideo extends YTNode {
static type = 'AutomixPreviewVideo';

playlist_video?: { endpoint: NavigationEndpoint };

constructor(data: any) {
constructor(data: RawNode) {
super();
if (data?.content?.automixPlaylistVideoRenderer?.navigationEndpoint) {
this.playlist_video = {
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/actions/AppendContinuationItemsAction.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
import Parser from '../../index.js';
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class AppendContinuationItemsAction extends YTNode {
static type = 'AppendContinuationItemsAction';

items;
target: string;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.items = Parser.parse(data.continuationItems);
this.target = data.target;
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/actions/OpenPopupAction.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
import Parser from '../../index.js';
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class OpenPopupAction extends YTNode {
static type = 'OpenPopupAction';

popup;
popup_type;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.popup = Parser.parse(data.popup);
this.popup_type = data.popupType;
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/AnalyticsMainAppKeyMetrics.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
import DataModelSection from './DataModelSection.js';
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class AnalyticsMainAppKeyMetrics extends YTNode {
static type = 'AnalyticsMainAppKeyMetrics';

period: string;
sections: DataModelSection[];

constructor(data: any) {
constructor(data: RawNode) {
super();
this.period = data.cardData.periodLabel;
const metrics_data = data.cardData.sections[0].analyticsKeyMetricsData;
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/AnalyticsRoot.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class AnalyticsRoot extends YTNode {
static type = 'AnalyticsRoot';
Expand All @@ -19,7 +20,7 @@ class AnalyticsRoot extends YTNode {
}[];
}[];

constructor(data: any) {
constructor(data: RawNode) {
super();
const cards = data.analyticsTableCarouselData.data.tableCards;

Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/AnalyticsShortsCarouselCard.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';
import NavigationEndpoint from '../NavigationEndpoint.js';

class AnalyticsShortsCarouselCard extends YTNode {
Expand All @@ -11,7 +12,7 @@ class AnalyticsShortsCarouselCard extends YTNode {
endpoint: NavigationEndpoint;
}[];

constructor(data: any) {
constructor(data: RawNode) {
super();
this.title = data.title;
this.shorts = data.shortsCarouselData.shorts.map((short: any) => ({
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/AnalyticsVideo.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import Thumbnail from '../misc/Thumbnail.js';
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class AnalyticsVideo extends YTNode {
static type = 'AnalyticsVideo';
Expand All @@ -13,7 +14,7 @@ class AnalyticsVideo extends YTNode {
is_short: boolean;
};

constructor(data: any) {
constructor(data: RawNode) {
super();
this.title = data.videoTitle;

Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/AnalyticsVodCarouselCard.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import Video from './AnalyticsVideo.js';
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class AnalyticsVodCarouselCard extends YTNode {
static type = 'AnalyticsVodCarouselCard';
Expand All @@ -8,7 +9,7 @@ class AnalyticsVodCarouselCard extends YTNode {
videos: Video[] | null;
no_data_message?: string;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.title = data.title;

Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/CtaGoToCreatorStudio.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class CtaGoToCreatorStudio extends YTNode {
static type = 'CtaGoToCreatorStudio';

title: string;
use_new_specs: boolean;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.title = data.buttonLabel;
this.use_new_specs = data.useNewSpecs;
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/DataModelSection.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class DataModelSection extends YTNode {
static type = 'DataModelSection';
Expand Down Expand Up @@ -36,7 +37,7 @@ class DataModelSection extends YTNode {
}
};

constructor(data: any) {
constructor(data: RawNode) {
super();

this.title = data.title;
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/analytics/StatRow.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
import Text from '../misc/Text.js';

import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class StatRow extends YTNode {
static type = 'StatRow';

title: Text;
contents: Text;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.title = new Text(data.title);
this.contents = new Text(data.contents);
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/comments/AuthorCommentBadge.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class AuthorCommentBadge extends YTNode {
static type = 'AuthorCommentBadge';
Expand All @@ -9,7 +10,7 @@ class AuthorCommentBadge extends YTNode {
tooltip: string;
style?: string;

constructor(data: any) {
constructor(data: RawNode) {
super();

this.icon_type = data.icon?.iconType || null;
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/comments/Comment.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import type Actions from '../../../core/Actions.js';
import Proto from '../../../proto/index.js';
import { InnertubeError } from '../../../utils/Utils.js';
import { YTNode, SuperParsedResult } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class Comment extends YTNode {
static type = 'Comment';
Expand Down Expand Up @@ -44,7 +45,7 @@ class Comment extends YTNode {
is_pinned: boolean;
is_member: boolean;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.content = new Text(data.contentText);
this.published = new Text(data.publishedTimeText);
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/comments/CommentActionButtons.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import type Button from '../Button.js';
import type ToggleButton from '../ToggleButton.js';
import type CreatorHeart from './CreatorHeart.js';
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class CommentActionButtons extends YTNode {
static type = 'CommentActionButtons';
Expand All @@ -12,7 +13,7 @@ class CommentActionButtons extends YTNode {
reply_button;
creator_heart;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.like_button = Parser.parseItem<ToggleButton>(data.likeButton);
this.dislike_button = Parser.parseItem<ToggleButton>(data.dislikeButton);
Expand Down
3 changes: 2 additions & 1 deletion src/parser/classes/comments/CommentDialog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import Thumbnail from '../misc/Thumbnail.js';
import type Button from '../Button.js';
import type EmojiPicker from './EmojiPicker.js';
import { YTNode } from '../../helpers.js';
import type { RawNode } from '../../index.js';

class CommentDialog extends YTNode {
static type = 'CommentDialog';
Expand All @@ -16,7 +17,7 @@ class CommentDialog extends YTNode {
emoji_button: Button | null;
emoji_picker: any | null;

constructor(data: any) {
constructor(data: RawNode) {
super();
this.editable_text = new Text(data.editableText);
this.author_thumbnail = Thumbnail.fromResponse(data.authorThumbnail);
Expand Down
Loading

0 comments on commit cfc1a18

Please sign in to comment.