feat!: feature complete contents parser #44

Wykerd · 2022-04-29T20:07:47Z

Feature complete contents parser

Description

Add parsers for all known renderers and provide complete typescript definitions for all of them.

This allows the library to parse all of the contents instead discarding a lot of the valuable information.

Type of change

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

I will attempt maintain backward compatibility with the existing APIs

TODO:

Common renderers for all endpoints
Integrate parsers into implementation whilst maintaining backward compatibility
Typescript declarations
Tests

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have checked my code and corrected any misspellings

This is useful for getting results other than videos, like playlists and channels.

LuanRT · 2022-04-29T20:55:20Z

Awesome! This looks very promising, thanks for contributing.

Wykerd · 2022-04-29T21:05:37Z

Just to confirm: This library uses snake_case for fields and camelCase for methods?
I've been using camelCase exclusively, so I'll need to refactor if this is the case.

LuanRT · 2022-04-29T21:06:18Z

lib/core/Actions.js

@@ -283,7 +283,7 @@ async function search(session, client, args = {}) {
  switch (client) {
    case 'YOUTUBE':
      if (args.query) {
-        data.params = Proto.encodeSearchFilter(args.options.period, args.options.duration, args.options.order);
+        data.params = args.options.hasOwnProperty('params') ? args.options.params : Proto.encodeSearchFilter(args.options.period, args.options.duration, args.options.order);


I think this won't be necessary, I've been working on the protobuf definitions for more search filters.

From what I could tell the channels and playlists are only returned if you do not send any params, but maybe I'm missing something?

Yup, that's correct.
But I assume that happens because the “type” search filter is missing in the protobuf definition, so YouTube is basically defaulting to “Video” when it should be “All”.

@Wykerd

This should be fixed in commit 43f3c3f

LuanRT · 2022-04-29T21:07:24Z

Just to confirm: This library uses snake_case for fields and camelCase for methods? I've been using camelCase exclusively, so I'll need to refactor if this is the case.

Yup, that's right.

Added common renderers used when searching artists

These are needed for channel tabs: Videos, Playlists, Community, Channels Additionally, do not merely return text as string, since they may include links which may be navigated to

Channels should be viewable when not logged in, also added 'navigation' type for use in NagivationEndpoint in the future.

The HomeFeed class remains compatible with the existing API

Wykerd

This adds typescript as a development dependency

Wykerd

I believe this is related to issue #21 which also suggested it not be written by hand!

LuanRT · 2022-05-01T01:13:26Z

This adds typescript as a development dependency

Nice! That will make things much easier to maintain.
Also, just to let you know there will be a refactor on Actions.js and Request.js to remove redundancy and simplify the way request errors are handled.

lib/core/HomeFeed.js

This follows commit 1bfe267

Wykerd · 2022-05-01T14:23:41Z

lib/core/Channel.js

+
+    /**
+     * @note home_page only returns videos!
+     * XXX: should some other API be made available to expose the content of the channel


I think advanced users should just use the Simplify API (which I will still document) to get the other types of content on the home page

Maybe the home_page should also expose the channel trailer, but that might break the existing API.

Maybe the home_page should also expose the channel trailer, but that might break the existing API.

The old channel parser was implemented quite recently, so you shouldn't worry about backward compatibility here.

Wykerd · 2022-06-15T19:26:22Z

I don't know if you have noticed, but now the parser traps arrays with a Proxy, that makes it easier to find items etc. Do you think that could completely replace the utilities class Simplify? After some testing I noticed it takes over 2000, 4000ms to find items and maybe the use of regex is to blame.

I've just removed Simplify completely 🎉

LuanRT · 2022-06-15T19:59:59Z

I've just removed Simplify completely 🎉

Perfect! Also, could you move all data access classes from /core to /parser/youtube? I'm going to start working on fully supporting YouTube Music and decided it would be more practical to keep their respective classes separated.

Wykerd · 2022-06-15T20:02:45Z

I've just removed Simplify completely 🎉

Perfect! Also, could you move all data access classes from /core to /parser/youtube? I'm going to start working on fully supporting YouTube Music and decided it would be more practical to keep their respective classes separated.

Sure. Some of those classes are very general though (Feed, TabbedFeed, FilterableFeed) and are meant to abstract away from the underlying results structure.

Also added MenuServiceItemDownload

LuanRT · 2022-06-15T20:27:14Z

Sure. Some of those classes are very general though (Feed, TabbedFeed, FilterableFeed) and are meant to abstract away from the underlying results structure.

No problem, since those are for core functionality then it should be fine.

Wykerd · 2022-06-15T20:48:44Z

To what degree must I try to maintain backward compatibility with v1?

The new getTrending response is completely different from v1

LuanRT · 2022-06-15T21:03:01Z

To what degree must I try to maintain backward compatibility with v1?

The new getTrending response is completely different from v1

Doesn't really matter at this point — we've already changed so many things that it wouldn't make sense to try and keep backward compatibility here and there, but not everywhere.

Wykerd · 2022-06-15T21:23:36Z

I think this PR is about ready to merge then I can continue work on this repo.

LuanRT · 2022-06-15T21:30:39Z

I think this PR is about ready to merge then I can continue work on this repo.

Yup, merging now.

Wykerd added 2 commits April 29, 2022 21:37

feat: allow setting search params to custom value

6a207bc

This is useful for getting results other than videos, like playlists and channels.

feat: add initial parsers for common renderers

a04deb1

LuanRT reviewed Apr 29, 2022

View reviewed changes

Wykerd added 11 commits April 29, 2022 23:22

feat: artist search renderers

fb5b4cb

Added common renderers used when searching artists

refactor: snake_case

d0e363e

feat: channel home page renderers

34d1ae7

feat: parsers for more channel tabs

163e58c

These are needed for channel tabs: Videos, Playlists, Community, Channels Additionally, do not merely return text as string, since they may include links which may be navigated to

feat: channel full metadata

e0e0f60

feat: renderers for playlists

4c93536

refactor!: Actions.browse

e0105bc

Channels should be viewable when not logged in, also added 'navigation' type for use in NagivationEndpoint in the future.

feat: home feed parsers

90f37a7

feat: watch page renderers

0367cc6

feat: start implementing HomeFeed API

a76bab4

The HomeFeed class remains compatible with the existing API

feat: generate types using tsc and jsdoc

bdadb1b

Wykerd commented May 1, 2022

View reviewed changes

Wykerd and others added 2 commits May 1, 2022 04:27

feat: browse continuations from navigationEndpoint

36c08cd

Merge branch 'main' into main

c7c9b32

LuanRT reviewed May 1, 2022

View reviewed changes

lib/core/HomeFeed.js Outdated Show resolved Hide resolved

Wykerd added 5 commits May 1, 2022 11:43

fix: Actions moved to session

ffe49ff

This follows commit 1bfe267

fix: add more typescript config

42f7eff

chore: use correct spaces and quotes

5a8b13d

feat: Trending API

87d7bf8

feat: reimplement existing channel API

f72cdf5

Wykerd commented May 1, 2022

View reviewed changes

feat: add base video feed class

af73c4b

Wykerd added 4 commits June 15, 2022 19:17

fix: liniting issues

fe6aba6

feat(feed): filterable feed for home

50dcf62

feat(feed): tabbed feed for trending & channel

5dc7a43

refactor: remove Simplify completely

9fb7cde

Wykerd added 2 commits June 15, 2022 21:47

chore: merge upstream

02cc213

chore: merge upstream (again)

3966836

Wykerd added 4 commits June 15, 2022 22:10

chore: lint

589a389

refactor: alias items with contents

9f4d62e

refactor: Search to extend Feed

7e5a6a4

fix: Search working

3556edc

Also added MenuServiceItemDownload

refactor: move Channel and Playlist

893d206

fix: pass all tests

fbaa378

Wykerd added 2 commits June 15, 2022 23:17

fix: linting errors

796261e

chore: merge upstream

4751a07

Wykerd marked this pull request as ready for review June 15, 2022 21:23

Wykerd mentioned this pull request Jun 15, 2022

bleeding-edge YouTube search results aren't parsed #73

Closed

4 tasks

LuanRT approved these changes Jun 15, 2022

View reviewed changes

LuanRT merged commit 4a10287 into LuanRT:main Jun 15, 2022

Wykerd mentioned this pull request Jun 15, 2022

No getContinuation for playlists #70

Closed

4 tasks

Wykerd added this to the v2 milestone Jul 13, 2022

LuanRT added enhancement New feature or request breaking-change labels Sep 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: feature complete contents parser #44

feat!: feature complete contents parser #44

Wykerd commented Apr 29, 2022 •

edited

Loading

LuanRT commented Apr 29, 2022

Wykerd commented Apr 29, 2022

LuanRT Apr 29, 2022

Wykerd Apr 29, 2022

LuanRT Apr 29, 2022

LuanRT May 11, 2022

LuanRT commented Apr 29, 2022

Wykerd left a comment

Wykerd left a comment

LuanRT commented May 1, 2022

Wykerd May 1, 2022

LuanRT May 1, 2022

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

feat!: feature complete contents parser #44

feat!: feature complete contents parser #44

Conversation

Wykerd commented Apr 29, 2022 • edited Loading

Feature complete contents parser

Description

Type of change

TODO:

Checklist:

LuanRT commented Apr 29, 2022

Wykerd commented Apr 29, 2022

LuanRT Apr 29, 2022

Choose a reason for hiding this comment

Wykerd Apr 29, 2022

Choose a reason for hiding this comment

LuanRT Apr 29, 2022

Choose a reason for hiding this comment

LuanRT May 11, 2022

Choose a reason for hiding this comment

LuanRT commented Apr 29, 2022

Wykerd left a comment

Choose a reason for hiding this comment

Wykerd left a comment

Choose a reason for hiding this comment

LuanRT commented May 1, 2022

Wykerd May 1, 2022

Choose a reason for hiding this comment

LuanRT May 1, 2022

Choose a reason for hiding this comment

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

Wykerd commented Jun 15, 2022

LuanRT commented Jun 15, 2022

Wykerd commented Apr 29, 2022 •

edited

Loading