feat!: better cross runtime support (#97)

* refactor: remove dependancies removes node-forge and uuid in favor of Web APIs * refactor!: commonjs to es6 To aid with #93 I will make all my changes in TypeScript instead. This is the first step into making that happen. Used: https://github.com/wessberg/cjstoesm * refactor!: NToken and Signature TS files Bring this PR up to speed with #93 * feat: cross platform cache (WIP) this is untested! should remove idb as dependecy. * feat: EventEmitter polyfill * refactor: remove events * feat: HTTPClient based on Fetch API (WIP) * refactor!: parsers refactor (WIP) Initial TS support for parsers as per #93 This adds several type safety checks to the parser which'll help to ensure valid data is returned by the parser. * refactor!: parsers refactor (WIP) Bring more in line with the existing implementations & make less verbose * refactor!: parser refactor I was overcomplicating things, this is much simpler and compatible with the existing JS API * fix: some missed parsers while refactoring * fix: better type inferance for parseResponse * feat(TS): typesafe YTNode casts * feat: more type safety in YTNode and Parser * refactor: VideoInfo download with fetch & TS (WIP) Again, this also does some work for #93 * fix: LiveChat in VideoInfo * refactor!: more typesafety in parser * refactor!: VideoInfo almost completed * refactor!: player and session refactors - Remove the Player class' dependance on Session. - Add additional context to the Session. * refactor!: move auth logic to Session (WIP) * refactor: TS port for Actions and Innertube My fingers hurt from typing out all those types :-P * refactor: NavigationEndpoint TS this is still a WIP and should be improved. NavigationEndpoint should probably be refactored further. * refactor!: VideoInfo compiles without errors * chore: delete old player * fix: import errors It compiles and runs!! * fix: Utils import fixes * fix: several runtime errors * fix: video streaming * chore: remove console.log debugging Whoops, forgot to remove these before I pushed the previous commit * chore: remove old unused dependencies * fix: typescript errors Now emitting declarations and source maps * refactor: TS feed * chore: delete old Feed * refactor: move streamToIterable into Utils * refactor: AccountManager TS * refactor: FilterableFeed to TS * refactor: InteractionManager to TS * refactor: PlaylistManager to TS * refactor: TabbedFeed to TS * refactor: Music to TS (WIP) more work to be done, see TODO comments * fix: getting the tests to pass (6/12) YouTube.js Tests Search ✓ Should search on YouTube (1152 ms) ✕ Should search on YouTube Music (705 ms) ✕ Should retrieve YouTube search suggestions (722 ms) ✓ Should retrieve YouTube Music search suggestions (233 ms) Comments ✓ Should retrieve comments (585 ms) ✕ Should retrieve next batch of comments (221 ms) ✕ Should retrieve comment replies (1 ms) General ✕ Should retrieve playlist with YouTube (732 ms) ✓ Should retrieve home feed (838 ms) ✓ Should retrieve trending content (543 ms) ✓ Should retrieve video info (639 ms) ✕ Should download video (5 ms) * fix: tests (7/12) YouTube.js Tests Search ✓ Should search on YouTube (1984 ms) ✕ Should search on YouTube Music (1139 ms) ✕ Should retrieve YouTube search suggestions (1433 ms) ✓ Should retrieve YouTube Music search suggestions (529 ms) Comments ✓ Should retrieve comments (324 ms) ✓ Should retrieve next batch of comments (395 ms) ✕ Should retrieve comment replies General ✕ Should retrieve playlist with YouTube (653 ms) ✓ Should retrieve home feed (1085 ms) ✓ Should retrieve trending content (513 ms) ✓ Should retrieve video info (921 ms) ✕ Should download video (3 ms) * fix: download tests (8/12) YouTube.js Tests Search ✓ Should search on YouTube (1293 ms) ✕ Should search on YouTube Music (927 ms) ✕ Should retrieve YouTube search suggestions (1250 ms) ✓ Should retrieve YouTube Music search suggestions (258 ms) Comments ✓ Should retrieve comments (803 ms) ✓ Should retrieve next batch of comments (511 ms) ✕ Should retrieve comment replies General ✕ Should retrieve playlist with YouTube (528 ms) ✓ Should retrieve home feed (1047 ms) ✓ Should retrieve trending content (548 ms) ✓ Should retrieve video info (825 ms) ✓ Should download video (1779 ms) * fix: tests (9/12) YouTube.js Tests Search ✓ Should search on YouTube (1276 ms) ✕ Should search on YouTube Music (955 ms) ✓ Should retrieve YouTube search suggestions (661 ms) ✓ Should retrieve YouTube Music search suggestions (491 ms) Comments ✓ Should retrieve comments (624 ms) ✓ Should retrieve next batch of comments (353 ms) ✕ Should retrieve comment replies General ✕ Should retrieve playlist with YouTube (672 ms) ✓ Should retrieve home feed (1277 ms) ✓ Should retrieve trending content (999 ms) ✓ Should retrieve video info (1106 ms) ✓ Should download video (2514 ms) * feat: key based type validation for parsers * fix: comments tests pass (10/12) YouTube.js Tests Search ✓ Should search on YouTube (938 ms) ✕ Should search on YouTube Music (850 ms) ✓ Should retrieve YouTube search suggestions (528 ms) ✓ Should retrieve YouTube Music search suggestions (224 ms) Comments ✓ Should retrieve comments (518 ms) ✓ Should retrieve next batch of comments (337 ms) ✓ Should retrieve comment replies (358 ms) General ✕ Should retrieve playlist with YouTube (466 ms) ✓ Should retrieve home feed (1051 ms) ✓ Should retrieve trending content (623 ms) ✓ Should retrieve video info (863 ms) ✓ Should download video (2656 ms) * refactor: type safety checks removing @ts-ignore * fix: playlist tests pass (11/12) YouTube.js Tests Search ✓ Should search on YouTube (991 ms) ✕ Should search on YouTube Music (924 ms) ✓ Should retrieve YouTube search suggestions (606 ms) ✓ Should retrieve YouTube Music search suggestions (225 ms) Comments ✓ Should retrieve comments (393 ms) ✓ Should retrieve next batch of comments (284 ms) ✓ Should retrieve comment replies (252 ms) General ✓ Should retrieve playlist with YouTube (578 ms) ✓ Should retrieve home feed (1148 ms) ✓ Should retrieve trending content (541 ms) ✓ Should retrieve video info (799 ms) ✓ Should download video (1419 ms) * fix: all tests pass for node 🎉 YouTube.js Tests Search ✓ Should search on YouTube (1053 ms) ✓ Should search on YouTube Music (761 ms) ✓ Should retrieve YouTube search suggestions (453 ms) ✓ Should retrieve YouTube Music search suggestions (221 ms) Comments ✓ Should retrieve comments (627 ms) ✓ Should retrieve next batch of comments (412 ms) ✓ Should retrieve comment replies (268 ms) General ✓ Should retrieve playlist with YouTube (565 ms) ✓ Should retrieve home feed (775 ms) ✓ Should retrieve trending content (498 ms) ✓ Should retrieve video info (875 ms) ✓ Should download video (1364 ms) * build: working Deno bundle Still need to test whether this bundle works in the browser * docs: update deno example to download video * refactor: MusicResponsiveListItem to TS * docs: TSDoc for Parser helpers * docs: Parser documentation for TS * docs: add note about parseItem and parseArray * test: remove browser tests since they're identical * feat: browser support and proxy example * fix: PlaylistManager TS after merge * feat: in-browser video streaming * refactor: cleanup the Dash example * feat: allow custom fetch implementations * feat: fetch debugger * fix: OAuth login * refactor: remove file extensions from imports * refactor: build scripts * fix: CustomEvent on node * fix: LiveChat * fix: linting * fix: liniting in build-parser-json * chore: update test workflow * fix: NToken errors after lint fixes * fix: codacy complaints * docs: update to reflect changes Definitly needs more work but its a start * refactor: cleanup imports/exports * fix: browser example - Remove user-agent before making request. - Fix cache on browsers * fix: cache on node * fix: stupid mistake * refactor: Session#signIn to wait untill success This also splits the 'auth' event up into 3 distinct events: - 'auth' -> fired on success - 'auth-pending' -> fired when pending authentication - 'auth-error' -> fired when an error occurred * refactor: freeze Constants * refactor: cleanup HTTPClient Request * refactor: debugFetch readability * chore: lint * refactor: replace jsdoc with tsdoc eslint plugin remove @param annotations without descriptions * fix: bunch of liniting warnings * refactor: better inference on YTNode#is As suggested by @MasterOfBob777 * fix: linting warnings * revert: undici import * refactor: rename `list_type` to `item_type`
LuanRT · Jul 20, 2022 · fb68e6b · fb68e6b
1 parent e2f455d
commit fb68e6b
Show file tree

Hide file tree

Showing 623 changed files with 13,524 additions and 18,553 deletions.
diff --git a/.eslintrc.yml b/.eslintrc.yml
@@ -1,17 +1,19 @@
 plugins:
-  [ jsdoc ]
+  [ '@typescript-eslint', 'eslint-plugin-tsdoc' ]
 env:
   commonjs: true
   es2021: true
   node: true
-extends: [ eslint:recommended, plugin:jsdoc/recommended ]
-globals: 
-  BROWSER: readonly
-settings:
-  jsdoc:
-    mode: 'typescript'
+extends: [ eslint:recommended, 'plugin:@typescript-eslint/recommended' ]
+parser: '@typescript-eslint/parser'
 parserOptions:
   ecmaVersion: latest
+overrides:
+  - 
+    files:
+      - '**/*.js'
+    rules:
+      'tsdoc/syntax': 'off'
 rules:
   max-len:
     - error
@@ -24,12 +26,10 @@ rules:
       ignoreRegExpLiterals: true
 
   quotes: [error, single] 
-
-  jsdoc/newline-after-description: 'off'
-  jsdoc/require-returns-description: 'off'
-  jsdoc/require-param-description: 'off'
-  jsdoc/no-undefined-types: 'off'
-  jsdoc/require-returns: 'off'
+
+  '@typescript-eslint/ban-types': 'off'
+  'tsdoc/syntax': 'warn'
+  '@typescript-eslint/no-explicit-any': 'off'
 
   no-template-curly-in-string: error
   no-unreachable-loop: error
@@ -42,7 +42,7 @@ rules:
   no-implied-eval: error
   arrow-spacing: error
   no-invalid-this: error
-  no-lone-blocks: error
+  no-lone-blocks: 'off'
   no-new-func: error
   no-new-wrappers: error
   no-new: error

diff --git a/.github/workflows/node.js.yml b/.github/workflows/node.js.yml
@@ -13,7 +13,7 @@ jobs:
 
     strategy:
       matrix:
-        node-version: [ 12.x, 14.x, 16.x ]
+        node-version: [ 16.x, 18.x ]
         # See supported Node.js release schedule at https://nodejs.org/en/about/releases/
 
     steps:

diff --git a/.gitignore b/.gitignore
@@ -62,3 +62,11 @@ pnpm-lock.yaml
 
 # Temporary files for testing
 tmp/
+
+# Build output
+dist/
+bundle/*.js.*
+bundle/*.js
+
+# MacOS
+.DS_Store
diff --git a/README_v2.0.0WIP.md b/README_v2.0.0WIP.md
@@ -88,17 +88,19 @@ Innertube is an API used across all YouTube clients, it was created to simplify[
 
 And huge thanks to [@gatecrasher777][gatecrasher] for his research on the workings of the Innertube API!
 
+If you have any questions or need help, feel free to contact us on our chat server [here](https://discord.gg/syDu7Yks54).
+
 <!-- GETTING STARTED -->
 ## Getting Started
 
 ### Prerequisites
-- [NodeJS][nodejs] v14 or greater
+YouTube.js runs on Node.js, Deno and in modern browsers.
 
-To verify things are set up
-properly, run this:
-```bash
-node --version
-```
+It requires a runtime with the following features:
+- [`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API)
+  - On Node we use [undici]()'s fetch implementation which requires Node.js 16.8+. You may provide your own fetch implementation if you need to use an older version. See [providing your own fetch implementation](#custom-fetch) for more information. 
+  - The `Response` object returned by fetch must thus be spec compliant and return a `ReadableStream` object if you want to use the `VideoInfo#download` method. (Implementations like `node-fetch` returns a non-standard `Readable` object.)
+- [`EventTarget`](https://developer.mozilla.org/en-US/docs/Web/API/EventTarget) and [`CustomEvent`](https://developer.mozilla.org/en-US/docs/Web/API/CustomEvent) required.
 
 ### Installation
 - NPM:
@@ -114,15 +116,113 @@ yarn add youtubei.js@latest
 npm install git+https://github.com/LuanRT/YouTube.js.git
 ```
 
+**TODO: Deno install instructions (esm.sh possibly?)**
+
 <!-- USAGE -->
 ## Usage
-
 Create an Innertube instance (or session):
-```js
-// const Innertube = require('youtubei.js');
-import Innertube from 'youtubei.js';
-const youtube = await new Innertube({ gl: 'US' });
+```ts
+// const { Innertube } = require('youtubei.js');
+import { Innertube } from 'youtubei.js';
+const youtube = await Innertube.create();
 ```
+
+## Browser Usage
+To use YouTube.js in the browser you must proxy requests through your own server. You can see our simple reference implementation in Deno in [`examples/browser/proxy/deno.ts`](https://github.com/LuanRT/YouTube.js/tree/main/examples/browser/proxy/deno.ts).
+
+You may provide your own fetch implementation to be used by YouTube.js. Which we will use here to modify and send the requests to through our proxy. See [`examples/browser/web`](https://github.com/LuanRT/YouTube.js/tree/main/examples/browser/web) for an simple example using [Vite](https://vitejs.dev/).
+
+```ts
+// Pre-bundled version for the web
+import { Innertube } from 'youtubei.js/bundle/browser';
+await Innertube.create({
+  fetch: async (input: RequestInfo | URL, init?: RequestInit) => {
+    // Modify the request
+    // and send it to the proxy
+
+    // fetch the url
+    return fetch(request, init);
+  }
+});
+```
+
+### Streaming
+YouTube.js supports streaming of videos in the browser by converting YouTube's streaming data into a MPEG-DASH manifest.
+
+The example below uses [`dash.js`](https://github.com/Dash-Industry-Forum/dash.js) to play the video.
+
+```ts
+import { Innertube } from 'youtubei.js';
+import dashjs from 'dashjs';
+
+const youtube = await Innertube.create({ /* setup - see above */ });
+
+// get the video info
+const videoInfo = await youtube.getInfo('videoId');
+
+// now convert to a dash manifest
+// again - to be able to stream the video in the browser - we must proxy the requests through our own server
+// to do this, we provide a method to transform the urls before writing them to the manifest
+const manifest = videoInfo.toDash(url => {
+  // modify the url
+  // and return it
+  return url;
+});
+
+const uri = "data:application/dash+xml;charset=utf-8;base64," + btoa(manifest);
+
+const videoElement = document.getElementById('video_player');
+
+const player = dashjs.MediaPlayer().create();
+player.initialize(videoElement, uri, true);
+```
+
+Our browser example in [`examples/browser/web`]() provides a full working example.
+
+
+<a name="custom-fetch"></a>
+
+## Providing your own fetch implementation
+You may provide your own fetch implementation to be used by YouTube.js. This may be useful in some cases to modify the requests before they are sent and transform the responses before they are returned (eg. for proxies).
+
+```ts
+// provide a fetch implementation
+const yt = await Innertube.create({
+  fetch: async (input: RequestInfo | URL, init?: RequestInit) => {
+    // make the request with your own fetch implementation
+    // and return the response
+    return new Response(
+      /* ... */
+    );
+  }
+});
+```
+
+<a name="caching"></a>
+
+## Caching
+To improve performance, you may wish to cache the transformed player instance which we use to decode the streaming urls.
+
+Our cache uses the `node:fs` module in Node-like environments, `Deno.writeFile` in Deno and `indexedDB` in browsers.
+
+```ts
+import { Innertube, UniversalCache } from 'youtubei.js';
+// By default, cache stores files in the OS temp directory (or indexedDB in browsers).
+const yt = await Innertube.create({
+  cache: new UniversalCache()
+});
+
+// You may wish to make the cache persistent (on Node and Deno)
+const yt = await Innertube.create({
+  cache: new UniversalCache(
+    // Enables persistent caching
+    true, 
+    // Path to the cache directory, will create the directory if it doesn't exist
+    './.cache' 
+  )
+});
+```
+
 ## API
 
 ## Innertube : `object`

diff --git a/browser.ts b/browser.ts
@@ -0,0 +1,12 @@
+// Deno and browser runtimes
+
+// Polyfill buffer
+import { Buffer } from 'buffer';
+if (!Reflect.has(globalThis, 'Buffer')) {
+  Reflect.set(globalThis, 'Buffer', Buffer);
+}
+
+import Innertube from './lib/Innertube';
+export { default as Innertube } from './lib/Innertube.js';
+export * from './lib/utils';
+export default Innertube;
diff --git a/build/browser.d.ts b/build/browser.d.ts