Owez · Owez · Jan 26, 2023 · Jan 26, 2023 · Jan 26, 2023 · Jan 26, 2023
diff --git a/.gitignore b/.gitignore
@@ -1,7 +1,16 @@
-.vscode
+# Editors/IDEs
+.vscode/
+.idea/
+
+# Yark archives
 demo/
 foobar/
-**/.DS*
-build/
-dist/
-__pycache__/
+
+# NixOS
+/.direnv
+
+# env
+.env
+
+# MacOS
+.DS*
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -5,11 +5,14 @@
   - [Don't know coding?](#dont-know-coding)
   - [Want to code?](#want-to-code)
 - [Internal Information](#internal-information)
+  - [Setting up development](#setting-up-development)
   - [Ideology](#ideology)
     - [Users](#users)
     - [KISS](#kiss)
     - [Dependencies](#dependencies)
+  - [Structure](#structure)
   - [Conventions](#conventions)
+    - [Ports](#ports)
     - [Branches](#branches)
     - [To/from archives](#tofrom-archives)
 
@@ -32,6 +35,27 @@ Please make sure you describe the change so any developer could understand it
 
 This section goes through some of the more complex internal information which will be useful if your making changes inside of Yark in a PR.
 
+## Setting up development
+
+Setting up a full development enviroment for Yark is made easy thanks to Makefiles. To setup your development enviroment, first make sure these three packages are installed:
+
+- Make ([Tutorial](https://www.gnu.org/software/make/#download); `apt install make`)
+- NPM ([Tutorial](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm/))
+- Python 3.11 ([Tutorial](https://www.python.org/downloads/); `apt install python3.11`)
+- Poetry ([Tutorial](https://python-poetry.org/docs/#installation); `pip3.11 install poetry`)
+
+If you install what you don't have quickly, it should take about 5-10 mins. You now have everything you need to develop except for some setup for the API; please read it's guide. Once that's done, you can start the programs you want to develop for:
+
+- To develop the API, do the [setup guide](./yark-api/README.md), then: `cd yark-api && make dev`
+- To develop the GUI, run the API and then `cd yark-pages && make dev`
+- To develop the library you don't have to run anything!
+
+Basically the top-level directory has a Makefile in it, and each specific project also has it's own specific Makefile. These files install all dependencies for you and run a nice development server if `dev` is available for you to use.
+
+This is the section to link new developers to so that they can setup their machine.
+
+[^extra]: You might need to run `cd yark-api && make build` beforehand to make sure an API build is ready
+
 ## Ideology
 
 ### Users
@@ -52,8 +76,32 @@ Dependencies inside of the actual Yark code are a bit of a different story. If a
 
 [^depsec]: Every dependency is a new [attack vector](https://en.wikipedia.org/wiki/Attack_vector) waiting to be exploited
 
+## Structure
+
+Yark is separated into three areas of concern:
+
+1. The core `yark` library which contains the underlying archiving logic
+   - *This needs Python + Poetry to develop on*
+2. The `yark-pages` project which contains the typical GUI users use
+   - *This needs NPM + Python + Poetry to develop on*
+3. The `yark-api` project which contains the behind-the-scenes API
+   - *This needs Python + Poetry to develop on*
+4. The `yark-cli` project which contains an old CLI interface
+   - *This needs Python + Poetry to develop on*
+
+When Yark is built into the app, it uses the `yark` library so all the logic works and builds the webpages from `yark-pages` for the [SvelteKit](https://kit.svelte.dev/)-based GUI. The app itself contains an API which these webpages connect to.
+
 ## Conventions
 
+### Ports
+
+Yark uses some pre-defined ports that we assume certain things to *probably* be running on. Everything ideally in development and production should be ran on the following ports:
+
+- GUI development server: `5173`[^guidev]
+- API: `7666`
+
+[^guidev]: This is SvelteKit's default port, this isn't used in production so it's fine
+
 ### Branches
 
 The convention for branches/tags are:
@@ -93,4 +141,4 @@ Whereas the `_b` and `_ib` ones would be used for a JSON child object which woul
 
 These two are used because the full objects (the first example) is easier to implement and are more self-contained, but we often need to select items by their ID straight from their parents.
 
-This system will be changed to dataclasses soon because it's not a good way to this. But for now this is what the archival conversion system looks like.
+This system will be changed to dataclasses soon (see [#107](https://github.com/Owez/yark/pull/107)) because it's not a good way to this. But for now this is what the archival conversion system looks like.
diff --git a/Makefile b/Makefile
@@ -0,0 +1,4 @@
+fmt:
+	$(MAKE) -C yark fmt
+	$(MAKE) -C yark-api fmt
+	$(MAKE) -C yark-pages fmt
diff --git a/README.md b/README.md
@@ -1,66 +1,18 @@
-<!-- TODO: logo; #2 <https://github.com/Owez/yark/issues/2> -->
-<!-- TODO: add when logos done; #2 <https://github.com/Owez/yark/issues/2>: <h1 align="center">yark</h1> -->
-
 # Yark
 
 YouTube archiving made simple.
 
 [Installation](#installation) · [Managing your Archive](#managing-your-archive) · [Viewing your Archive](#viewing-your-archive)
 
-Yark lets you continuously archive all videos and metadata for YouTube channels and playlists. You can also view your archive as a seamless offline website ✨
-
 ## Installation
 
-To install Yark, simply download [Python 3.9+](https://www.python.org/downloads/) and [FFmpeg](https://ffmpeg.org/) (optional), then run the following:
-
-```shell
-$ pip3 install yark
-```
-
-## Managing your Archive
-
-Once you've installed Yark, think of a name for your archive (e.g., foobar) and copy the channel/playlist url:
-
-```shell
-$ yark new foobar https://www.youtube.com/channel/UCSMdm6bUYIBN0KfS2CVuEPA
-```
-
-Now that you've created the archive, you can tell Yark to download all videos and metadata using the refresh command:
-
-```shell
-$ yark refresh foobar
-```
-
-Once everything has been downloaded, Yark will automatically give you a status report of what's changed since the last refresh:
-
-<p><img src="examples/image/../images/cli_dark.png" alt="Report Demo" title="Report Demo" width="600" /></p>
-
-## Viewing your Archive
-
-Viewing you archive is easy, just type `view` with your archives name:
-
-```shell
-$ yark view foobar
-```
-
-This will pop up an offline website in your browser letting you watch all videos 🚀
-
-<p><img src="https://raw.githubusercontent.com/Owez/yark/master/examples/images/viewer_light.png" alt="Viewer Demo" title="Viewer Demo" width=650 /></p>
-
-Under each video is a rich history report filled with timelines and graphs, as well as a noting feature which lets you add timestamped and permalinked comments 👐
-
-<p><img src="https://raw.githubusercontent.com/Owez/yark/master/examples/images/viewer_stats_light.png" alt="Viewer Demo – Stats" title="Viewer Demo – Stats" width=650 /></p>
-
-Light and dark modes are both available and automatically apply based on the system's theme.
+Yark lets you continuously archive all videos and metadata for YouTube channels and playlists. You can also view your archive as a seamless offline website ✨
 
-## Roadmap
+TODO: new installation steps, also link to self-hosting
 
-Here's the general roadmap for Yark going a few months into the future for the upcoming features and changes:
+## Showcase
 
-1. Yark 1.3 (Current): [Comments](https://github.com/Owez/yark/issues/50), [viewer tabs](https://github.com/Owez/yark/issues/30), [playlists](https://github.com/Owez/yark/issues/27), [viewed](https://github.com/Owez/yark/issues/51)
-2. Yark 1.4 (Expected ??): [Run command](https://github.com/Owez/yark/issues/83), [better homepage](https://github.com/Owez/yark/issues/65), [more graphs](https://github.com/Owez/yark/issues/74), [translations](https://github.com/Owez/yark/issues/73)
-3. Yark 1.5 (Expected ??): Full [GUI control](https://github.com/Owez/yark/issues/86) for archives and viewer polish/rewrite
-4. Onwards: Not sure yet, [create an issue](https://github.com/Owez/yark/issues/new) to help out :) 
+TODO: new viewer showcase
 
 ## Details
 

diff --git a/examples/Flow.drawio b/examples/Flow.drawio
@@ -0,0 +1 @@
+<mxfile host="app.diagrams.net" modified="2023-01-25T11:05:36.302Z" agent="5.0 (Macintosh)" etag="JIiy6rqfE_XDCuMq9ATC" version="20.8.11" type="device"><diagram name="Page-1" id="tkwtmS5oVSkWOoKScORv">7Z3fc6M2EMf/Gk+f2jEIsPN4cXJJZ5LOzWXmenm6kUEGGkCuLP/qX18JRAwRjrnDeLHDU2CRMHyXj9AuCxmgSby5Y3gePFKPRANz6G0G6GZgmqZhDsUfadlmFsMwR5nFZ6GnbDvDU/gfUUbV0V+GHlmUGnJKIx7Oy0aXJglxecmGGaPrcrMZjcq/Osc+0QxPLo5069+hx4PMOs7PQtrvSegH+S8bzlW2JcZ5Y3UmiwB7dF0wodsBmjBKebYUbyYkkurlumT9Pu/Z+npgjCS8Tgfrm4k39Hpr/7Du//18Hyx+rNDvpjraFY6W6ozV0fJtLoHP6HKumhHGyaZKeDzNmw/1AzNeT1dcKITGhLOtaJLv6AplXfJrZKh2sd4Jbo6VLSiIbedGrJzsv+57p4NYUFL8hCyGpsoDxZ68fhiN5Z9QnO1bnYRMiUfkbocDdL0OQk6e5tiVW9eCDmELeCwO48YQi2IP0YRGlKV9kYfJeOYK+4Iz+kIKWxx3TKYzsWWvB4pKv+NkXX+l91iX+5XbotxOW2qbmto3dJ1EmeLiJ7CHORaL04hOL0Z1w4GW3dG0JJ4Y+9QqZTygPk1wdLuzXpfV3rV5oHSuNP6HcL5VAzleclr2ANmE/Lvs/oet1p4LW242as/pyjZfScTpFjrJ1efitl23dC3vl52fPKlf8ZxQhi6ZSw4PExwzn7y7w+orgZEI83BVPrqjuxlpdE0wJz5l4YKU+QoT4S1ziC6GMdOCZmzUM9aQMbMmYwiSMUtj7Atmb/GaiQtfSIXd4GIAQyY0YOMesIaAoZqAWZCA2RpgT3hF0mDw0qbjVkX4c1qmrjQte6Z+jimrJlM2JFN5iqUA1SMRByyDejVDJIujYkUMzyajKqyunBHCTitY5bvJcelM/GUY58dZibIddFCc2TU5M4bV18aJQNMTHJMlE4gJ23IRJn6aLtyE8TK+XN7gYzHDOj/eOnZfy28Zh4EDTXkYes5jl1GcpMf/QtKEh5RrwwkTHhWLEfV9wi4WQfhozbB7BJsiWDchYoBmRAxHR5BEhAs/SvdjvlzIXczkb8tmoUdkJOfSZeQlv3GxOJWEegpb2e1CqYSP94w+idKYyrpZlIwLMCpHGpWt5lE6xRn4c06jT6w05sypy9kIkjPzDEP7jnnarB1qgMb2pp5Ea7lUpEtjKviQasLG9AXMdtCdG2i1AwpQzkAjR+NDOdqEHVH1bOlXMmNkEQjjJ+YG8ggaDaitDY/ww6EedT+9hPNCIC0WaRJtNQGFFLysUvn2ktCEvLkXKROOQj8Rq66Qjgj7tRQ2dHH0SW2IQ89LcaxyS9lxM5rwHGnj+G7KI4Grmn5CrfmpIg7L/FSoF/m4ftoz27BtaL85Zj/deI1tDj6cq/t0DoHebxy93rH3aWOfmqAPgBzQsKAjs8XjcwqaU3H6CKAFn1qwZX16Xd8zZi8nmN6//0oSsq3SzAP8wamtT+w1kdp+T8sZHX5Pq1IVqzVZLL3w+iuZC8zhQ0V7z4MesEvI0lk755fa9unblcxkHqv0jwD2jWWHq2jrltFaoHW0VtXY7EQyTPfClVj05eInT6IWytibLHha9Je1Eb9ZaFbRM62NkNUS6TPa6dJ9EWKoZlP2tuPb3V0Iz/Dlunb/TK8x0LVnpaBVEpaenTsq0AIfguOPTjR8QbANm827CKLr1mNYoLkDa9wy0YG4ED460PDlxTZsKvcigB7VBXoMCrT+3Zo7khD1Ug1TOYALAQu+QtjuX51pDNa4Llh7HoqeKCNbNfc9caLRNn4x0YiGbQGAdFm+hWTdgTTjnnkV2FCB9KnWWacZ9+jblTSj1WclDoxkhz+AUXfKg0CnPEif8jzgZeKmZXFf/hzIgiH5uwnlgXyJccgDRiR4F4IafAbQ6vMFjVmrOwtCoLOgPDVVxdpK3PeP/JIwJFbwaTirj9obY3VVEysLtNTOrkrDnTi4QN2LLfTB5q9OhBZ7pjtwoYX+Wd47mWcdJjiWaZ+Beb1j4HzG5z0ydyXCQH3u58Bwdnh4rvsuIwJ9xwqZGl+P2YdSSDzn8jjwMUakDsEFH1Og/osojemq+xnm7PoGo0v/KNH5fsPyAFfwQQXqv27emCuzLlctfWlIrO7+PUa6rfBfRtDt/w==</diagram></mxfile>
diff --git a/examples/images/cli_dark.png b/examples/images/cli_dark.png
diff --git a/examples/images/viewer_light.png b/examples/images/viewer_light.png
diff --git a/examples/images/viewer_stats_light.png b/examples/images/viewer_stats_light.png
diff --git a/pyproject.toml b/pyproject.toml
diff --git a/yark-api/.gitignore b/yark-api/.gitignore
@@ -0,0 +1,160 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/