From 4d3743f955bb25130aa533611d42405ca5baf301 Mon Sep 17 00:00:00 2001 From: Son Roy Almerol Date: Sun, 17 Nov 2024 13:39:25 -0500 Subject: [PATCH 1/3] Update README.md --- README.md | 79 ++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 58 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index ef06324..dd76cdd 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Streamline your IPTV experience by consolidating multiple M3U playlists into a s Uses the channel title or `tvg-name` (as fallback) to merge multiple identical channels into one. This is not an xTeVe/Threadfin replacement but is often used with it. > [!IMPORTANT] -> Starting `0.16.0`, Redis is **removed** as a dependency. There will be **no** databases required for the proxy from this version moving forward. +> Redis has been **removed** as a dependency starting `0.16.0`. The proxy should now be (mostly) **stateless**. > Migrating to `0.16.0` is as easy as removing the Redis container from your compose file. > Due to a major change on how data is being processed, any Redis persistence cannot be migrated over and a sync from the original M3U sources will be required. @@ -18,15 +18,18 @@ Uses the channel title or `tvg-name` (as fallback) to merge multiple identical c 1. **Initialization and M3U Playlist Consolidation:** - The service loads M3U playlists from specified URLs, consolidating streams into `/playlist.m3u`. - - The consolidation process merges streams based on their names and saves them in a database. + - The consolidation process merges streams based on their names and saves them into one M3U file. - Each unique stream name aggregates corresponding URLs into the consolidated playlist. 2. **HTTP Endpoints:** - **Playlist Endpoint (`/playlist.m3u`):** - Access the merged M3U playlist containing streams from different sources. - - **Stream Endpoint (`/stream/{streamID}.{fileExt}`):** + - **Stream Endpoint (`/p/{originalBasePath}/{streamToken}.{fileExt}`):** - Request video streams for specific stream IDs. + - `originalBasePath`: Parsed from one of the original source. This is to prevent clients to miscategorize the stream due to a missing keyword (e.g. live, vod, etc.). + - `streamToken`: An encoded string that contains the stream title and an array of the original stream URLs associated with the stream title. This token allows the proxy to be **stateless** as the M3U itself is the "database". + - `fileExt`: Parsed file extension from one of the original source. 3. **Load Balancing:** - The service employs load balancing by cycling through available stream URLs. @@ -78,52 +81,86 @@ services: Access the generated M3U playlist at `http://:8080/playlist.m3u`. -## Configuration +## Environment Variable Configurations > [!NOTE] > This configuration list only applies to the latest release version. > To see the README of a specific version, navigate to the specific tag of the desired version (e.g. [`0.10.0`](https://github.com/sonroyaalmerol/m3u-stream-merger-proxy/tree/0.10.0)). +> [!TIP] +> For variables needing Go regexp (regular expression) values, make sure to [build the regexp](https://regex101.com/) with the Golang flavor. +> If you only need to filter out a specific substring, then putting in the substring itself in those variables should work just fine. + +### Container Configs | ENV VAR | Description | Default Value | Possible Values | |-----------------------------|----------------------------------------------------------|---------------|------------------------------------------------| | PORT | Set listening port of service inside the container. | 8080 | Any valid port | | PUID | Set UID of user running the container. | 1000 | Any valid UID | | PGID | Set GID of user running the container. | 1000 | Any valid GID | | TZ | Set timezone | Etc/UTC | [TZ Identifiers](https://nodatime.org/TimeZones) | + +### Playlist Source Configs +| ENV VAR | Description | Default Value | Possible Values | +|-----------------------------|----------------------------------------------------------|---------------|------------------------------------------------| | M3U_URL_1, M3U_URL_2, M3U_URL_X | Set M3U URLs as environment variables. | N/A | Any valid M3U URLs | -| M3U_MAX_CONCURRENCY_1, M3U_MAX_CONCURRENCY_2, M3U_MAX_CONCURRENCY_X | Set max concurrency. | 1 | Any integer | +| M3U_MAX_CONCURRENCY_1, M3U_MAX_CONCURRENCY_2, M3U_MAX_CONCURRENCY_X | Set max concurrency. The "X" should match the M3U URL. | 1 | Any integer | +| USER_AGENT | Set the User-Agent of HTTP requests. | IPTV Smarters/1.0.3 (iPad; iOS 16.6.1; Scale/2.00) | Any valid user agent | +| SYNC_CRON | Set cron schedule expression of the background updates. | 0 0 * * * | Any valid cron expression | +| SYNC_ON_BOOT | Set if an initial background syncing will be executed on boot | true | true/false | +| CACHE_ON_SYNC | Set if an initial background cache building will be executed after sync. Requires BASE_URL to be set. | false | true/false | +| CLEAR_ON_BOOT | Set if an initial database clearing will be executed on boot | false | true/false | + +### Load Balancer Configs +| ENV VAR | Description | Default Value | Possible Values | +|-----------------------------|----------------------------------------------------------|---------------|------------------------------------------------| | MAX_RETRIES | Set max number of retries (loop) across all M3Us while streaming. 0 to never stop retrying (beware of throttling from provider). | 5 | Any integer greater than or equal 0 | | RETRY_WAIT | Set a wait time before retrying (looping) across all M3Us on stream initialization error. | 0 | Any integer greater than or equal 0 | | STREAM_TIMEOUT | Set timeout duration in seconds of retrying on error before a stream is considered down. | 3 | Any positive integer greater than 0 | +| BUFFER_MB | Set buffer size in mb. **This is not a shared buffer (for now).** | 0 (no buffer) | Any positive integer | + +### Playlist Output (`/playlist.m3u`) Configs +> [!NOTE] +> Filter configs (e.g. `INCLUDE_GROUPS_X`, `EXCLUDE_GROUPS_X`, `INCLUDE_TITLE_X`, `EXCLUDE_TITLE_X`) only applies **every sync** from source. +> Changes in the values will not reflect immediately unless the cache is cleared which forces the sync to trigger. +> Also, the `X` values on these env vars are **not associated** with the `X` values of the M3U URLs. They are simply a way for you to use multiple filters for each. + +| ENV VAR | Description | Default Value | Possible Values | +|-----------------------------|----------------------------------------------------------|---------------|------------------------------------------------| +| BASE_URL | Sets the base URL for the stream URls in the M3U file to be generated. | http/s:// (e.g. ) | Any string that follows the URL format | | SORTING_KEY | Set tag to be used for sorting the stream list | tvg-id | tvg-id, tvg-chno | -| USER_AGENT | Set the User-Agent of HTTP requests. | IPTV Smarters/1.0.3 (iPad; iOS 16.6.1; Scale/2.00) | Any valid user agent | -| PARSER_WORKERS | Set number of workers to spawn for M3U parsing. | 5 | Any positive integer | -| BUFFER_MB | Set buffer size in mb. | 0 (no buffer) | Any positive integer | | INCLUDE_GROUPS_1, INCLUDE_GROUPS_2, INCLUDE_GROUPS_X | Set channels to include based on groups (Takes precedence over EXCLUDE_GROUPS_X) | N/A | Go regexp | | EXCLUDE_GROUPS_1, EXCLUDE_GROUPS_2, EXCLUDE_GROUPS_X | Set channels to exclude based on groups | N/A | Go regexp | | INCLUDE_TITLE_1, INCLUDE_TITLE_2, INCLUDE_TITLE_X | Set channels to include based on title (Takes precedence over EXCLUDE_TITLE_X) | N/A | Go regexp | | EXCLUDE_TITLE_1, EXCLUDE_TITLE_2, EXCLUDE_TITLE_X | Set channels to exclude based on title | N/A | Go regexp | -| TITLE_SUBSTR_FILTER | Sets a regex pattern used to exclude substrings from channel titles | none | Go regexp | -| BASE_URL | Sets the base URL for the stream URls in the M3U file to be generated. | http/s:// (e.g. ) | Any string that follows the URL format | -| SYNC_CRON | Set cron schedule expression of the background updates. | 0 0 * * * | Any valid cron expression | -| SYNC_ON_BOOT | Set if an initial background syncing will be executed on boot | true | true/false | -| CACHE_ON_SYNC | Set if an initial background cache building will be executed after sync. Requires BASE_URL to be set. | false | true/false | -| CLEAR_ON_BOOT | Set if an initial database clearing will be executed on boot | false | true/false | +| TITLE_SUBSTR_FILTER | Sets a regex pattern used to exclude substrings from channel titles. This modifies the title of the streams when rendered in `/playlist.m3u`. | none | Go regexp | + +### Logging Configs +| ENV VAR | Description | Default Value | Possible Values | +|-----------------------------|----------------------------------------------------------|---------------|------------------------------------------------| | DEBUG | Set if verbose logging is enabled | false | true/false | | SAFE_LOGS | Set if sensitive info are removed from logs. Always enable this if submitting a log publicly. | false | true/false | +## Sponsors ✨ +Huge thanks to those who donated for the development of this project! -## Endpoints +

+ +

-- `/playlist.m3u`: Get the merged M3U playlist. +

+kpirnie  jbumgard  aniel300   +

-- `/stream/{streamID}.{fileExt}`: Stream video content for specified stream IDs. +## Contributors ✨ -## Sponsors + + + -

- -

+ + + + ## Contributing From 7d60f27844fa3fb05af961be72e783866a4b8819 Mon Sep 17 00:00:00 2001 From: Son Roy Almerol Date: Sun, 17 Nov 2024 13:40:57 -0500 Subject: [PATCH 2/3] Update pr-publish.yml --- .github/workflows/pr-publish.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/pr-publish.yml b/.github/workflows/pr-publish.yml index d286e20..becf059 100644 --- a/.github/workflows/pr-publish.yml +++ b/.github/workflows/pr-publish.yml @@ -5,6 +5,8 @@ on: - synchronize - opened - reopened + paths-ignore: + - '**/README.md' jobs: fetch-pr-details: From f8eb773795a089a66e52bf62450110871fb35d25 Mon Sep 17 00:00:00 2001 From: Son Roy Almerol Date: Sun, 17 Nov 2024 13:45:42 -0500 Subject: [PATCH 3/3] Update pr-publish.yml --- .github/workflows/pr-publish.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pr-publish.yml b/.github/workflows/pr-publish.yml index becf059..a114d46 100644 --- a/.github/workflows/pr-publish.yml +++ b/.github/workflows/pr-publish.yml @@ -5,8 +5,12 @@ on: - synchronize - opened - reopened - paths-ignore: - - '**/README.md' + paths: + - '**.go' + - 'Dockerfile' + - '**.sh' + - 'go.mod' + - 'go.sum' jobs: fetch-pr-details: