Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major rework: removing all database dependencies #187

Merged
merged 49 commits into from
Nov 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
4274a71
removing database dependencies
sonroyaalmerol Nov 17, 2024
b8eb391
fix test source index + lint errors
sonroyaalmerol Nov 17, 2024
8061320
remove unnecessary remnants
sonroyaalmerol Nov 17, 2024
2623f0e
use channel to avoid deadlock
sonroyaalmerol Nov 17, 2024
73a8e6b
fix m3u generate and add more revalidating logs
sonroyaalmerol Nov 17, 2024
e352228
deal with cache lock on generate
sonroyaalmerol Nov 17, 2024
01c638a
use base64 and fix sorting
sonroyaalmerol Nov 17, 2024
6f42b94
fix base64 decoding
sonroyaalmerol Nov 17, 2024
ba295ca
use datadog zstd
sonroyaalmerol Nov 17, 2024
9d72211
enable cgo on build
sonroyaalmerol Nov 17, 2024
20866ee
ignore 3018
sonroyaalmerol Nov 17, 2024
60b96da
add logs to zstd compression
sonroyaalmerol Nov 17, 2024
84ada3f
publish every pr commit workflow
sonroyaalmerol Nov 17, 2024
a6930d6
use writer and reader for zstd
sonroyaalmerol Nov 17, 2024
6abbcf3
add revalidating wait option
sonroyaalmerol Nov 17, 2024
37ac381
use memory mapping for performance
sonroyaalmerol Nov 17, 2024
10a652e
fix lint errcheck
sonroyaalmerol Nov 17, 2024
e49c2f2
rework revalidation wait
sonroyaalmerol Nov 17, 2024
971a67f
fix log for wait revalidation
sonroyaalmerol Nov 17, 2024
388d45d
remove revalidation flag and rely on mutex
sonroyaalmerol Nov 17, 2024
01f135e
move download sources to tmp and cache to data
sonroyaalmerol Nov 17, 2024
c44bcdc
only use title and urls in json + readme edit
sonroyaalmerol Nov 17, 2024
031b598
fix readme
sonroyaalmerol Nov 17, 2024
f3effff
separate goroutines for each m3u index
sonroyaalmerol Nov 17, 2024
a6fb94e
prevent blocking http access while loading proxy
sonroyaalmerol Nov 17, 2024
fde2bef
fix blocking revalidation
sonroyaalmerol Nov 17, 2024
4a54c11
fix blocking test
sonroyaalmerol Nov 17, 2024
6254f9c
only generate slugs on url generate
sonroyaalmerol Nov 17, 2024
4319637
use channels for m3u generate
sonroyaalmerol Nov 17, 2024
b5bdd80
use fprint instead
sonroyaalmerol Nov 17, 2024
1cd8400
use http writer response
sonroyaalmerol Nov 17, 2024
ed63e49
use performant zstd
sonroyaalmerol Nov 17, 2024
c2a0944
add read lock to revalidation
sonroyaalmerol Nov 17, 2024
8aa243c
fix cache on sync
sonroyaalmerol Nov 17, 2024
79936e6
add try rlock and read cache
sonroyaalmerol Nov 17, 2024
c55fc17
separate goroutine for cache to file
sonroyaalmerol Nov 17, 2024
8715983
fix linting error
sonroyaalmerol Nov 17, 2024
20b15fc
fix blocking issue
sonroyaalmerol Nov 17, 2024
9ac0eed
close datachan on file read
sonroyaalmerol Nov 17, 2024
ea4fe2d
use pointers for egress chan
sonroyaalmerol Nov 17, 2024
198450a
remove trylock
sonroyaalmerol Nov 17, 2024
d9a1193
fix mutex lock for write
sonroyaalmerol Nov 17, 2024
ff26ea2
fix #EXTM3U header
sonroyaalmerol Nov 17, 2024
cb18cce
fix ext header for cache from file
sonroyaalmerol Nov 17, 2024
e2bed72
fix write cache to file
sonroyaalmerol Nov 17, 2024
385d6c6
revert back to non-channel handling
sonroyaalmerol Nov 17, 2024
144e74f
shorter subpath
sonroyaalmerol Nov 17, 2024
f22ccfb
add log on finish m3u generate
sonroyaalmerol Nov 17, 2024
18b673c
use RWMutex for concurrenct
sonroyaalmerol Nov 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 0 additions & 32 deletions .github/workflows/pr-build.yml

This file was deleted.

23 changes: 9 additions & 14 deletions .github/workflows/pr-publish.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
name: Pull Request Publish
on:
issue_comment:
pull_request:
types:
- created
- synchronize
- opened
- reopened

jobs:
fetch-pr-details:
if: ${{ github.event.comment.body == '/publish' && github.event.issue.pull_request }}
runs-on: ubuntu-latest
outputs:
sha: ${{ steps.get-sha.outputs.result }}
Expand All @@ -17,18 +18,12 @@ jobs:
with:
result-encoding: string
script: |
const prNumber = context.issue.number;
const response = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: prNumber
});
const sha = response.data.head.sha;
const prNumber = context.payload.pull_request.number;
const sha = context.payload.pull_request.head.sha;
return sha;

build-and-push:
needs: fetch-pr-details
if: ${{ github.event.comment.body == '/publish' && github.event.issue.pull_request }}
runs-on: ubuntu-latest
steps:
- name: Checkout
Expand All @@ -53,7 +48,7 @@ jobs:
with:
images: |
ghcr.io/${{ github.repository }}
tags: type=raw,value=pr-${{ github.event.issue.number }}
tags: type=raw,value=pr-${{ github.event.pull_request.number }}
flavor: latest=false

- name: Docker - Build and Push
Expand All @@ -69,8 +64,8 @@ jobs:
with:
header: Built and pushed Docker image for PR
recreate: true
number: ${{ github.event.issue.number }}
number: ${{ github.event.pull_request.number }}
message: |
The Docker image for this pull request has been built and pushed to GHCR.

Image URL: `ghcr.io/${{ github.repository }}:pr-${{ github.event.issue.number }}`
Image URL: `ghcr.io/${{ github.repository }}:pr-${{ github.event.pull_request.number }}`
12 changes: 2 additions & 10 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,8 @@ RUN go mod download
# Copy the source code from the current directory to the Working Directory inside the container
COPY . .

# fire up redis server and test and build the app.
# hadolint ignore=DL3018
RUN \
if [ "$(uname -m)" = "x86_64" ]; then \
apk --no-cache add redis && \
sed -i "s/bind .*/bind 127.0.0.1/g" /etc/redis.conf && \
redis-server --daemonize yes && \
go test ./...; \
fi && \
go build -ldflags='-s -w' -o m3u-proxy .
# test and build the app.
RUN go test ./tests/... && go build -ldflags='-s -w' -o m3u-proxy .

# End from the latest alpine image
# hadolint ignore=DL3007
Expand Down
29 changes: 9 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ Streamline your IPTV experience by consolidating multiple M3U playlists into a s
Uses the channel title or `tvg-name` (as fallback) to merge multiple identical channels into one. This is not an xTeVe/Threadfin replacement but is often used with it.

> [!IMPORTANT]
> All versions after `0.10.0` will require an external Redis/Valkey instance. The SQLite database within the data folder will not be used going forward. For data persistence, refer to the [Redis](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/) docs. The sample `docker-compose.yml` below has also been modified to include Redis.
> To see the README of a specific version, navigate to the specific tag of the desired version (e.g. [`0.10.0`](https://github.com/sonroyaalmerol/m3u-stream-merger-proxy/tree/0.10.0)).
> Starting `0.16.0`, Redis is **removed** as a dependency. There will be **no** databases required for the proxy from this version moving forward.
> Migrating to `0.16.0` is as easy as removing the Redis container from your compose file.
> Due to a major change on how data is being processed, any Redis persistence cannot be migrated over and a sync from the original M3U sources will be required.

## How It Works

Expand Down Expand Up @@ -53,7 +54,6 @@ Deploy with ease using the provided `docker-compose.yml`:

```yaml

version: '3'
services:
m3u-stream-merger-proxy:
image: sonroyaalmerol/m3u-stream-merger-proxy:latest
Expand All @@ -72,26 +72,19 @@ services:
- M3U_MAX_CONCURRENCY_2=1
- M3U_URL_X=
restart: always
depends_on:
- redis
redis:
image: redis
restart: always
healthcheck:
test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
interval: 1s
timeout: 3s
retries: 5
# Redis persistence is OPTIONAL. This will allow you to reuse the database across restarts.
# command: redis-server --save 60 1
# [OPTIONAL] Cache persistence: This will allow you to reuse the M3U cache across container recreates.
# volumes:
# - ./data:/data
# - ./data:/m3u-proxy/data
```

Access the generated M3U playlist at `http://<server ip>:8080/playlist.m3u`.

## Configuration

> [!NOTE]
> This configuration list only applies to the latest release version.
> To see the README of a specific version, navigate to the specific tag of the desired version (e.g. [`0.10.0`](https://github.com/sonroyaalmerol/m3u-stream-merger-proxy/tree/0.10.0)).

| ENV VAR | Description | Default Value | Possible Values |
|-----------------------------|----------------------------------------------------------|---------------|------------------------------------------------|
| PORT | Set listening port of service inside the container. | 8080 | Any valid port |
Expand All @@ -103,12 +96,8 @@ Access the generated M3U playlist at `http://<server ip>:8080/playlist.m3u`.
| MAX_RETRIES | Set max number of retries (loop) across all M3Us while streaming. 0 to never stop retrying (beware of throttling from provider). | 5 | Any integer greater than or equal 0 |
| RETRY_WAIT | Set a wait time before retrying (looping) across all M3Us on stream initialization error. | 0 | Any integer greater than or equal 0 |
| STREAM_TIMEOUT | Set timeout duration in seconds of retrying on error before a stream is considered down. | 3 | Any positive integer greater than 0 |
| REDIS_ADDR | Set Redis server address | N/A | e.g. localhost:6379 |
| REDIS_PASS | Set Redis server password | N/A | Any string |
| REDIS_DB | Set Redis server database to be used | 0 | 0 to 15 |
| SORTING_KEY | Set tag to be used for sorting the stream list | tvg-id | tvg-id, tvg-chno |
| USER_AGENT | Set the User-Agent of HTTP requests. | IPTV Smarters/1.0.3 (iPad; iOS 16.6.1; Scale/2.00) | Any valid user agent |
| ~~LOAD_BALANCING_MODE~~ (removed on version 0.10.0) | Set load balancing algorithm to a specific mode | brute-force | brute-force/round-robin |
| PARSER_WORKERS | Set number of workers to spawn for M3U parsing. | 5 | Any positive integer |
| BUFFER_MB | Set buffer size in mb. | 0 (no buffer) | Any positive integer |
| INCLUDE_GROUPS_1, INCLUDE_GROUPS_2, INCLUDE_GROUPS_X | Set channels to include based on groups (Takes precedence over EXCLUDE_GROUPS_X) | N/A | Go regexp |
Expand Down
56 changes: 0 additions & 56 deletions database/concurrency_utils.go

This file was deleted.

100 changes: 0 additions & 100 deletions database/database_test.go

This file was deleted.

Loading
Loading