Skip to content

Commit

Permalink
README: Update readme! Add code coverage reports.
Browse files Browse the repository at this point in the history
  • Loading branch information
@hjpotter92
hjpotter92 committed Aug 24, 2022
1 parent 0b5ed92 commit 97ee917
Show file tree
Hide file tree
Showing 2 changed files with 142 additions and 96 deletions.
11 changes: 9 additions & 2 deletions .github/workflows/test.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }}

- name: Restore go modules cache
id: cache-go-mod
uses: actions/cache@v3
with:
path: |
Expand All @@ -40,5 +41,11 @@ jobs:

- name: Trigger test suite
run: |
# Skip orch-tester, because we not released as a binary
go test $(go list ./... | grep -v 'orch-tester') -v
go test ./... --short -v --covermode=atomic --coverprofile=coverage.out
- name: Upload coverage reports
uses: codecov/codecov-action@v3
with:
files: ./coverage.out
name: ${{ github.event.repository.name }}
verbose: true
227 changes: 133 additions & 94 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,168 +1,205 @@
# Stream tester

Stream tester is a tool to test Livepeer network performance and stability.
It sends RTMP streams into broadcaster nodes and download back transcoded data and calculates success rate.
It can be used as standalone command line tool or as part of [test harness](https://github.com/livepeer/test-harness).
As part of test harness, it works in server mode, inside docker image, and is controlled through REST interface.
[![Docker Image CI][1]][2]

Stream tester is a tool to test Livepeer network performance and
stability. It sends RTMP streams into broadcaster nodes and download
back transcoded data and calculates success rate. It can be used as
standalone command line tool or as part of [test harness][3]. As part
of test harness, it works in server mode, inside docker image, and is
controlled through REST interface.

Also it can be used to pull arbitrary .m3u8 stream and save it to disk.

When running from command line, Stream tester has two different modes.
First one is for (load) testing Livepeer Broadcaster and second one was developed for running long test against Livepeer Wowza plugin, but can be used with any transcoder that takes RTMP input and output HLS.

First one is for (load) testing Livepeer Broadcaster and second one
was developed for running long test against Livepeer Wowza plugin, but
can be used with any transcoder that takes RTMP input and output HLS.

## Binaries
Automatically built binaries can be found on [releases](https://github.com/livepeer/stream-tester/releases) page.

## Video test files
[official_test_source_2s_keys_24pfs.mp4](https://storage.googleapis.com/lp_testharness_assets/official_test_source_2s_keys_24pfs.mp4)
Automatically built binaries can be found on
[releases](https://github.com/livepeer/stream-tester/releases) page.

[official_test_source_2s_keys_24pfs_3min.mp4](https://storage.googleapis.com/lp_testharness_assets/official_test_source_2s_keys_24pfs_3min.mp4)
## Video test files

[bbb_sunflower_1080p_30fps_normal_t02.mp4](https://storage.googleapis.com/lp_testharness_assets/bbb_sunflower_1080p_30fps_normal_t02.mp4)
- [official_test_source_2s_keys_24pfs.mp4](https://storage.googleapis.com/lp_testharness_assets/official_test_source_2s_keys_24pfs.mp4)

[bbb_sunflower_1080p_30fps_normal_2min.mp4](https://storage.googleapis.com/lp_testharness_assets/bbb_sunflower_1080p_30fps_normal_2min.mp4)
- [official_test_source_2s_keys_24pfs_3min.mp4](https://storage.googleapis.com/lp_testharness_assets/official_test_source_2s_keys_24pfs_3min.mp4)

- [bbb_sunflower_1080p_30fps_normal_t02.mp4](https://storage.googleapis.com/lp_testharness_assets/bbb_sunflower_1080p_30fps_normal_t02.mp4)

- [bbb_sunflower_1080p_30fps_normal_2min.mp4](https://storage.googleapis.com/lp_testharness_assets/bbb_sunflower_1080p_30fps_normal_2min.mp4)

## Command line

If name of the file to stream is not specified, then default one - `official_test_source_2s_keys_24pfs.mp4` is used. It should be placed in the same directory as Stream Tester. Can be downloaded from [here](https://storage.googleapis.com/lp_testharness_assets/official_test_source_2s_keys_24pfs.mp4).
If name of the file to stream is not specified, then default one -
`official_test_source_2s_keys_24pfs.mp4` is used. It should be placed
in the same directory as Stream Tester. Can be downloaded from
[here](https://storage.googleapis.com/lp_testharness_assets/official_test_source_2s_keys_24pfs.mp4).

All options can be put into config file and used like `./streamtester -config local.cfg`
All options can be put into config file and used like

```sh
./streamtester -config local.cfg
```

Config file example: [local.cfg](local.cfg)

### Load testing mode

Streams file to its end, or for the time specified. Can stream arbitrary number of streams simultaneously and repeat streaming any number of times. Counts number of segments streamed and number of segments that was readed back. Reports success rate (readed segments / segments should have been readed). Calculate transcode latency (should be enabled from command line).
Streams file to its end, or for the time specified. Can stream
arbitrary number of streams simultaneously and repeat streaming any
number of times. Counts number of segments streamed and number of
segments that was readed back. Reports success rate (readed segments /
segments should have been readed). Calculate transcode latency (should
be enabled from command line).

Usage:
`./streamtester -host localhost -rtmp 1935 -media 8935 -profiles 2 -repeat 1 -sim 1 -file file_to_stream.mp4`

```sh
./streamtester -host localhost -rtmp 1935 -media 8935 -profiles 2 -repeat 1 -sim 1 -file file_to_stream.mp4
```

Params:

- `-host` Host name of broadcaster to stream to
- `-media-host` Host name to read transcoded stream from. If not specified, `-host` will be used
- `-rtmp` Port number to stream RTMP stream to
- `-media` Port number to download media from
- `-profiles` How many transcoding profiles broadcaster configured with
- `-sim` How many simultaneous streams stream into broadcaster
- `-repeat` How many times to repeat streaming
- `-latency` Measure transcoding latency
- `-time` Time to stream streams (40s, 4m, 24h45m). Not compatible with repeat option
- `-http-ingest` Use HTTP push instead of RTMP
- `-file` Name of the file to stream
- `-host` Host name of broadcaster to stream to
- `-media-host` Host name to read transcoded stream from. If not specified, `-host` will be used
- `-rtmp` Port number to stream RTMP stream to
- `-media` Port number to download media from
- `-profiles` How many transcoding profiles broadcaster configured with
- `-sim` How many simultaneous streams stream into broadcaster
- `-repeat` How many times to repeat streaming
- `-latency` Measure transcoding latency
- `-time` Time to stream streams (40s, 4m, 24h45m). Not compatible with repeat option
- `-http-ingest` Use HTTP push instead of RTMP
- `-file` Name of the file to stream

### Infinite stream testing mode
In this mode Stream Tester streams video to RTMP ingest point and read HLS stream back. Streaming is stopped only on error, so it will be infinite if transcoding is done ideally.

In this mode Stream Tester streams video to RTMP ingest point and read
HLS stream back. Streaming is stopped only on error, so it will be
infinite if transcoding is done ideally.

Errors can be reported to Discord.

Checks downloaded segments for validity by parsing them using `joy4` lib. Can save segments with errors to Google Storage.
Checks downloaded segments for validity by parsing them using `joy4`
lib. Can save segments with errors to Google Storage.

Checks for gaps in HLS stream (PTS of the first frame of the segment should be PTS of first frame of previous segment plus length of previous segment).
Checks for gaps in HLS stream (PTS of the first frame of the segment
should be PTS of first frame of previous segment plus length of
previous segment).

Checks for time drift between transcoded streams (At each given time, PTSs of first segments in different media streams shouldn't differ for more than four seconds. (Apple's HLS validator tool report error if difference is more than two seconds)).
Checks for time drift between transcoded streams (At each given time,
PTSs of first segments in different media streams shouldn't differ for
more than four seconds. (Apple's HLS validator tool report error if
difference is more than two seconds)).

Stops streaming if there is no new segments in HLS stream for a 30 seconds.


Usage:
`./streamtester -ignore-time-drift -ignore-gaps -wowza -wait-for-target 150s -media-url http://site.com:1935/something.m3u8 -rtmp-url rtmp://site.com:1935/something -profiles 3 -file file_to_stream.mp4`

```sh
./streamtester -ignore-time-drift \
-ignore-gaps -wowza -wait-for-target 150s \
-media-url http://site.com:1935/something.m3u8 \
-rtmp-url rtmp://site.com:1935/something \
-profiles 3 \
-file file_to_stream.mp4
```

Params:

- `-wowza` Should be specified if streaming to Wowza. Removes Wowza's session cookies from manifest names
- `-rtmp-url` URL to stream RTMP stream to
- `-media-url` URL of main .m3u8 manifest to pull transcoded stream back from
- `-profiles` How many transcoded (not including source) profiles should be in resulting (.m3u8) stream
- `ignore-no-codec-error` Do not stop streaming if segment without codec's info downloaded
- `ignore-gaps` Do not stop streaming if gaps found
- `ignore-time-drift` Do not stop streaming if time drift detected
- `wait-for-target` Timeout for trying to connect to RTMP ingest host. Should be specified in seconds or minutes (10s, 4m)
- `discord-url` URL of Discord's webhook to send messages to Discord channel
- `discord-users` Id's of users to notify in case of failure
- `discord-user-name` User name to use when sending messages to Discord
- `gsbucket` Google Storage bucket (to store segments that was not successfully parsed)
- `gskey` Google Storage private key (in json format (actual key, not file name))
- `-file` Name of the file to stream
- `-wowza` Should be specified if streaming to Wowza. Removes Wowza's session cookies from manifest names
- `-rtmp-url` URL to stream RTMP stream to
- `-media-url` URL of main .m3u8 manifest to pull transcoded stream back from
- `-profiles` How many transcoded (not including source) profiles should be in resulting (.m3u8) stream
- `ignore-no-codec-error` Do not stop streaming if segment without codec's info downloaded
- `ignore-gaps` Do not stop streaming if gaps found
- `ignore-time-drift` Do not stop streaming if time drift detected
- `wait-for-target` Timeout for trying to connect to RTMP ingest host. Should be specified in seconds or minutes (10s, 4m)
- `discord-url` URL of Discord's webhook to send messages to Discord channel
- `discord-users` Id's of users to notify in case of failure
- `discord-user-name` User name to use when sending messages to Discord
- `gsbucket` Google Storage bucket (to store segments that was not successfully parsed)
- `gskey` Google Storage private key (in json format (actual key, not file name))
- `-file` Name of the file to stream

### Infinite HLS pull testing mode
In this mode Stream Tester pulls arbitrary HLS stream and runs same checks as in previous mode.
To use it specify `-media-url` without specifying `-rtmp-url`.



In this mode Stream Tester pulls arbitrary HLS stream and runs same
checks as in previous mode. To use it specify `-media-url` without
specifying `-rtmp-url`.

### Saving arbitrary stream to file

Running

`./streamtester -save -infinite-pull http://site.com:1935/live/main_playlist.m3u8`
```sh
./streamtester -save -infinite-pull http://site.com:1935/live/main_playlist.m3u8
```

will save pull `main_playlist.m3u8` stream and save all the segments along with (`VOD`) manifests to current directory.

## Server mode

Run
`./streamtester -server -serverAddr localhost:7934`
```
./streamtester -server -serverAddr localhost:7934
```

This command block until interrupted.

It provide these REST endpoints:

---

GET `/stats?latencies&base_manifest_id=basemanifestid`

if optional parameter `latencies` present, `/stats` will return raw latencies values
if optional parameter `base_manifest_id` present, then stats will be filtered by that base manifest id

returns object:

```json
{
"rtmp_active_streams": 1,
"rtmp_streams": 1,
"media_streams": 1,
"sent_segments": 3,
"downloaded_segments": 7,
"should_have_downloaded_segments": 9,
"failed_to_download_segments": 0,
"bytes_downloaded": 2721864,
"success_rate": 77.77777777777779,
"connection_lost": 0,
"start_time": "2020-01-07T21:59:51.008881+02:00",
"finished": false,
"raw_source_latencies": [
79210696,
87020114
],
"raw_transcoded_latencies": [
298718923,
324982522
]
"rtmp_active_streams": 1,
"rtmp_streams": 1,
"media_streams": 1,
"sent_segments": 3,
"downloaded_segments": 7,
"should_have_downloaded_segments": 9,
"failed_to_download_segments": 0,
"bytes_downloaded": 2721864,
"success_rate": 77.77777777777779,
"connection_lost": 0,
"start_time": "2020-01-07T21:59:51.008881+02:00",
"finished": false,
"raw_source_latencies": [79210696, 87020114],
"raw_transcoded_latencies": [298718923, 324982522]
}
```

Where
Where

- `rtmp_active_streams` - number of streams is being currently streamed
- `rtm_pstreams` - total number of RTMP streams (active plus finished)
- `rtm_pstreams` - total number of media streams (downloading back segments)
- `sent_segments` - total number of segments sent to broadcaster
- `downloaded_segments` - total number of segments downloaded back
- `should_have_downloaded_segments` - number of segments should be downloaded back for 100% success rate. It equals `sent_segments` * (`numberOfTranscodingProfiles` + 1)
- `failed_to_download_segments` - number of segments failed to download with some error
- `bytes_downloaded` - number of bytes downloaded
- `success_rate` - success rate, percent
- `connection_lost` - number of times streamer lost connection to broadcaster. Streamer does not attempt to reconnect, so statistics will only have data till connection was lost
- `finished` - indicates that all streaming tasks are finished
- `rtmp_active_streams` - number of streams is being currently streamed
- `rtm_pstreams` - total number of RTMP streams (active plus finished)
- `rtm_pstreams` - total number of media streams (downloading back segments)
- `sent_segments` - total number of segments sent to broadcaster
- `downloaded_segments` - total number of segments downloaded back
- `should_have_downloaded_segments` - number of segments should be downloaded back for 100% success rate. It equals `sent_segments` \* (`numberOfTranscodingProfiles` + 1)
- `failed_to_download_segments` - number of segments failed to download with some error
- `bytes_downloaded` - number of bytes downloaded
- `success_rate` - success rate, percent
- `connection_lost` - number of times streamer lost connection to broadcaster. Streamer does not attempt to reconnect, so statistics will only have data till connection was lost
- `finished` - indicates that all streaming tasks are finished

---

POST `/start_streams`

Accepts object:

```json
{
"host": "localhost",
Expand All @@ -176,29 +213,31 @@ Accepts object:
"do_not_clear_stats": false,
"measure_latency": true,
"http_ingest": false
}

}
```

`file_name` - should exists in local filesystem of streamer.
`do_not_clear_stats` - if true, on new call to `/start_streams` do not clear old stats, but instead append to it


Returns
Returns

```json
{
"success": true,
"base_manifest_id": "manifest_id_here"
"success": true,
"base_manifest_id": "manifest_id_here"
}
```

Can return 404 if request contains wrong parameters.


---

GET `/stop`

Stop currently running streams.
Stop currently running streams.

Doesn't return any data, only status 200

Doesn't return any data, only status 200
[1]: https://github.com/livepeer/stream-tester/actions/workflows/docker.yaml/badge.svg
[2]: https://github.com/livepeer/stream-tester/actions/workflows/docker.yaml
[3]: https://github.com/livepeer/test-harness

0 comments on commit 97ee917

Please sign in to comment.