feat(dash): Enable Stateful Mode and rsq Corruptions for DASH (#66)

* feat(dash): Enable Stateful Mode and rsq Corruptions for DASH - Imported STATEFUL and newState utilities for DASH (segment.ts, dashManifestUtils.ts and dashManifestUtils.test.ts) to support stateful mode. - Ensured rsq corruptions are possible for DASH when the proxy is running in stateful mode, similar to how rsq corruptions are enabled for HLS. This update allows for the use of relative sequence numbers in DASH manifests, aligning the functionality with HLS. The DASH proxy now properly handles stateful operations and rsq parameters, enabling more flexible stream corruptions. * chore: Apply linting to codebase * chore: Run pretty with --write mode for all code * chore: Run prettier on history files * test: Add unit test for STATEFUL=true npm run dev mode, both for DASH and HLS case * fix: Add removed import statement from utils and include .history in .gitignore * chore: run linting and prettier on updated codebase * docs: Update README, including change with rsq corruptions for DASH in STATEFUL mode * docs: Update README with example url of rsq corruption for DASH * chore: clean up .history files make sure not to be included in merge --------- Co-authored-by: Nfrederiksen <[email protected]>
Eyevinn · Jul 2, 2024 · 1a1fea7 · 1a1fea7
1 parent e3c81b4
commit 1a1fea7
Show file tree

Hide file tree

Showing 11 changed files with 107 additions and 18 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,3 +3,4 @@ node_modules/
 dist/
 .env
 dev
+.history/
diff --git a/.prettierignore b/.prettierignore
@@ -1,3 +1,4 @@
 dist/
 *.yml
 README.md
+.history/
diff --git a/README.md b/README.md
@@ -107,8 +107,8 @@ e.i. `https://<chaos-proxy>/api/v2/manifests/hls/proxy-master.m3u8?url=<some_url
 Across all corruptions, there are 3 ways to target a segment in a playlist for corruption.
 
 1. `i`: The segment's list index in any Media Playlist, with HLS segments starting at 0 and MPEG-DASH segments starting at 1. For a Media Playlist with 12 segments, `i`=11, would target the last segment for HLS and `i`=12, would target the last segment for MPEG-DASH.
-2. `sq`: The segment's Media Sequence Number for HLS, or the "$Number$" or "$Time$" part of a segment URL for DASH. For an HLS Media Playlist with 12 segments, and where `#EXT-X-MEDIA-SEQUENCE` is 100, `sq`=111 would target the last segment. When corrupting a live HLS stream it is recommended to target with `rsq`.
-3. `rsq`: A relative sequence number, counted from where the live stream is currently at when requesting manifest. Can also use a negative integer, which enables counting backwards from the end of the manifest instead. (**HLS SUPPORTED ONLY IN STATEFUL MODE**)
+2. `sq`: The segment's Media Sequence Number for HLS, or the "$Number$" or "$Time$" part of a segment URL for DASH. For an HLS Media Playlist with 12 segments, and where `#EXT-X-MEDIA-SEQUENCE` is 100, `sq`=111 would target the last segment. When corrupting a live HLS or DASH stream it is recommended to target with `rsq`.
+3. `rsq`: A relative sequence number, counted from where the live stream is currently at when requesting manifest. Can also use a negative integer, which enables counting backwards from the end of the manifest instead. (**SUPPORTED ONLY IN STATEFUL MODE FOR BOTH HLS AND DASH**)
 
 Below are configuration JSON object templates for the currently supported corruptions. A query should have its value be an array consisting of any one of these 3 types of items:
 
@@ -258,6 +258,11 @@ https://<chaos-proxy>/api/v2/manifests/dash/proxy-master.mpd?url=https://livesim
 https://<chaos-proxy>/api/v2/manifests/dash/proxy-master.mpd?url=https://f53accc45b7aded64ed8085068f31881.egress.mediapackage-vod.eu-north-1.amazonaws.com/out/v1/1c63bf88e2664639a6c293b4d055e6bb/64651f16da554640930b7ce2cd9f758b/66d211307b7d43d3bd515a3bfb654e1c/manifest.mpd&throttle=[{i:*,rate:10000}]
 ```
 
+7. LIVE: Example of MPEG-DASH with a segment delay of 5000ms on segment with relative sequence number equal to 2:
+
+```
+https://<chaos-proxy>/api/v2/manifests/dash/proxy-master.mpd?url=https://livesim.dashif.org/livesim/testpic_2s/Manifest.mpd&delay=[{rsq:2, ms:5000}]
+```
 ## Development Environment
 
 To deploy and update development environment create and push a tag with the suffix `-dev`, for example `my-feat-test-dev`. If you run `npm run deploy:dev` it will automatically create a tag based on git revision with the `-dev` suffix and push it.

diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -55,7 +55,7 @@
     "eslint-plugin-prettier": "^4.2.1",
     "husky": "^2.7.0",
     "lint-staged": "^12.3.4",
-    "prettier": "^2.8.7",
+    "prettier": "2.8.8",
     "ts-jest": "^27.1.3",
     "tsc-watch": "^5.0.3",
     "typescript": "^3.7.3"

diff --git a/src/manifests/handlers/dash/segment.ts b/src/manifests/handlers/dash/segment.ts
@@ -4,7 +4,9 @@ import {
   composeALBEvent,
   generateErrorResponse,
   isValidUrl,
-  segmentUrlParamString
+  segmentUrlParamString,
+  STATEFUL,
+  newState
 } from '../../../shared/utils';
 import delaySCC from '../../utils/corruptions/delay';
 import statusCodeSCC from '../../utils/corruptions/statusCode';
@@ -56,6 +58,10 @@ export default async function dashSegmentHandler(
     }
     const reqSegmentIndexInt = parseInt(reqSegmentIndexOrTimeStr);
 
+    const stateKey = STATEFUL
+      ? newState({ initialSequenceNumber: undefined })
+      : undefined;
+
     // Replace RepresentationID in url if present
     if (representationIdStr) {
       segmentUrl = segmentUrl.replace(
@@ -85,7 +91,8 @@ export default async function dashSegmentHandler(
     const dashUtils = dashManifestUtils();
     const mergedMaps = dashUtils.utils.mergeMap(
       reqSegmentIndexInt,
-      allMutations
+      allMutations,
+      stateKey
     );
     const segUrl = new URL(segmentUrl);
     const cleanSegUrl = segUrl.origin + segUrl.pathname;

diff --git a/src/manifests/utils/configs.test.ts b/src/manifests/utils/configs.test.ts
@@ -257,6 +257,60 @@ describe('configs', () => {
         );
       });
     });
+
+    describe('not in stateful mode', () => {
+      let nonStatefulConfig;
+      beforeAll(() => {
+        delete process.env.STATEFUL; // Ensure STATEFUL is not set
+        nonStatefulConfig = require('./configs');
+      });
+
+      it('should handle DASH mode without stateful', () => {
+        // Arrange
+        const configs = nonStatefulConfig.corruptorConfigUtils(
+          new URLSearchParams(
+            'statusCode=[{rsq:15,code:400}]&throttle=[{sq:15,rate:1000}]'
+          )
+        );
+
+        configs.register(statusCodeConfig).register(throttleConfig);
+
+        // Act
+
+        const [err, actual] = configs.getAllManifestConfigs(0, true);
+
+        // Assert
+        expect(err).toEqual({
+          status: 400,
+          message:
+            'Relative sequence numbers on DASH are only supported when proxy is running in stateful mode'
+        });
+        expect(actual).toBeNull();
+      });
+
+      it('should handle HLS mode without stateful', () => {
+        // Arrange
+        const configs = nonStatefulConfig.corruptorConfigUtils(
+          new URLSearchParams(
+            'statusCode=[{rsq:15,code:400}]&throttle=[{sq:15,rate:1000}]'
+          )
+        );
+
+        configs.register(statusCodeConfig).register(throttleConfig);
+
+        // Act
+
+        const [err, actual] = configs.getAllManifestConfigs(0, false);
+
+        // Assert
+        expect(err).toEqual({
+          status: 400,
+          message:
+            'Relative sequence numbers on HLS are only supported when proxy is running in stateful mode'
+        });
+        expect(actual).toBeNull();
+      });
+    });
   });
 
   describe('getAllSegmentConfigs', () => {

diff --git a/src/manifests/utils/configs.ts b/src/manifests/utils/configs.ts
@@ -167,6 +167,16 @@ export const corruptorConfigUtils = function (
               null
             ];
           }
+          if (hasRelativeSequences && !STATEFUL && isDash) {
+            return [
+              {
+                status: 400,
+                message:
+                  'Relative sequence numbers on DASH are only supported when proxy is running in stateful mode'
+              },
+              null
+            ];
+          }
 
           // If bitrate is set, filter out segments that doesn't match
           params = params.filter(

diff --git a/src/manifests/utils/dashManifestUtils.test.ts b/src/manifests/utils/dashManifestUtils.test.ts
@@ -157,7 +157,8 @@ describe('utils.mergeMap', () => {
     // Act
     const actual = dashManifestUtils().utils.mergeMap(
       mockReqSegIndex,
-      mockAllCorruptions
+      mockAllCorruptions,
+      undefined
     );
     const expected = new Map<string, CorruptorConfig>()
       .set('a', { fields: { ms: 100 } })
@@ -191,7 +192,11 @@ describe('utils.mergeMap', () => {
     const actual: CorruptorConfigMap[] = [];
     for (let segIdx = 0; segIdx < 3; segIdx++) {
       actual.push(
-        dashManifestUtils().utils.mergeMap(segIdx, mockAllCorruptions)
+        dashManifestUtils().utils.mergeMap(
+          segIdx,
+          mockAllCorruptions,
+          undefined
+        )
       );
     }
 
@@ -226,7 +231,11 @@ describe('utils.mergeMap', () => {
     const actual: CorruptorConfigMap[] = [];
     for (let segIdx = 0; segIdx < 3; segIdx++) {
       actual.push(
-        dashManifestUtils().utils.mergeMap(segIdx, mockAllCorruptions)
+        dashManifestUtils().utils.mergeMap(
+          segIdx,
+          mockAllCorruptions,
+          undefined
+        )
       );
     }
     const expected = [
@@ -252,7 +261,8 @@ describe('utils.mergeMap', () => {
     // Act
     const actual = dashManifestUtils().utils.mergeMap(
       mockReqSegIndex,
-      mockAllCorruptions
+      mockAllCorruptions,
+      undefined
     );
     const expected = new Map<string, CorruptorConfig>().set('a', {
       fields: {}

diff --git a/src/manifests/utils/dashManifestUtils.ts b/src/manifests/utils/dashManifestUtils.ts
@@ -7,7 +7,8 @@ import { URLSearchParams } from 'url';
 interface DASHManifestUtils {
   mergeMap: (
     segmentListSize: number,
-    configsMap: IndexedCorruptorConfigMap
+    configsMap: IndexedCorruptorConfigMap,
+    stateKey: string | undefined
   ) => CorruptorConfigMap;
 }
 

diff --git a/src/shared/utils.ts b/src/shared/utils.ts
@@ -14,7 +14,7 @@ import {
   FastifyInstance
 } from 'fastify';
 import { addSSMUrlParametersToUrl } from './aws.utils';
-
+import { URLSearchParams } from 'url';
 import dotenv from 'dotenv';
 import { Readable } from 'stream';
 import NodeCache from 'node-cache';