Adding support for new tests to the Observation Framework

When new tests are added to the dcptf-tests repository, support for these will also need to be added to the Observation Framework. The scale of changes required will depend on how much the new tests diverge from existing tests, as follows.

When test and observations are the same as an existing test

For example, the 'playback-of-encrypted-content' test uses the same Observation Framework test code and observations as the (unencrypted) 'sequential_track_playback' test.

To add such a test simply requires adding a new test name mapping to the existing test module and class name in the "of_testname_map.json" file. For example:

"playback-of-encrypted-content.html": [
    "sequential_track_playback",
    "SequentialTrackPlayback"
],

When correct observations exist but new test requires different combination of observations

For example, the 'regular-playback-of-a-cmaf-presentation' test uses the same test logic as the 'sequential_track_playback' test, however it requires a different list of observations.

Create a new test code module containing a new test class derived from an existing test. Then override the method to provide the correct list of observations, for example:

class RegularPlaybackOfACmafPresentation(SequentialTrackPlayback):
    def _init_observations(self) -> None:
        self.observations = [...]

Add the new test name, python module, and class name to the "of_testname_map.json" file.

When correct observations exist within an existing test but require different parameters

For example, the 'random-access-to-time' test uses the same observations as the 'sequential_track_playback' test, however it requires different parameters to be passed.

Create a new test code module containing a new test class derived from an existing test with the same observations. Then override the appropriate methods to provide the correct parameters for the observations, for example:

class RandomAccessToTime(SequentialTrackPlayback):
    def _init_parameters(self) -> None:
        [...]

    def _get_first_frame_num(self, frame_rate: Fraction) -> int:
        [...]

    def _get_expected_track_duration(self) -> float:
        [...]

Add the new test name, python module, and class name to the "of_testname_map.json" file.

When new observations are required

Create new observation class(es) either derived from an existing observation if an appropriate one exists, or derived from the 'Observation' base class. Override methods as needed and provide a make_observation() method. For example,

class EverySampleRendered(Observation):
    def __init__(self, global_configurations: GlobalConfigurations, name: str = None):
        [...]

    def make_observation( self, test_type, mezzanine_qr_codes: List[MezzanineDecodedQr],
                        _unused, parameters_dict: dict ) -> Dict[str, str]:
        [...]

Create a new test code module and class as described above. Implement a make_observations() method that calls the required observations and returns a list of pass/fail results. For example:

def make_observations(
    self,
    mezzanine_qr_codes: List[MezzanineDecodedQr],
    test_status_qr_codes: List[TestStatusDecodedQr],
) -> List[dict]:
    [...]

Add the new test name, python module, and class name to the "of_testname_map.json" file.