Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3support #293

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

S3support #293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0475d79
> Task :javadoc FAILED
rjaycocks Feb 26, 2021
37c0dd0
Added S3 support for IMPAnalyzer
rjaycocks Mar 9, 2021
56b49cc
Fix unit test. Was not taking into account different line endings per…
rjaycocks Mar 26, 2021
43daf73
Support obtaining the meta data from the footer partition. MXF files …
rjaycocks Apr 30, 2021
b0d2133
Check whether the header is closed. If the header is not complete, ob…
rjaycocks May 14, 2021
bf33635
Added check of footer partition to validateEssencePartition if the he…
rjaycocks Jun 1, 2021
5292ee8
Handle http style locator paths differently to s3 style locator paths…
rjaycocks Jun 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Photon is a Java implementation of the Interoperable Master Format (IMF) standar

The goal of the Photon is to provide a simple standardized interface to completely validate an IMP.

## Origins

The code in this repo was originally derived from https://github.com/Netflix/photon.git. This repo adds S3 support to the IMP Analyzer.

## Build

### JDK requirements
Expand Down Expand Up @@ -98,3 +102,31 @@ java -cp build\libs\*; com.netflix.imflibrary.st2067_2.Composition composition_p
```
java -cp build\libs\*; com.netflix.imflibrary.app.IMPAnalyzer IMP_folder_path
```

### S3 Support

This repo adds S3 support to the IMPAnalyzer. Extras command lines options can be passed in to specify the AWS credentials when accessing files on S3. If the path beings with `s3://`, `http://` or `https://` it is currently assumed to be on S3. To indicate a directory, a `/` needs to be appended to the path.

**Configures basic S3 credentials**
` --aws.accesskey=ACCESS_KEY`
` --aws.secretkey=SECRET_KEY`
` --aws.token=TOKEN (optional)`

**Uses profile as defined in the aw3 credentials file.**
` --aws.profile=PROFILE`

**Specifies a role for accessing the S3 bucket.**
` --aws.rolearn=ROLEARN`
` --aws.externalid=EXTERNALID (optional)`

**Uses anonymouse credentials.**
` --aws.anonymous`

**Specifies the endpoint to use.**
` --aws.endpoint=ENPOINT`

All the above are optional.

E.G:

`java -cp build\libs\*; com.netflix.imflibrary.app.IMPAnalyzer s3://bucket/imp/ --aws.profile=test_profile`
62 changes: 57 additions & 5 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,59 @@
plugins {
id 'nebula.netflixoss' version '9.1.0'
id 'nebula.netflixoss' version '9.1.0' apply false
id 'java'
id 'pmd'
id 'com.github.spotbugs' version "4.4.4"
id 'jacoco'
}

group = 'com.netflix.photon'
if (project.hasProperty('dalet')) {
apply plugin: 'maven-publish'

publishing {
publications {
maven(MavenPublication) {
group='com.dalet.imf'
version='4.9.2'
artifactId = 'Photon'

from components.java

pom {
licenses {
license {
name = 'The Apache Software License, Version 2.0'
url = 'http://www.apache.org/licenses/LICENSE-2.0.txt'
}
}
}
}
}
repositories {
maven {
url "$System.env.NEXUS_URL"
credentials {
username "$System.env.NEXUS_USER"
password "$System.env.NEXUS_PASSORD"
}
}
}
}

jar {
manifest {
attributes 'Implementation-Title': 'com.amberfin.photon#Photon;4.9.2'
attributes 'Implementation-Version': '4.9.2'
attributes 'Module-Origin': 'https://github.com/Netflix/photon.git'
attributes 'Build-Date': new java.text.SimpleDateFormat('yyyy-MM-dd_HH:mm:ss').format(new Date())
attributes 'Change': 'git rev-parse --short HEAD'.execute().text.trim()
attributes 'Full-Change': 'git rev-parse HEAD'.execute().text.trim()
attributes 'Branch': 'git branch --show-current'.execute().text.trim()
}
}

} else {
apply plugin: 'nebula.netflixoss'
}

repositories {
mavenCentral()
Expand Down Expand Up @@ -55,14 +102,16 @@ javadoc {
}

dependencies {

compileOnly "com.github.spotbugs:spotbugs-annotations:${spotbugs.toolVersion.get()}"
implementation "org.slf4j:slf4j-api:1.7.+"
implementation "org.slf4j:slf4j-log4j12:1.7.+"
implementation "org.slf4j:slf4j-api:1.7.30"
implementation "org.slf4j:slf4j-log4j12:1.7.30"
implementation "com.amazonaws:aws-java-sdk-bom:1.11.964"
/**
* Following includes the RegXMLLib dependency from Maven Central.
*/
implementation "com.sandflow:regxmllib:1.1.3"
implementation 'com.amazonaws:aws-java-sdk-s3:1.11.964'
implementation 'com.amazonaws:aws-java-sdk-sts:1.11.964'
/**
* Following should be enabled and the above should be disabled
* when necessary to verify changes to the RegXMLLib library that are
Expand All @@ -71,6 +120,9 @@ dependencies {
/*compile "com.sandflow:regxmllib:${revRegXMLSNAPSHOT}"*/
testImplementation "org.mockito:mockito-core:3.3+"
testImplementation "org.testng:testng:7.+"
testImplementation ("com.adobe.testing:s3mock:2.1.28") {
exclude group: "ch.qos.logback"
}
}

test {
Expand Down
44 changes: 22 additions & 22 deletions src/main/java/com/netflix/imflibrary/IMFConstraints.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

import com.netflix.imflibrary.exceptions.IMFException;
import com.netflix.imflibrary.exceptions.MXFException;
import com.netflix.imflibrary.st0377.HeaderPartition;
import com.netflix.imflibrary.st0377.HeaderOrFooterPartition;
import com.netflix.imflibrary.st0377.PartitionPack;
import com.netflix.imflibrary.st0377.header.*;
import com.netflix.imflibrary.st2067_201.IABEssenceDescriptor;
Expand Down Expand Up @@ -69,9 +69,9 @@ public static HeaderPartitionIMF checkIMFCompliance(MXFOperationalPattern1A.Head
{
int previousNumberOfErrors = imfErrorLogger.getErrors().size();

HeaderPartition headerPartition = headerPartitionOP1A.getHeaderPartition();
HeaderOrFooterPartition headerOrFooterPartition = headerPartitionOP1A.getHeaderPartition();

Preface preface = headerPartition.getPreface();
Preface preface = headerOrFooterPartition.getPreface();
GenericPackage genericPackage = preface.getContentStorage().getEssenceContainerDataList().get(0).getLinkedPackage();
SourcePackage filePackage;
filePackage = (SourcePackage)genericPackage;
Expand Down Expand Up @@ -159,12 +159,12 @@ public static HeaderPartitionIMF checkIMFCompliance(MXFOperationalPattern1A.Head
waveAudioEssenceDescriptor.getChannelAssignmentUL(), new MXFUID(IMFConstraints.IMF_CHANNEL_ASSIGNMENT_UL), packageID.toString()));
}
//RFC-5646 spoken language is a part of the MCALabelSubDescriptor and SoundFieldGroupLabelSubdescriptors according to Section 5.3.6.5 st2067-2:2016 has language around RFC-5646 primary spoken language
if (headerPartition.getAudioEssenceSpokenLanguage() == null) {
if (headerOrFooterPartition.getAudioEssenceSpokenLanguage() == null) {
imfErrorLogger.addError(IMFErrorLogger.IMFErrors.ErrorCodes.IMF_CORE_CONSTRAINTS_ERROR, IMFErrorLogger.IMFErrors.ErrorLevels.NON_FATAL, IMFConstraints.IMF_ESSENCE_EXCEPTION_PREFIX + String.format("WaveAudioEssenceDescriptor in the IMFTrackFile represented by ID %s does not have a RFC5646 spoken language indicated, language code shall be set in the SoundFieldGroupLabelSubDescriptor, unless the AudioEssence does not have a primary spoken language.", packageID.toString()));
} else {
//Section 6.3.6 st377-4:2012
if (!IMFConstraints.isSpokenLanguageRFC5646Compliant(headerPartition.getAudioEssenceSpokenLanguage())) {
List<String> strings = IMFConstraints.getPrimarySpokenLanguageUnicodeString(headerPartition.getAudioEssenceSpokenLanguage());
if (!IMFConstraints.isSpokenLanguageRFC5646Compliant(headerOrFooterPartition.getAudioEssenceSpokenLanguage())) {
List<String> strings = IMFConstraints.getPrimarySpokenLanguageUnicodeString(headerOrFooterPartition.getAudioEssenceSpokenLanguage());
imfErrorLogger.addError(new ErrorLogger.ErrorObject(IMFErrorLogger.IMFErrors.ErrorCodes.IMF_ESSENCE_COMPONENT_ERROR, IMFErrorLogger.IMFErrors.ErrorLevels.NON_FATAL, String.format("Language Code (%s) in SoundFieldGroupLabelSubdescriptor in the IMFTrackfile represented by ID %s is not RFC5646 compliant", strings, packageID.toString())));
}
}
Expand All @@ -173,14 +173,14 @@ public static HeaderPartitionIMF checkIMFCompliance(MXFOperationalPattern1A.Head
if (!StructuralMetadata.isAudioWaveClipWrapped(waveAudioEssenceDescriptor.getEssenceContainerUL().getULAsBytes()[14])) {
imfErrorLogger.addError(IMFErrorLogger.IMFErrors.ErrorCodes.IMF_CORE_CONSTRAINTS_ERROR, IMFErrorLogger.IMFErrors.ErrorLevels.NON_FATAL, IMFConstraints.IMF_ESSENCE_EXCEPTION_PREFIX + String.format("WaveAudioEssenceDescriptor indicates that the Audio Essence within an Audio Track File is not Wave Clip-Wrapped in the IMFTrackFile represented by ID %s.", packageID.toString()));
}
List<InterchangeObject.InterchangeObjectBO> subDescriptors = headerPartition.getSubDescriptors();
List<InterchangeObject.InterchangeObjectBO> subDescriptors = headerOrFooterPartition.getSubDescriptors();
//Section 5.3.6.2 st2067-2:2016
if (subDescriptors.size() == 0) {
imfErrorLogger.addError(IMFErrorLogger.IMFErrors.ErrorCodes.IMF_CORE_CONSTRAINTS_ERROR, IMFErrorLogger.IMFErrors.ErrorLevels.NON_FATAL, IMFConstraints.IMF_ESSENCE_EXCEPTION_PREFIX +
String.format("WaveAudioEssenceDescriptor in the IMFTrackFile represented by ID %s indicates a channel count of %d, however there are %d AudioChannelLabelSubdescriptors, every audio channel should refer to exactly one AudioChannelLabelSubDescriptor and vice versa.", packageID.toString(), waveAudioEssenceDescriptor.getChannelCount(), subDescriptors.size()));
} else {
//Section 5.3.6.2 st2067-2:2016
Map<Long, AudioChannelLabelSubDescriptor> audioChannelLabelSubDescriptorMap = headerPartition.getAudioChannelIDToMCASubDescriptorMap();
Map<Long, AudioChannelLabelSubDescriptor> audioChannelLabelSubDescriptorMap = headerOrFooterPartition.getAudioChannelIDToMCASubDescriptorMap();
if (waveAudioEssenceDescriptor.getChannelCount() == 0 || waveAudioEssenceDescriptor.getChannelCount() != audioChannelLabelSubDescriptorMap.size()) {
imfErrorLogger.addError(IMFErrorLogger.IMFErrors.ErrorCodes.IMF_CORE_CONSTRAINTS_ERROR, IMFErrorLogger.IMFErrors.ErrorLevels.NON_FATAL, IMFConstraints.IMF_ESSENCE_EXCEPTION_PREFIX +
String.format("WaveAudioEssenceDescriptor in the IMFTrackFile represented by ID %s indicates a channel count of %d, however there are %d AudioChannelLabelSubdescriptors, every audio channel should refer to exactly one AudioChannelLabelSubDescriptor and vice versa.", packageID.toString(), waveAudioEssenceDescriptor.getChannelCount(), audioChannelLabelSubDescriptorMap.size()));
Expand All @@ -207,9 +207,9 @@ public static HeaderPartitionIMF checkIMFCompliance(MXFOperationalPattern1A.Head
imfErrorLogger.addError(IMFErrorLogger.IMFErrors.ErrorCodes.IMF_CORE_CONSTRAINTS_ERROR, IMFErrorLogger.IMFErrors.ErrorLevels.NON_FATAL, IMFConstraints.IMF_ESSENCE_EXCEPTION_PREFIX +
String.format("WaveAudioEssenceDescriptor in the IMFTrackFile represented by ID %s refers to a SoundFieldGroupLabelSubDescriptor that is missing one/all of MCATitle, MCATitleVersion, MCAAudioContentKind, MCAAudioElementKind, %n%s.", packageID.toString(), soundFieldGroupLabelSubDescriptorBO.toString()));
}
SoundFieldGroupLabelSubDescriptor soundFieldGroupLabelSubDescriptor = (SoundFieldGroupLabelSubDescriptor) headerPartition.getSoundFieldGroupLabelSubDescriptors()
SoundFieldGroupLabelSubDescriptor soundFieldGroupLabelSubDescriptor = (SoundFieldGroupLabelSubDescriptor) headerOrFooterPartition.getSoundFieldGroupLabelSubDescriptors()
.get(0);
List<InterchangeObject> audioChannelLabelSubDescriptors = headerPartition.getAudioChannelLabelSubDescriptors();
List<InterchangeObject> audioChannelLabelSubDescriptors = headerOrFooterPartition.getAudioChannelLabelSubDescriptors();
//Section 6.3.2 st377-4:2012
if (soundFieldGroupLabelSubDescriptor.getMCALinkId() == null) {
imfErrorLogger.addError(IMFErrorLogger.IMFErrors.ErrorCodes.IMF_ESSENCE_COMPONENT_ERROR, IMFErrorLogger.IMFErrors.ErrorLevels.NON_FATAL, IMFConstraints.IMF_ESSENCE_EXCEPTION_PREFIX +
Expand Down Expand Up @@ -360,12 +360,12 @@ else if (partitionPack.hasIndexTableSegments())
* runtime exception is thrown in case the MXF file contains audio essence that is not clip-wrapped
* This method does nothing if the MXF file does not contain audio essence
*
* @param headerPartition the header partition
* @param headerOrFooterPartition the header partition
* @param partitionPacks the partition packs
*/
public static void checkIMFCompliance(HeaderPartition headerPartition, List<PartitionPack> partitionPacks)
public static void checkIMFCompliance(HeaderOrFooterPartition headerOrFooterPartition, List<PartitionPack> partitionPacks)
{
Preface preface = headerPartition.getPreface();
Preface preface = headerOrFooterPartition.getPreface();
MXFDataDefinition filePackageMxfDataDefinition = null;
IMFErrorLogger imfErrorLogger = new IMFErrorLoggerImpl();
GenericPackage genericPackage = preface.getContentStorage().getEssenceContainerDataList().get(0).getLinkedPackage();
Expand Down Expand Up @@ -435,18 +435,18 @@ public MXFOperationalPattern1A.HeaderPartitionOP1A getHeaderPartitionOP1A()
return this.headerPartitionOP1A;
}

public boolean hasMatchingEssence(HeaderPartition.EssenceTypeEnum essenceType)
public boolean hasMatchingEssence(HeaderOrFooterPartition.EssenceTypeEnum essenceType)
{
MXFDataDefinition targetMXFDataDefinition;
if (essenceType.equals(HeaderPartition.EssenceTypeEnum.MainImageEssence))
if (essenceType.equals(HeaderOrFooterPartition.EssenceTypeEnum.MainImageEssence))
{
targetMXFDataDefinition = MXFDataDefinition.PICTURE;
}
else if(essenceType.equals(HeaderPartition.EssenceTypeEnum.MainAudioEssence))
else if(essenceType.equals(HeaderOrFooterPartition.EssenceTypeEnum.MainAudioEssence))
{
targetMXFDataDefinition = MXFDataDefinition.SOUND;
}
else if(essenceType.equals(HeaderPartition.EssenceTypeEnum.IABEssence))
else if(essenceType.equals(HeaderOrFooterPartition.EssenceTypeEnum.IABEssence))
{
targetMXFDataDefinition = MXFDataDefinition.SOUND;
}
Expand Down Expand Up @@ -565,9 +565,9 @@ private boolean hasWaveAudioEssenceDescriptor()
* A method that returns the IMF Essence Component type.
* @return essenceTypeEnum an enumeration constant corresponding to the IMFEssenceComponent type
*/
public HeaderPartition.EssenceTypeEnum getEssenceType(){
HeaderPartition headerPartition = this.headerPartitionOP1A.getHeaderPartition();
Preface preface = headerPartition.getPreface();
public HeaderOrFooterPartition.EssenceTypeEnum getEssenceType(){
HeaderOrFooterPartition headerOrFooterPartition = this.headerPartitionOP1A.getHeaderPartition();
Preface preface = headerOrFooterPartition.getPreface();
MXFDataDefinition filePackageMxfDataDefinition = null;

GenericPackage genericPackage = preface.getContentStorage().getEssenceContainerDataList().get(0).getLinkedPackage();
Expand All @@ -581,10 +581,10 @@ public HeaderPartition.EssenceTypeEnum getEssenceType(){
filePackageMxfDataDefinition = sequence.getMxfDataDefinition();
}
}
List<HeaderPartition.EssenceTypeEnum> essenceTypes = headerPartition.getEssenceTypes();
List<HeaderOrFooterPartition.EssenceTypeEnum> essenceTypes = headerOrFooterPartition.getEssenceTypes();
if(essenceTypes.size() != 1){
StringBuilder stringBuilder = new StringBuilder();
for(HeaderPartition.EssenceTypeEnum essenceTypeEnum : essenceTypes){
for(HeaderOrFooterPartition.EssenceTypeEnum essenceTypeEnum : essenceTypes){
stringBuilder.append(String.format("%s, ", essenceTypeEnum.toString()));
}
String message = String.format("IMF constrains MXF essences to mono essences only, however more" +
Expand Down
14 changes: 8 additions & 6 deletions src/main/java/com/netflix/imflibrary/IMPAsset.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,9 @@

import com.netflix.imflibrary.exceptions.IMFException;
import com.netflix.imflibrary.st0429_8.PackingList;

import javax.annotation.concurrent.Immutable;
import java.io.File;
import com.netflix.imflibrary.utils.Locator;
import java.net.URI;
import javax.annotation.concurrent.Immutable;

/**
* This class represents a thin, immutable wrapper around a PackingList {@link com.netflix.imflibrary.st0429_8.PackingList.Asset Asset}. It holds
Expand All @@ -34,13 +33,16 @@ public final class IMPAsset
{
private final URI uri;
private final PackingList.Asset asset;
private final Locator.Configuration configuration;

/**
* Constructor for an {@link IMPAsset IMPAsset} from a PackingList Asset and its URI. Construction
* fails if the URI is not absolute
* @param uri the absolute URI
* @param asset the corresponding asset
* @param configuration the locator configuration.
*/
public IMPAsset(URI uri, PackingList.Asset asset)
public IMPAsset(URI uri, PackingList.Asset asset, Locator.Configuration configuration)
{
if (!uri.isAbsolute())
{
Expand All @@ -51,7 +53,7 @@ public IMPAsset(URI uri, PackingList.Asset asset)
throw new IMFException(message, imfErrorLogger);
}
this.uri = uri;

this.configuration = configuration;
this.asset = asset;
}

Expand All @@ -61,7 +63,7 @@ public IMPAsset(URI uri, PackingList.Asset asset)
*/
public boolean isValid()
{//TODO: this implementation needs to improve
return new File(this.uri).exists();
return Locator.of(uri, configuration).exists();
}

@Override
Expand Down
Loading