Custom template in metadata file not working- which minimal fix / Failing to find right preset for custom template

[ffiore1]

Hi,
we need to personalise the solution, using our own video settings, creating at least 2 custom templates tobe used for different business scenarios.
We created our own template and I am trying to use that specifying it in my metadata file.
{
"srcVideo": "DIG_MINIMATCH_ARS_TOT-Test-AWS-Sol.mp4",
"ArchiveSource": false,
"FrameCapture": false,
"JobTemplate": "Company-XX_VOD_50fps",
"Priority": 100,
"AccelerationSettings": {
"Mode": "ENABLED"
}
}

I got 2 issues, not sure if they are related, so for the time being I am just creating this unique issue, happy to split this in 2.

Issues:

Issue 1)
the solution does not seem to consider properly the template specified in the metadata file as you can see from the attached logs. Here a quick summary for your analysis/review.

• In the input-validate function the event object has not the template value in it, while that is present in the parameters data ("JobTemplate": "Company-XX_VOD_50fps")

• that value ("JobTemplate": "Company-XX_VOD_50fps") is still present in the vod-profiler function parameters data.

• that value ("JobTemplate": "Company-XX_VOD_50fps") is still present in the vod-profiler function parameters data.

• in the profiler function code (line 73), the original code seems to search the value in the event.jobTemplate , i.e. if (!event.jobTemplate)

  console.log("EVENT\n" + JSON.stringify(event, null, 2));
  // Update:: added support to pass in a custom encoding Template instead of using the
  if (!event.jobTemplate) {
  // Match the jobTemplate to the encoding Profile.
  const jobTemplates = {
  '2160': event.jobTemplate_2160p,
  '1080': event.jobTemplate_1080p,
  '720': event.jobTemplate_720p
  };

So, before that line, I added some print code

//ff
console.log('ff - event jobTemplate before:: ', event.jobTemplate); <-- checking the value of the event object attribute, it is undefined

Question: why is that and how would we need to fix that compatibly with the original codebase?

Issue 2:

to overcome that for debugging purposes and for testing our template programmatically, I added the following code in the encode function before the if (!event.jobTemplate) :
event.jobTemplate = 'Company-XX-Vod-Hd'

when I hard-code the template in the encode function, that finally uses the right template but then I get the below error: "The specified preset was not found: presetArn=arn:aws:mediaconvert:ap-southeast-2:160924941480:presets/VOD_Mp4_Avc_Aac_16x9_1920x1080p_24Hz_6Mbps_qvbr"

Question is : in which function is specified to look for VOD_Mp4_Avc_Aac_16x9_1920x1080p_24Hz_6Mbps_qvbr and not look for the presets assigned to the new template? Please keep in mind that the same template works with the AWS console ui, so it would seem that the template association with the presets should be correct.

So, summarising, my questions are:

• how to pass properly a custom template in the metadata file.
• is that really working? Anyone has ever used that?
• what do you make of my investigation?
  Finally, I could fix that, but I would like to keep the current codebase without any (or minimal) changes. according to my investigation that seems necessary. any advice on this?

how do we need to manage the presets for custom templates? Ideally, we would create at least 2 custom templates

2019-09-18T04:57:50.146Z 82bc3222-4094-46e4-98ea-6498c4a93cf0 { NotFoundException: The specified preset was not found: presetArn=arn:aws:mediaconvert:ap-southeast-2:160924941480:presets/VOD_Mp4_Avc_Aac_16x9_1920x1080p_24Hz_6Mbps_qvbr.
at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
message: 'The specified preset was not found: presetArn=arn:aws:mediaconvert:ap-southeast-2:160924941480:presets/VOD_Mp4_Avc_Aac_16x9_1920x1080p_24Hz_6Mbps_qvbr.',
code: 'NotFoundException',
time: 2019-09-18T04:57:50.145Z,
requestId: '9fab83da-0777-450d-a0e8-7ba8adbef9e3',
ticket1.txt
ticket1.1.txt

[ffiore1]

@dscpinheiro @shsenior @rromanchuk @arivera-skycred @georgebearden

any advice? your experience?

[dscpinheiro]

Hi @ffiore1 ,

Unfortunately the README documentation is incorrect.

In order for the metadata options to overwrite the defaults, they must be in camelCase format (instead of JobTemplate, you should use jobTemplate):

{
    "srcVideo": "my-video.mp4",
    "archiveSource": false,
    "frameCapture": false,
    "jobTemplate": "my-template"
}

In the next release, we’ll update the documentation to point that out.

Could you let us know if, after making that change, your metadata file is applied correctly?

[rromanchuk]

For visibility responding here too. For the ignored metadata i just normalized the keys, per #23

I also renamed the default job templates that are created along with my presets in custom-resource/media-convert You can get the idea here https://gist.github.com/rromanchuk/4a38be8f3f03d82b68e0600ba1e89b23

I have a pretty crazy fork of this solution that extends this solution even further in terms of my production architecture, but I had to make the fork private because I needed to abstract out internal modifications to the cloudformation template.

My deployment does everything including

• adding/updating/removing my job templates/presets, that are now version controlled in the repo.
• attaching auto scaling to dynamodb
• Adding a global secondary index to srcVideo on dynamodb
• attaching a SNS https topic to a webhook on new .mp4/.mov creation
• attaching a sns https topic to webhook on pipeline state change
• Changing the profiler to pick the correct template from portrait orientation media sources
• Attaching ACM cert SNI for custom r53 subdomain on cloudfront
• rewriting/normalizing the payload
• Creating end-to-end deployment script ./deploy-development.sh 1.1 ./deploy-production.sh 1.1 that exist on two different regions

[ffiore1]

Hi @ffiore1 ,

Unfortunately the README documentation is incorrect.

In order for the metadata options to overwrite the defaults, they must be in camelCase format (instead of JobTemplate, you should use jobTemplate):

{
    "srcVideo": "my-video.mp4",
    "archiveSource": false,
    "frameCapture": false,
    "jobTemplate": "my-template"
}

In the next release, we’ll update the documentation to point that out.

Could you let us know if, after making that change, your metadata file is applied correctly?

yes, that fixed it. thanks @dscpinheiro

[ffiore1]

For visibility responding here too. For the ignored metadata i just normalized the keys, per #23

I also renamed the default job templates that are created along with my presets in custom-resource/media-convert You can get the idea here https://gist.github.com/rromanchuk/4a38be8f3f03d82b68e0600ba1e89b23

I have a pretty crazy fork of this solution that extends this solution even further in terms of my production architecture, but I had to make the fork private because I needed to abstract out internal modifications to the cloudformation template.

My deployment does everything including

• adding/updating/removing my job templates/presets, that are now version controlled in the repo.
• attaching auto scaling to dynamodb
• Adding a global secondary index to srcVideo on dynamodb
• attaching a SNS https topic to a webhook on new .mp4/.mov creation
• attaching a sns https topic to webhook on pipeline state change
• Changing the profiler to pick the correct template from portrait orientation media sources
• Attaching ACM cert SNI for custom r53 subdomain on cloudfront
• rewriting/normalizing the payload
• Creating end-to-end deployment script ./deploy-development.sh 1.1 ./deploy-production.sh 1.1 that exist on two different regions

@rromanchuk
yes, I think I am also going to add autoscale and indexes plus adding additional sns topic.

My idea would be to minimize changes to the stack so that I can update it quickly, if that is updated for new features. I am going to create a kind of wrapper around this stack to integrate with another solution, so probably I will not use r53 custom domain / cert, but surely that is very nice/useful change. I may change my mind on that....

By the way @dscpinheiro, helpful the heads-up for the metadata/file trigger and the manual update of the stack, i.e. creating manually the S3 event for the json when switching from file to metadata trigger. In my case, I had to discover that before I found/read the hint. Probably, the note was added after I had started playing with the solution (or did not read it... lol).

Question @dscpinheiro: do you have a roadmap for this stack? are you still going to work on this in the next months to add new features? just to understand how to approach it and how much customize or possibly keep minimal changes.

[dscpinheiro]

Hi @ffiore1,

Yes, we do plan on working on this solution in the next few months. Our main priority for the next release is to add support for MediaPackage VOD and address any outstanding issues (such as the metadata file not overwriting the defaults).

If you have ideas or features that would be useful, please create issues for them so we can discuss.

[ffiore1]

@dscpinheiro thanks for your reply.

can you clarify what you mean by "add support for MediaPackage VOD"?

do you mean get the output of a MP vod job and convert it? or get the output of a MP endpoint and convert it?

I have just finished implementing a MP live2vod solution, using a MP endpoint. Already deployed it but could improve it if MC can accept MP feeds.

[dscpinheiro]

Hi @ffiore1,

I know this issue has been closed, but I noticed I skipped one of your questions, so I'd like to answer it:

how do we need to manage the presets for custom templates?

If you want to use custom presets, you have to create them separately (either from the console or the CLI / API). You can obviously use some of the existing items as a reference.

Please let us know if you need any more information.