add gatsby-transformer-video (beta available)

[axe312ger]

This integrates fluent-ffmpeg with Gatsby 🎥

• A beta is already released: npm i gatsby-transformer-video
• Docs: https://github.com/gatsbyjs/gatsby/tree/gatsby-transformer-video/packages/gatsby-transformer-video
• Example: https://github.com/gatsbyjs/gatsby/tree/gatsby-transformer-video/examples/using-gatsby-transformer-video

Features

• Source: Works with nodes from gatsby-source-filesystem and gatsby-source-contentful
• Defaults optimized for small files with decent quality and quick and seamless streaming
• Supported codecs: h264, h265, VP9, WebP & gif
• Several parameters to tweak the output: maxWidth/maxHeight, overlay/watermark, saturation, duration, fps, ...
• Create video conversion profiles. Create a converter function using fluent-ffmpeg to unlock all FFMPEG features.
• Downloads binaries from official sources when FFMPEG is not (properly) installed
• Take screenshots at any position of the video
• Multiple cache layers to prevent accidental reconversion (Discussion see below)
• Example project to demonstrate the usage

Todos:

• Find somebody with extended FFMPEG knowledge and verify our default profile settings: https://github.com/gatsbyjs/gatsby/tree/gatsby-transformer-video/packages/gatsby-transformer-video/src/profiles
• Videos converted via the h265 profile work fine on iOS & in VLC, but Safari & Quicktime struggle to play the files
• Overlays have fixed size, FFMPEG can resize them to better fit into the video. We should support this
• We currently cache into .node_moduels/.cache and queue our FFMPEG executions on our own. This is just a workaround as Gatsby has no system to queue such time consuming transformations. We as community should implement an API that allows all plugins play well next to gatsby-transformer-sharp, this video transformer & others.
• Write tests
• ... There is more, look in the discussion below...

[me4502]

Generally GIFs should never be used on the web - https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/replace-animated-gifs-with-video/

[DSchau]

@me4502 while I'd agree, exposing gif nicely enables an experience where video elements are preferred and we can fall back to gif as a last resort. (like in a picture/source stack as the example shows).

So... noted, but I think it's reasonable to provide it so we could create e.g. gatsby-video component or something.

[axe312ger]

Currently thinking of providing one field per codec. Opens up flexibility and would remove that preview field.

That way users could generate a gif which way less fps as the other previews. Or just simply provide a jpg as preview.

[axe312ger]

Did not try it out yet but the following should speed up the playback start delay:

Add -movflags +faststart. This will relocate the moov atom from the end of the file to the beginning. This will allow the video to begin playback while it is still being downloaded. Otherwise the whole video must be completely downloaded before it can begin playing.

Source: https://stackoverflow.com/a/25573078

[axe312ger]

FYI: planning to work on this on christmas holidays :)

[me4502]

I personally have been pre-encoding all videos with the recommendations from here, https://gist.github.com/mikoim/27e4e0dc64e384adbcb91ff10a2d3678

These seem to perform super well, but the benefits of this plugin to allow better sizing / multiple formats will be amazing! :)

[axe312ger]

So my idea is to refactor this till end of the year, including all the given feedback and especially:

Currently thinking of providing one field per codec. Opens up flexibility and would remove that preview field.

So instead of a video && videoPreview field with the codecs as subfields, I'd create a field per codec/file-type, supporting codec/file-format related attributes + easy support for fragments.

To use the current version of the plugin you would do something similar like:

fragment BlogPostData on BlogPost {
  ...
  mediaFieldWithVideoData {
    # convert source video into optimized for web version with default settings
    video {
      h264
      h265
    }
    # Create a preview version of the video with reduced dimension, speed up and very limited frames per second
    videopreview(width: 600, fps: 4, duration: 3) {
      mp4
      webp
      gif
    }
  }
}

While the refactored way I propose would allow you way more fine granular configuration. Which is necessary as video conversion is complex and the settings depend on a lot of content and project specific factors.

fragment BlogPostData on BlogPost {
  ...
  mediaFieldWithVideoData {
    videoH264 (maxBuffer: '2M', ...) {
      path
      size
    }
    videoH265 (maxBuffer: '1M', fancyH265Setting: 'bar', ...) {
      path
      size
    }
    previewMp4: videoH264 (desaturate: true, width: 600, fps: 8, duration: 3) {
      path
    }
    # Note that we can give the gif way smaller dimensions to reduce preview file size for clients without mp4 support
    previewGif: videoGif (desaturate: true, width: 300, fps: 4, duration: 3) {
      path
    }
    ...
  }
}

[muescha]

muescha unassigned KyleAMathews 2 days ago

sorry Kyle - this must done by mistake - i don't remember doing something like this :(
i reassigning you

[axe312ger]

So refactoring is done to have one field per codec. Additionally it uses the new schema customisation API and removes all code-smells from the last reviews.

This plugin now adds the following subfields to all file nodes:

• videoH264
• videoH265
• videoGif
• videoWebP

Next steps:

• Apply recommendations from above (tweak FFMPEG)
• Add VP9 codec support
• Add videoPreset which executes a configurable function to execute fluent-ffmpeg on your own
• Posterimage feature
• Improve logging && implement node-debug for detailed output

[axe312ger]

I implemented VP9 and some performance & quality improvements for h264/h265.

Thanks to https://developers.google.com/media/vp9/settings/vod/#bitrate we can even automatically determine a fitting bitrate based on our source data and given field args (like maxWidth)

The generated files now have a decent (not super high) quality, small file size and the file sizes differ as expected based on coded.

• source mov: 15,7 MB (720p)
• h265: 1.5mb
• VP9: 3.5mb
• h264: 4.9mb

I won't claim the settings are perfect but its a clear improvement to the first version :)

[muescha]

some suggestions

[axe312ger]

Crunching more numbers:

I got 7 source videos, mostly 720p from iPhone cameras or final cut, converting it in 3 target videos including 3 additonal previews.

It takes about ~16 minutes to convert all videos on my i9. (query below)

Gatsby cloud took 1h 5min to convert all of it.

Restarting the build with the same cache, 30 seconds build time 💪 😍

query ExampleQuery {
  contentfulAsset(file: {contentType: {regex: "/^video/"}}) {
    title
    file {
      contentType
      fileName
      url
    }
    videoH264 {
      path
    }
    videoH265 {
      path
    }
    videoVP9 {
      path
    }
    previewH264: videoH264(maxWidth: 560, fps: 4, duration: 2, saturation: 0) {
      path
    }
    previewWebP: videoWebP(maxWidth: 560, fps: 4, duration: 2, saturation: 0) {
      path
    }
    previewGif: videoGif(maxWidth: 280, fps: 4, duration: 2, saturation: 0) {
      path
    }
  }
}

[axe312ger]

Next refactoring is done, took everything @muescha mentioned into account.

The code should be more flexible and readable as well as I moved the codec specific code into a subdirectory, one file per codec/profile

https://github.com/gatsbyjs/gatsby/tree/gatsby-transformer-video/packages/gatsby-transformer-video/src/profiles

Implementing the custom profile function should now be pretty low efford :)

I will do a complete local run on my machine and publish v0.1.1 afterwards.

[axe312ger]

I cleaned up the code and applied several fixes and improvements we stumbled upon while using v0.1 in actual projects.

0.2.0 (2020-11-02)

Build System / Dependencies

• upgrade dependencies (2457d172)

Chores

• remove debug console.log (7ee4556d)

Documentation Changes

• mark as beta and link usage example (8438fccd)

New Features

• add additional information when video could not be analyzed cus of a broken or unsupported file (a70ea950)
• add width, height and aspect ratio to video nodes (e1370884)
• align file naming for files and contentful assets (240eefe9)
• clean up logging and reporting (91d50ec7)
• slightly increase default quality for h264 and h265 (4b05c482)
• move default directory into static folder (767e75f9)
• use new reporter API (dd80e1aa)
• move screenshots to extra videoScreenshots resolver (242e99ea)
• switch to medium preset for h264 and h265 for higher quality, smaller size and almost same conversion time (370c0d07)

Bug Fixes

• redownload files if actual file on Contentful changed (53909a2e)
• ensure screenshots are queued (c61df862)

Refactors

• adjust example to match new screenshots resolver (73f67b33)
• update babelrc to use babel-preset-gatsby-package (c7371733)

Code Style Changes

• reapply eslint (1c21a16f)

[axe312ger]

Planning on a v0.3 release that supports Gatsby v3, is less spammy in the logs and actually caches screenshots/poster images

[ascorbic]

@axe312ger Would it make sense to move this into its own repo? It's not going to be merged any time soon, and publishing releases from a branch isn't great practice.

[axe312ger]

@ascorbic yeah maybe we should do that.

But I was hoping to not end up alone as maintainer 😇

[wardpeet]

cc @KyleAMathews what do you suggest here? I rather not add it to the monorepo as we don't have any knowledge or time to dig into this. We also don't add ffmpeg to cloud so this wouldn't work there.

[axe312ger]

We also don't add ffmpeg to cloud so this wouldn't work there.

it does work, just needs to download a 90mb zip with two binaries 🙈

[KyleAMathews]

I love the idea of this but... if image processing trips up a lot of people using Gatsby, well, video processing is like the sun to image processing's little match. It's immensely expensive & I don't think people would have a good experience using a video transform plugin. I think most people would be much better off using a dedicated service for video hosting. There's some sites for sure where this transformer would be great but I don't think the number is enough that we'd want to make it an official project.

[ascorbic]

Yes, and if sharp causes people issues, ffmpeg is another level. This is a great plugin, but making it official would take a lot of work, including fixing caching.

[axe312ger]

This is a great plugin, but making it official would take a lot of work, including fixing caching.

Fixing/improving the cache would be very helpful, for this plugin, but also for others 🙃

[axe312ger]

I'll move this out of this repo and make a community plugin out of it as soon I find time for it.

When doing this, I'll also rewrite to Gatsby v3 and maybe to TS.

[ascorbic]

There's now an experimental flag to enable retaining downloaded assets when auto-cleaning. I wonder if that will be enough to let this use the regular asset cache?

[axe312ger]

Sounds good. You will hear from me for sure when I move this code 😅

[axe312ger]

Hello everyone,

I moved the code of this PR to its own repository: https://github.com/hashbite/gatsby-transformer-video
All open ToDos and known bugs are moved into issues: https://github.com/hashbite/gatsby-transformer-video/issues

Also v0.3.0 got released. It just bumps the dependencies for Gatsby v3. Works fine in my first tests.
More features and bug fixes will follow as soon I find time, especially the screenshot bug is annoying.

Hope to get some collaboration going in the new repo, see ya over there 👋