use html5 video tag for animations

[sagetrac-whuss]

This ticket is about adding support for creating Ogg Theora or WebM videos from animation objects. The resulting video files are embedded into the notebook using the HTML 5 video tag.

The show method of animations now works more like the one of Graphics objects. Animations can now be embeddend in html fragments, for example using html.table(). By default animations are still created as animated GIFs. To get a modern video from an animation "a" use one of these:

a.show(mimetype="video/ogg")
a.show(format="webm")

The original ticket was designed to use libtheora and libogg from #7297 and only supported Theora. The currently attached branch builds on FFmpeg and supports a variety of formats, including Theora and WebM. Support for calling FFmpeg is already in place, so the focus here is on user interface, mainly integration with the show method and using HTML 5 video tags.

CC: @sagetrac-mhampton @nilesjohnson @novoselt

Component: graphics

Keywords: animation, video

Author: Martin von Gagern

Branch/Commit: ea5dfe2

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/7298

[sagetrac-whuss]

Attachment: ogv.patch.gz

[sagetrac-pang]

Reviewer: pang

[sagetrac-pang]

comment:4

The "iterations" optional argument does not work, but I don't think it's needed, as html5 has video controls. I'd suggest to either remove this argument, or change the documentation of 'ogv' from:

• iterations - integer (default: 0); number of
  iterations of animation. If 0, loop forever.

to:

• iterations - this argument is ignored, but kept for
  compatibility with the 'gif' function

Otherwise, everything's fine. The issue above is absolutely minor, so I'm giving a positive review.

[qed777]

comment:5

I've updated the Author(s) and Reviewer(s) fields with full names. Please correct them, if I'm wrong.

[qed777]

Changed reviewer from pang to Pablo Angulo

[qed777]

Merged: sage-4.5.2.alpha0

[qed777]

Author: Wilfried Huss

[qed777]

comment:6

By the way,

• Mercurial patches for Sage should include the ticket number in the first line of the commit string.
• The patch filename should include the ticket number. A common form is trac_7298-add_ogv_format.patch.

[qed777]

comment:7

I'm getting a docbuild warning from sage/plot/animate.py:

:0: (WARNING/2) Literal block expected; none found.

I think the problem is that the ogv method's docstring ends with

        .. note::
           If libtheora is not installed, you will get an error
           message like this::

Could someone please fix this? For now, I'm removing attachment: ogv.patch from 4.5.2.alpha0.

[qed777]

Changed merged from sage-4.5.2.alpha0 to none

[sagetrac-whuss]

only apply this patch

[sagetrac-whuss]

comment:8

Attachment: trac-7298-ogv.patch.gz

Replying to @qed777:

I'm getting a docbuild warning from sage/plot/animate.py:

:0: (WARNING/2) Literal block expected; none found.

I think the problem is that the ogv method's docstring ends with

        .. note::
           If libtheora is not installed, you will get an error
           message like this::

Could someone please fix this? For now, I'm removing attachment: ogv.patch from 4.5.2.alpha0.

I added the error message to the documentation in the new patch attachment: trac-7298-ogv.patch.

[sagetrac-git]

Changed commit from 2f6a670 to 025c6c1

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

025c6c1

Trac #7298: Enable HTML5 video in the Sage Notebook

[gagern]

comment:62

Yet another rebase. Perhaps it can be merged for Sage 6.7?

Right now, I support videos if the user explicitely requests a given format. When no format is specified, animated GIF remains the only option. I know Volker doesn't like it if people have to specify such low level stuff like file formats. And I wouldn't mind some sane automatic choices based on requests of a higher level, e.g. requests for full-color output or whatever. But getting that to work is far from trivial, so I'd ask you to postpone any fancy automatic format detection to a follow-up ticket, so that we can get the basic support in without getting lost in the details of fancy automatisms.

Right now, I don't support APNG as a format for show yet. #16650 should take care of that once this here gets merged. There might be some controversy ahead that way, so again I consider it best to get this bit here in before expanding on it.

Right now, I only support Sage Notebook, since that is all I use when creating animations. While I could test small examples using command line or the IPython notebook, I'm not using either on a regular basis, so I might be missing important aspects of how they operate. I'd prefer to merge this support for what I know, and postpone support for other backends. I have some ideas there, but I'd be asking more input from users of said backends, which could possibly delay things somewhat.

So please try to review this change with the scope I chose, keeping in mind that this is but the first step in a direction aiming for more high-level requests, more consistent support for various formats, and more support from various backends other than the Sage notebook. Feel free to create follow-up tickets if you want to start a discussion for one of these subsequent steps. If this gets merged early in the 6.7 cycle, we might get some of the other things in as well.

[vbraun]

comment:63

I don't mind it if the initial ticket only supports SageNB. But OutputVideoAny is just a slap in the face of any attempt at sanity. The whole point of the rich output model is to separate different output types. The rich output container must not contain anything but the data. No mime types, no file extensions. The type is unique to the output. By lumping it into a single OutputAllVideoThatSageNBunderstands you destroy any hope of ever supporting other notebooks without reverting your branch.

[gagern]

comment:64

Replying to @vbraun:

But OutputVideoAny is just a slap in the face of any attempt at sanity. The whole point of the rich output model is to separate different output types.

I can understand some of your concerns. To me, a supported output type is a promise. For most simple output types the promise is “if you give me a file of said type, I'll display it to the user”. But with video, it's not as simple. It's not the backend which decides what video formats are supported, but instead it's the browser or the system setup. So the promise I associate with OutputVideoAny is “if you give me a file of said type, I'll present it to a user in such a way that it either plays or the user gets notified about what went wrong”. Which in a browser context means “I'll create a HTML5 <video> tag for it”.

Due to this nature, OutputVideoAny should never be used to auto-guess supported formats. It doesn't mean “this client supports any video format”, but instead “this client can present any video format to the user, but that presentation might be in the form or an error message”. So the machinery shouldn't use it to throw arbitrary formats at the back end, but as a means to throw explicitly requested formats at it.

I wouldn't mind having specific formats. For video, a full file format description consists of a container format, a video codec format, and possibly a bunch of other constraints as well. So we'd have OutputVideoWebMVP8 and OutputVideoWebMVP9 and OutputVideoOggTheora and OutputVideoAviH264 and OutputVideoFlashH264 and OutputVideoMp4H264 and OutputVideoQuicktimeH264 and probably a hundred other combinations. (We could parse the list of formats FFmpeg supports.) And in my opinion, each of these would indicate a promise to actually support the given type. So we'd have to somehow detect support. Or state that such a type being listed with the supported types does not imply it's actually supported. I have no idea on how to do auto-detection, and having a bunch of maybe supported formats feels no different to my OutputVideoAny.

So do you have any suggestions of how to handle this situation in a better way? I understand your criticism from an idealistic point of view, but on the pragmatic side, unless one can do active format support detection, all formats are essentially the same to the backend, and a single representation for all of them avoids useless overhead.

The rich output container must not contain anything but the data. No mime types, no file extensions. The type is unique to the output. By lumping it into a single OutputAllVideoThatSageNBunderstands you destroy any hope of ever supporting other notebooks without reverting your branch.

It's not “output all video that Sage NB understands”. It's more like “output all video which can possibly be handled by a <video> tag”, combined with “output all video which gets opened by launching up the user's default media player”. For the <video> tag you need the MIME type. For the media player, you need the correct file extension. But if you consider the <video> element as the deliverable of the output, then the mime type is very much part of the data needed to assemble that.

One more comment: if there is one thing I can't stand about computer programs, it's if these try to be more clever than the user. The user should be in charge, and if the user says “I want to view this animation in that format”, the code should not second-guess that decision, should not forbid the operation just because the named format isn't in a list of known formats, or support by the browser or media player can't be verified up front. Please leave the users in charge of their application, not the other way round!

[vbraun]

comment:65

Supported means that we have code for trying to display that particular output type. There is not necessarily a guarantee that it'll work, you might be missing an external dependency (e.g. java). But if its unsupported then it can't possibly work because there is no code to do something with it.

At the same time you don't have to make a different output type for each sub-type, its ok to lump together formats that are likely to be supported at the same time (e.g. because they are common in some type of container). For example, there is no OutputImagePdf for each pdf version. But formats with a different mime type, say, aren't going to always be supported by the same player. In that sense their taxonomy is similar to mime types (but not exactly the same, e.g. support for multi-file output, jmol, ...) Just use your judgement. Is every video player going to be able to play animated gifs? I don't think so -> different output types.

On top of my head, I would suggest the following:

• OutputVideoAnimatedGif
• OutputVideoAnimatedPNG
• OutputVideoWebm
• OutputVideoMp4
• OutputVideoFlash
• OutputVideoQuicktime

[gagern]

comment:66

I fully agree with the Animated PNG format, that should be a type on its own. The OutputImageGif alreads includes animated GIF, according to its docstring. Should this be changed? Should the browser-based backends declare support for WebM, Ogg and MPEG4, even though the browser might be lacking support for either? Is there any way how a backend could, at some later time, indicate that indeed it does support a given video format?

[vbraun]

comment:67

I would prefer to make animated gif its own output type since many stand-alone image viewers will not show the animation. I just hadn't thought through the case of animations when I wrote it.

Ideally, browser-based backends would have some browser detection and then only return the actually supported output types. Though right now this is not implemented. Lacking that, you should return everything that a modern browser would support.

[gagern]

comment:68

Is Internet Explorer 11 a modern browser, even though it lacks WebM support? If you are tempted to answer this “no” and instead opt for MPEG-4 AVC as the codec with best cross-browser support, then please keep in mind that use of that codec might make the resulting content patent-encumbered, so I would very much like to keep at least one free alternative around. So for pragmatic reasons I'd declare recent releases of Chrome and Firefox as the only modern browsers here, and thus pretend that all web browsers support all three formats mentioned above. Or in other words, I'll go with the fact that the formats might be supported, which is all my OutputVideoAny approach did in any case. I'll adapt the code along these lines soon.

[vbraun]

comment:69

Fine with me. I'm not saying that it should behave differently than your OutputVideoAny, I'm just asking you to make it possible for other backends to rely on the output types.

[sagetrac-git]

Changed commit from 025c6c1 to 00e1b48

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

00e1b48

Trac #7298: Implement output types for various video container formats

[vbraun]

comment:71

Thanks, looks much better.

I still dislike that HTML 5 tag attributes are attached to the rich output types, that makes no sense outside of a browser. The settings in question:

• autoplay: Pointless, you ran animation.show() so obviously you want to see the animation
• controls: Who would ever want no video controls? Hard to imagine any use case for that on a scientific data visualization platform.
• loop: Seems desirable
  So IMHO we should only have a loop=True optional keyword argument on the video output types, and no attrs={} dictionary.

[gagern]

comment:72

Replying to @vbraun:

Thanks, looks much better.

Glad you say so. I'm still a bit concerned by the fact that a format supported by backend and FFmpeg still won't be supported only because it's not in our list.

I'm slightly worried by the fact that we save the video to some temporary file, only to copy it to the current working directory in a subsequent step, where SageNB can pick it up. This feels kind of wasteful. As far as I see, the temporary file used for the rich output never gets deleted (if this is wrong, where is the code doing that), which means a long-running SageNB process will accumulate temporary files without association to notebook cells, and might exhaust some tempfs due to this. Should this be a separate ticket?

I still dislike that HTML 5 tag attributes are attached to the rich output types, that makes no sense outside of a browser.

Well, most of the attributes map to other players as well. To support that claim, take e.g. xine which has -Z to suppress autoplay, --hide-gui and --no-gui to disable controls, and --loop or perhaps --loop=repeat to loop. So while the names of the attributes were taken from HTML5, the semantics can be found elsewhere as well.

The main problem is that each viewer has a different set of command line arguments to represent these options, and as long as we open stuff through xdg-open (which I consider the most likely approach for BackendIPythonCommandline on Linux), we have little chance of actually supporting any of this outside the browser in the near future.

The settings in question:

• autoplay: Pointless, you ran animation.show() so obviously you want to see the animation

Use case: Creating the animation takes five minutes, so you work in some other window while Sage gets its act together. When you switch back you want to be able to view the animation from the beginning using a single click. Or perhaps you have a loop creating several animations, and want to show them one after the other. Controlling when an animation starts may be particularly important in an environment where you have an audience, and some effect visible in the animation which you don't want to divulge prematurely.

• controls: Who would ever want no video controls? Hard to imagine any use case for that on a scientific data visualization platform.

I guess you are right there. I guess the main reason to have that in HTML is when you have your own controls written in JavaScript. Not something we need to worry about.

• loop: Seems desirable

Mapping loop to e.g. the IPython command line backend sounds like it's going to be really tricky, but I think we simply have to live with the fact that some backends won't support some of the features.

So IMHO we should only have a loop=True optional keyword argument on the video output types, and no attrs={} dictionary.

If I could convince you with the autoplay use cases, should I add two keyword arguments? I guess so.

[vbraun]

comment:73

Temporary files are always a kludge, which is why the OutputBuffer is designed around in-memory buffers. In general there is no guarantee that the compute kernel and the notebook server share a common file system. Like all temporary files, they are eventually deleted by the separate sage-cleaner process.

There is a technical solution to not playing stuff in hidden browser tabs: http://caniuse.com/#feat=pagevisibility. No need to force the user to do it by hand.

If you want to show multiple animations one after the other then you really want a single animation with the frames concatenated. Presumably you can add animation objects...

If you don't want the audience to see the animation then don't evaluate the cell that shows the animation. Thats about equally disruptive in a presentation as having to click on something to play the animation. If the UI has a special "slides" mode (like the IPython notebook) then it should handle

[sagetrac-git]

Changed commit from 00e1b48 to ea5dfe2

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a8d97b7	Trac #7298: Enable HTML5 video in the Sage Notebook
f6e6412	Trac #7298: Implement output types for various video container formats
ea5dfe2	Trac #7298: Drop support for the autoplay and controls attributes for video

[gagern]

comment:75

Rebased, and incorporated latest review comments in that last commit. Is this ready for merge now?

Perhaps in the long run we should have something like a prepare method for animation objects, which creates the video file without showing it, and on which one can call show or save later on, with minimal cost. But I'll leave that idea for later.

[vbraun]

Reviewer: Volker Braun

[gagern]

comment:77

Thanks for the review. Now I can build on this for #16650.

[vbraun]

Changed branch from u/gagern/ticket/7298 to ea5dfe2