VapourSynth/video_output.c: populate alpha make_frame functions earlier

lsmas used to relying on new_output_video_frame to call determine_colorspace_conversion
to set vs_vohp->make_frame[output_index] for output_index 1 at get_frame time.

However, the guard to determine_colorspace_conversion in new_output_video_frame is
input argument input_pix_fmt_change, which does not take output_index into account.

The end result is that, if the alpha frame is requested before the main clip frames,
then new_output_video_frame notices that input_pix_fmt_change is true, and calls
appropriate functions to set the make_frame function for the output_index 1.

However, if the main clip frame is retrieved first, then new_output_video_frame will
cal determine_colorspace_conversion for output_index 0, which sets results in the flag
vshp->frame_prop_change_flags being cleared (as update_scaler_configuration_if_needed
determined that vshp already matches av_frame's properties.

You might wonder why does the code work if the request order is reversed (alpha first
and then main)? Good question: it is because during vs_setup_video_rendering,
determine_colorspace_conversion is already called for the main clip.

To fix this convoluted mess, we make vs_setup_video_rendering also call
determine_colorspace_conversion for output_index 1 (the alpha clip) as well.

Fixes #8.

Signed-off-by: akarin <[email protected]>

VapourSynth/video_output.c

@@ -968,12 +968,18 @@ int vs_setup_video_rendering
 {
     vs_video_output_handler_t *vs_vohp = (vs_video_output_handler_t *)lw_vohp->private_handler;
     const VSAPI *vsapi = vs_vohp->vsapi;
-    enum AVPixelFormat output_pixel_format;
+    enum AVPixelFormat output_pixel_format, alpha_pixel_format;
     if( determine_colorspace_conversion( vs_vohp, 0, ctx->pix_fmt, &output_pixel_format ) )
     {
         set_error_on_init( out, vsapi, "lsmas: %s is not supported", av_get_pix_fmt_name( ctx->pix_fmt ) );
         return -1;
     }
+    if( av_pix_fmt_desc_get( ctx->pix_fmt )->flags & AV_PIX_FMT_FLAG_ALPHA &&
+        determine_colorspace_conversion( vs_vohp, 1, ctx->pix_fmt, &alpha_pixel_format ) )
+    {
+        set_error_on_init( out, vsapi, "lsmas: %s's alpha format is not supported", av_get_pix_fmt_name( ctx->pix_fmt ) );
+        return -1;
+    }
     vs_vohp->direct_rendering &= vs_check_dr_available( ctx, ctx->pix_fmt );
     int (*dr_get_buffer)( struct AVCodecContext *, AVFrame *, int ) = vs_vohp->direct_rendering ? vs_video_get_buffer : NULL;
     setup_video_rendering( lw_vohp, SWS_FAST_BILINEAR,