Feature Request: (RIFE) Please add support for rational frame rate changes (like vs-rife has)

[Samhayne]

Unfortunately multi is an int.

vs-rife replaced multi with factor_num, factor_den with version 3.0.0 to support rational frame rate changes.

see: https://github.com/HolyWu/vs-rife/releases

[hooke007]

Unfortunately, the feature in 2f6d598 is not as safe as factor_num, factor_den in rife-ncnn-vk.
If I tried with mpv, it would used up memory and make my program frozen.

[WolframRhodium]

How to reproduce it in vsedit/vs-preview?

[hooke007]

I cannot. I remembered holywu once pushed some commits too in his original repo to fix the similar issue. But the related issues are gone by deleting his repo.

[WolframRhodium]

Since we have used it successfully in encoding, without reliable reproduction we can not validate the claim.

This repository should contain the relevant commit. Do you mean this one?

[hooke007]

It is included i think.

[WolframRhodium]

Why do you think fps_num is better? It is equivalent to multi=target.fps / source.fps for CFR videos, and I don't think that implementation handles VFR videos.

[hooke007]

No, i never talked about fps_num...
To make it clear, factor_num=5 factor_num_den=2 works fine in rife-ncnn-vk. multi=Fraction(5,2) makes the program frozen.

[WolframRhodium]

Then factor_num and factor_den, which is equivalent to multi=Fraction(factor_num, factor_den), isn't it?

multi=Fraction(5,2) makes the program frozen.

I can't reproduce that with vsedit or vs-preview.

[hooke007]

I can't reproduce that with vsedit or vs-preview.

I knew, that's why I said But the related issues are gone by deleting his repo.
The more details explaination are only recorded in those deleted issue.

[WolframRhodium]

Then we can do nothing.

And we are using different implementation here compared to the original approach.

[aloola18]

multi=Fraction(5,2)

I can confirm this, I tried with MPV and MPC, and both froze. MPC sometimes froze for ~3 minutes and then ran successfully but it froze most of the time.

[WolframRhodium]

but it froze most of the time

Could you provide more information? For example, does the freezing occur at the same frame? Is it related to scene change, and what if you disable scene change detection?

[aloola18]

but it froze most of the time

Could you provide more information? For example, does the freezing occur at the same frame? Is it related to scene change, and what if you disable scene change detection?

It froze when opening any video file or when seeking(when it ran successfully).
it's eating a lot of RAM while freezing

I disabled scene change detection, still froze.

[WolframRhodium]

Are you using SVP with "Duplicate frames removal" enabled?

By "opening any video file", do you mean it does not return any frame at all?

[aloola18]

yeah, just a black screen and froze.

no, I'm not using SVP.

[WolframRhodium]

Does the frozen time depends on the number of frames of the video?

[aloola18]

Does the frozen time depends on the number of frames of the video?

seems so, video > 20 minutes took forever to load.

[WolframRhodium]

That's a new information. What if you change your script to the following:

• before:

  output = vsmlrt.RIFE(xxx)

• after:

  output = vsmlrt.RIFE(xxx).std.BlankClip(color=[0.5] * 3)

Does the playback still froze?

[aloola18]

cut2 = RIFE(cut1, multi=Fraction(5,2), model=46, backend=BackendV2.TRT(force_fp16=True, num_streams=2, output_format=1, static_shape=True)).std.BlankClip(color=[0.5] * 3)

video still froze

[WolframRhodium]

Do you know Python? I want to know which line of code in the RIFE() function of vsmlrt.py blocks.

This can be checked by inserting a line return clip before each function calls (i.e. line 1147, 1164-1167, 1169, 1176-1179). With this change the original video is returned, but some insertions may not have immediate display.

[aloola18]

I inserted return clip into:
line 1146: video play normal w/o interpolation
line 1163: froze about 5 seconds then started playing normally w/o interpolation
line 1168: video froze

so these lines cause issues?

left_clip = core.akarin.PickFrames(clip, left_indices)
right_clip = core.akarin.PickFrames(clip, right_indices)
tp_clip = core.std.BlankClip(clip, format=gray_format, length=len(timepoints))
tp_clip = tp_clip.akarin.PropExpr(lambda: dict(_tp=timepoints)).akarin.Expr('x._tp')

[WolframRhodium]

line 1163: froze about 5 seconds then started playing normally w/o interpolation

This is alarming because it should not froze.

In the original program, what if you repeat line 1142-1162

vs-mlrt/scripts/vsmlrt.py

Lines 1142 to 1162 in 2f6d598

	left_indices = []
	right_indices = []
	timepoints = []
	output_indices = []
	for i in range(dst_frames):
	current_time = dst_duration * i
	if current_time % src_duration == 0:
	output_indices.append(current_time // src_duration)
	else:
	left_index = current_time // src_duration
	if left_index + 1 >= src_frames:
	# approximate last frame with last frame of source
	output_indices.append(src_frames - 1)
	break
	output_indices.append(src_frames + len(timepoints))
	left_indices.append(left_index)
	right_indices.append(left_index + 1)
	left_time = src_duration * left_index
	tp = (current_time - left_time) / src_duration
	timepoints.append(tp)

several times? Will each repetition increase the frozen time by 5 seconds?

[aloola18]

Will each repetition increase the frozen time by 5 seconds?

yes, it freeze for a very long time. around 30~40 seconds

[WolframRhodium]

That's a very useful information.

What if you simply call len(timepoints) multiple times in the loop without assignment, does it affect performance?

Alternatively, insert the following piece of code

with open("num_frames.txt", "w") as f:
    f.write(str(clip.num_frames))

This will create a file. What's the value in it?

[aloola18]

called len(timepoints) 5 times
took 10 seconds to play a 90 seconds video
25s for 20 minutes video

What's the value in it?

10810800

[WolframRhodium]

10810800

Thanks, now I fully understand what happens.

Video playback is different from encoding in that the player will create a clip with a large number of frame, while for encoding the value is at most hundreds of thousands. This make the initialization much more demanding for vudeo playback.

I will start to fix the issue now.

[hooke007]

Tried (5,2)
Looks like duration is wrong. It feels like turing down the playback speed.

[WolframRhodium]

This line is necessary to prevent integer overflow when multi > 16, however.

[hooke007]

No, I mean the version I tested had the problem. Not this line caused the problem.
I also tested the inital version, it also had the same issue.

[WolframRhodium]

What about multi=2? Will it slows down the video?

[hooke007]

2 works fine.

[WolframRhodium]

What if you change this line to src_fps = Fraction(24, 1) or other values?

[hooke007]

Emmm, doesn't help.
mpv has a quirk on fps and duration.

For example, the original change_fps function doesn't work correctly in mpv.
I made a little bit tweak to make it match the duration properly.

So I finally used a workaround below.

...
clip = RIFE(...multi=Fraction(5, 2), video_player=True...)
clip = clip.std.AssumeFPS(fpsnum=5 * container_fps * 1000, fpsden=2 * 1000)

[netExtra]

What difference does setting video_payer=True make?

[WolframRhodium]

Allows the fractional RIFE to be usable in video players.

[netExtra]

Allows the fractional RIFE to be usable in video players.

So won't make any difference with mpv using svp? I've tried it by the way but didn't see anything. I assume SVP is not using fractional rife. Sorry one more question, how does fractional rife help?

[aloola18]

Allows the fractional RIFE to be usable in video players.

So won't make any difference with mpv using svp? I've tried it by the way but didn't see anything. I assume SVP is not using fractional rife. Sorry one more question,

SVP has its method to handle Fraction and it works very well. I don't know if this will conflict with SVP though.

how does fractional rife help?

like if you want 24fps video to 60fps, then use a fraction(5, 2).

clip = RIFE(...multi=Fraction(5, 2), video_player=True...)
clip = clip.std.AssumeFPS(fpsnum=5 * container_fps * 1000, fpsden=2 * 1000)

this works well for me. btw do you know any way to improve MPV seek time when using RIFE? MPV reload vapoursynth on every seeking, makes it slow compared with MPC-HC which instantly on seeking.

[hooke007]

btw do you know any way to improve MPV seek time when using RIFE?

IMO, This requires an expert to rewrite the implement of vs in mpv. Any heavy filters suffer the similar pain.

[WolframRhodium]

No difference and conflict with existing svp.

Yes svp should be rewritten.

[netExtra]

btw do you know any way to improve MPV seek time when using RIFE?

IMO, This requires an expert to rewrite the implement of vs in mpv. Any heavy filters suffer the similar pain.

Tbh I don't have major mpv seek issues any more. V1 with the akarin update is as good as V2 which has always been good.

[netExtra]

Allows the fractional RIFE to be usable in video players.

So won't make any difference with mpv using svp? I've tried it by the way but didn't see anything. I assume SVP is not using fractional rife. Sorry one more question,

SVP has its method to handle Fraction and it works very well. I don't know if this will conflict with SVP though.

how does fractional rife help?

like if you want 24fps video to 60fps, then use a fraction(5, 2).

clip = RIFE(...multi=Fraction(5, 2), video_player=True...)
clip = clip.std.AssumeFPS(fpsnum=5 * container_fps * 1000, fpsden=2 * 1000)

this works well for me.

Where do I place this code? I assume I have to replace some existing code?

[hooke007]

Tbh I don't have major mpv seek issues any more

Are you sure you can seek as fast as playing without svp? Or you do not think it's a 'major' problem here?

Where do I place this code?

I assume you were using svp, you have to wait for its support.

[aloola18]

btw I tested with multi = Fraction(3, 2) but it not working

3.18.18

[netExtra]

Tbh I don't have major mpv seek issues any more

Are you sure you can seek as fast as playing without svp? Or you do not think it's a 'major' problem here?

SVP with Rife v4.13(v2) At 1080p x 2 (3D) downscaled to 1080p with
catmull_rom dscale + antiring
lanczossharpest cscale,
correct and linear downscaling.
adaptive-sharpen shader

hwdec=no : 1080p or 4K is about a second or less between each seek.
hwdec=d3d11va-copy : 1080p seek is almost instant. See attached. The last couple of seeks are chapter seeks.

V_20231208_153338_ES3.mov

[hooke007]

btw I tested with multi = Fraction(3, 2) but it not working

Apparently the limitation is hardcoded in vsmlrt.

vs-mlrt/scripts/vsmlrt.py

Line 1066 in 9f527a9

if multi < 2:

@netExtra You didn't get the point here. It can only prove your hardware is strong enough to handle it. But it can not cover the design flaw. And it's off-topic I think. Nothing here can do to fix it.

[netExtra]

btw I tested with multi = Fraction(3, 2) but it not working

Apparently the limitation is hardcoded in vsmlrt.

vs-mlrt/scripts/vsmlrt.py

Line 1066 in 9f527a9

if multi < 2:

I don't know if I have this right?
multi: typing.Union[int, Fraction] = (5, 2),

But it works with no errors. Maybe even a little better than before because I was able to manually configure SC to fix the spider-man issue and get smooth enough slow pans. But it still causes stutters elsewhere so I'm setting it to 0.20 which if I understand correctly either disables SC or reduces its effect.

@netExtra You didn't get the point here. It can only prove your hardware is strong enough to handle it. But it can not cover the design flaw. And it's off-topic I think. Nothing here can do to fix it.

I thought you might say it was hardware related :)

[netExtra]

clip = RIFE(...multi=Fraction(5, 2), video_player=True...)
clip = clip.std.AssumeFPS(fpsnum=5 * container_fps * 1000, fpsden=2 * 1000)

this works well for me. btw do you know any way to improve MPV seek time when using
RIFE?

I tried the clip.std.AssumeFPS changes but I couldn't see any obvious differences for fast movement.

[hooke007]

I tried the clip.std.AssumeFPS changes but I couldn't see any obvious differences for fast movement.

It's mpv's workaround to fix duration. Svp probably did sth else. And it did nothing with fast movement.

[chainikdn]

Note that all these AssumeFps's will break videos with variable frame rate.
For the reference, what SVP does with a lot less lines of code:

smooth = RIFE(input, multi=frc_num, ...)
smooth = FixFrameDurations(smooth.std.SelectEvery(cycle=frc_den, offsets=0) if frc_den>1 else smooth, input, frc_num, frc_den)

... where

def FixFrameDurations(out, src, num , den):
    src0 = core.std.Interleave([src] * num)
    src1 = src.std.Trim(first=1)
    src1 = core.std.Interleave([src1] * num)
    if den>1:
        src0 = src0.std.SelectEvery(cycle=den, offsets=0)
        src1 = src1.std.SelectEvery(cycle=den, offsets=0)

    def set_duration(n, f):
        fout = f[0].copy()
        if not '_DurationNum' in f[1].props or not '_DurationNum' in f[2].props:
              return fout
        dur_rel = den / num
        tm = n * dur_rel - int(n * dur_rel)
        dur_src0 = f[1].props['_DurationNum'] / f[1].props['_DurationDen'] / dur_rel
        dur_src1 = f[2].props['_DurationNum'] / f[2].props['_DurationDen'] / dur_rel
        dur = dur_src0 * dur_rel if tm + dur_rel <= 1.0 else dur_src0 * (1.0 - tm) + dur_src1 * (tm + dur_rel - 1.0)
        fout.props['_DurationNum'] = int(dur*10000000)
        fout.props['_DurationDen'] = 10000000
        return fout
    res = core.std.ModifyFrame(clip=out, clips=[out,src0,src1], selector=set_duration)
    return res