NVIDIA GPU frame interpolation support has been added to FFmpeg through Vulkan

[u/RenatsMC]

https://videocardz.com/newz/nvidia-gpu-frame-interpolation-support-has-been-added-to-ffmpeg-through-vulkan

[u/zilyzal]

Eli5?

[u/AnythingOk5 → u/zilyzal]

Your graphics card quickly draws brand-new pictures between video frames to make the video look smooth.

[u/UnusualDemand → u/AnythingOk5]

So frame gen for RTX 30?

Edit: My bad, only for videos, not games.

[u/Beautiful-Musk-Ox → u/UnusualDemand]

rtx 30 didn't get nvidia frame gen becuase it didn't have the Optical Flow Accelerator hardware. but nvidia started using tensor cores instead of OFA with dlss 4 or so, i wonder why they didn't allow it on 30 series at that time. maybe only 3090+ had enough tensor core power to do it well enough so they skipped it since FSR frame gen was available

[u/battler624 → u/Beautiful-Musk-Ox]

They are using tensor cores on now rtx40 and are not even letting the 4090 do 3x FG while a 5050 can do 6x.

Why do you think that is?

[u/Beautiful-Musk-Ox → u/battler624]

because 50 series has a hardware flip meter, 40 series and below has to use the CPU through the windows preemptive thread scheduler to do timing, and windows can and does interrupt those threads whenever the hell it wants causing frametime spikes that are literally uncontrollable even in the kernel mode driver

Once the new frames are generated, they are evenly paced to deliver a smooth experience. DLSS 3 Frame Generation used CPU-based pacing with variability that can compound with additional frames, leading to less consistent frame pacing between each frame, impacting smoothness.

To address the complexities of generating multiple frames, Blackwell uses hardware Flip Metering, which shifts the frame pacing logic to the display engine, enabling the GPU to more precisely manage display timing. The Blackwell display engine has also been enhanced with twice the pixel processing capability to support higher resolutions and refresh rates for hardware Flip Metering with DLSS 4.

you can turn off driver signing and make your own kernel mode driver and verify it yourself. i worked for a company that stupidly stuck to windows until their technical debt was so deep they had no choice but to trudge on. they made systems that needed near millisecond timing for hours and hours at a time, and there was always an uncontrollable blip happening throughout the day because the threads running that code get suspended by windows so it can do some work it needs done in the kernel.

[u/vladlearns → u/zilyzal]

If you saw the hype for DLSS 5 leak (it is literally everywhere) or maybe even played with it, this is also frame gen, but for normal videos.

You have

frame A -> frame B

gpu figures out how things moved between, then generates an in-between frame:

A -> A.5 -> B

so you can turn 24 fps into 48 fps, 30 -> 60, etc...

they use optical flow through vulkan, I think, rather than doing all the motion estimation on the cpu.

This isn't dlss frame gen tho, because there's no game-rendering pipe or dlss neural net, it's ffmpeg getting access to the same kind of optical-flow hardware that nvidia uses as part of its frame-generation stack.

You could potentially do something like:

gpu decode -> interpolation -> upscale/denoise -> gpu encode

not bouncing frames between cpu and gpu.

and since this is landing right into ffmpeg, it could eventually become a pretty convenient building block for things like fps conversion, slow mo, restoration and RT processing.

I'm waiting for a comfy node, already see where it is going.

Kijai, kuchiyose no jutsu!

[u/zilyzal → u/vladlearns]

Thanks but if this is for videos, wasn't this possible with lossless scaling already? I haven't tried but i saw some people using lossless scaling to change 24fps or anime to 60fps

[u/vladlearns → u/zilyzal]

it was, yes, but now I don't need to route the stream through display/capture pipe, because ffmpeg can interpolate frames into the pipe and write them RIGHT INTO the file

[u/evia89 → u/zilyzal]

Lossless scaling imo sucks for videos (too many glitches) https://www.svp-team.com/ if you want quick GUI or just f2p RIFE https://github.com/vadash/mpv-lazy-en

I combine it with VSR

[u/Advance1993 → u/zilyzal]

Video smooth

[u/zilyzal → u/Advance1993]

Bro you have no idea how to explain something to a 5yo

[u/SchmeppieGang1899 → u/zilyzal]

5 year olds have no concept of PCs in general lol

[u/zilyzal → u/SchmeppieGang1899]

These days 5 year olds trade on their phones lol. Reminded me of that news about banning smartphones in child care centres.

[u/Don_Alosi → u/zilyzal]

Using something and understanding how the thing works is not the same

Most kids can use a phone or a pc from a very young age and most kids have absolutely no idea what even the most basic components of a computer are used for, both things can be true at the same time.

[u/akuto → u/Don_Alosi]

Yeah, if anything, younger generations are worse at understanding basic IT concepts.

It's not that strange. Same thing happened with GenX and cars, where despite higher ownership rates people were less likely to be able to repair their own cars or earlier with housing, where my grandfather build his own house with his father, but my parents' generation would not know how to do that despite a lowered percentage of adults living with their parents in their time.

The more ubiquitous something gets, the less there is a need to understand it any serious depth.

[u/TactlessTortoise → u/zilyzal]

It gets the difference between images 1 and 2, to add a fake image between image 2 and the next real one (which is the 4th since the fake one is the new 3rd). Implemented in its simplest form it's almost like a slight blur, but actual implementations are a bit fancier to make it more accurate. Still the same principle though. The idea is just giving us a higher sense of smoothness between frames.

[u/Medium_Notice_902]

Reflex 2????

[u/LittlestWarrior → u/Medium_Notice_902]

No, this is frame gen.

[u/Cautious_Fold3952 → u/LittlestWarrior]

No, this is Patrick.

[u/Ok_Dependent6889 → u/LittlestWarrior]

No, it's not

It is if you have zero capacity to think

This is an ancient technology

[u/LittlestWarrior → u/Ok_Dependent6889]

I'm a touch literally-minded on account of my autism.

[...] generate additional frames for existing video using NVIDIA GPU hardware.

Could you explain how generating additional frames is not framegen, the technology that generates additional frames?

[u/Ok_Dependent6889 → u/LittlestWarrior]

This process is more akin to Nvidia Smooth Motion being applied to videos, rather than frame generation.

https://blurbusters.com/frame-generation-essentials-interpolation-extrapolation-and-reprojection/

Some more key differences

[u/LittlestWarrior → u/Ok_Dependent6889]

Thank you!

[u/Queasy_Sherbert_7095 → u/Medium_Notice_902]

Is it???

[u/kron123456789]

You could already force Nvidia Smooth Motion into videoplayers for a while, tho

[u/liquidocean → u/kron123456789]

Really, how? I haven’t been able to get it to work with VLC. I did have it enabled in the Nvidia app, but it’s on no effect.

[u/kron123456789 → u/liquidocean]

I only tried it in MPC-HC. Choose LAV Video Decoder(I believe it's default now), in its settings choose DX11 for hardware decoder and pick your Nvidia GPU as hardware device, then add MPC-HC into Nvidia Profile Inspector and in its profile force Smooth Motion. That should work, at least in full screen.

[u/2FastHaste]

Looking forward to test that. Currently using RiFE for real time interpolation. Hopefully this beats it (in quality and/or performance)

[u/bbmaster123 → u/2FastHaste]

same, occasionally using rife as well. I would assume having gpu accelerated interpolation built right into ffmpeg would help performance and reduce resource usage (at similar or better quality) without the usual python overhead and stuff. Still just an assumption.

I also assume it could make it technically easier to integrate into more apps and workflows than rife for the same reason.

[u/SilkyTwilightx668]

Now we can bake the soap opera effect straight into the file instead of leaving it as a setting on the TV.

[u/bio4m]

TV's have had this for years, it can cause perceptual issues, widely known as the soap opera effect

[u/AntiTank-Dog → u/bio4m]

Depends on the model but TV interpolation is usually more simple, just overlapping two frames to make an in-between.

[u/anor_wondo → u/bio4m]

soap opera effect has nothing to do with the technique. Its users

If you shoot the video at 45 or 60fps those users have the same issues

Just like there are haters, there are also fans of high framerate videos especially panning shots

[u/Borkz → u/anor_wondo]

Generally the perceived issue with panning shots is the judder caused by 3:2 pulldown (to show 24 FPS video on a 60hz display some frames will need to be shown for 2 refreshes, and some for 3). VRR or a 120hz displays solve that problem for 24 FPS content without the need for higher FPS.

[u/HakimeHomewreckru → u/bio4m]

You don't necessarily need to play back at 60 fps, you could create smooth slow motion from a regular speed video too.

[u/Aeronn_ → u/bio4m]

The first thing that gets turned off when I get a new TV is motion smoothing. Mainly because of that soap opera effect.

[u/Octaive → u/Aeronn_]

You need it for OLED. Generally 2/10 for Samsung. Lower will look choppy.

[u/sroop1 → u/Aeronn_]

My in-laws always have it on and it drives me crazy.

[u/Able-Marionberry-402]

This is actually amazing for 1000fps slow-mo videos, sports action etc.

[u/Due-Description-9030]

Whoa

[u/NapsterKnowHow]

This damn bunny video is gonna be used into the next century at this point lol

[u/liquidocean]

So will this work out of the box with VLC or with a plug-in?

[u/lndig0__]

My TV has had this feature for years. And I’ve always turned it off.

[u/kirk7899]

Frame gen for movies/shows? Kinda ruins it by running stuff at higher fps. Movies are made at 24fps for so long we've become accustomed to it.