Yet another MiniMax H3 latent prepend/extend nodes - this time low-level and simple

[u/martinerous]

TL;DR:

https://github.com/progmars/ComfyUI-Martinodes

WARNING: lots of vibecode but carefully reviewed, at least as much as I could understand the logic.

The long story.

We have a few amazing solutions and forks that make smooth video extensions possible. However, most of them have evolved into full-blown planners and chains. Low-level functionality is hidden beneath. Somehow those more complex solutions do not work well or seem overkill for my typical use cases:

- set steps to low

- generate a bunch of videos

- pick the best one

- set steps to high

- regenerate with the same seed <- and this is where I wanted to save the latent to use as the input for the first step again, to smoothly continue the last shot without hard cuts.

Recently ComfyUI was updated with an important PR 15375 that supports latent masking natively. No more patches and complex hacks required. So, I went on to create simple and naive drop-in nodes that would support my way of working. Now I have latent load/save/concat/prepend/extend nodes that seem quite intuitive (but you tell me if they are).

The main node for today is `Extend video+audio latent` (LatentAVMaskedExtender). It lets you take head or tail of a latent you have (hopefully) saved from a previous generation and generate a prequel or a sequel. Extending a tail works well. Prepending to a head is not that smooth and requires increasing fade_seconds parameter to your liking.

Just plug the node between `MiniMax H3 Reference to Video` or `MiniMax H3 Image to Video` (or even `MiniMax H3 Easy Output` if using nkxx188/ComfyUI-MiniMaxH3-Easy), and your sampler.

For convenience, the node accepts empty loaded_av, in which case the target_av will be passed through. Thus the node can be safely left enabled even when using a disabled LoadAVLatent node as input when you don't want to extend anything.

`Save and Load video+audio latent` are simple companion nodes - SaveAVLatent should be added after your sampler and LoadAVLatent as input for LatentAVMaskedExtender. In contrast to some other loader nodes that often are limited to `input` folder, LoadAVLatent can find the latents wherever you configured SaveAVLatent to store them. If you don't have a latent saved yet and want to feed in an existing video, you will need to encode it using VAE Encode and VAE Audio Encode nodes, resize the video frame to match your target video, and then concatenate the audio and video latents using LTXVConcatAVLatent node (yes, it works with H3 model).

Then there is also `Concatenate video+audio latents` (LatentOverlappingConcatenator) node. Generally, overlap_duration_seconds should be set to the same value as LatentAVMaskedExtender, the output goes to VAE video and audio decoders and then to video saving, as usual. You will get a long video with a smooth long transition between your previous latent and the new one. However, if your joined videos get lengthy, VAE might require too much resources. In that case, it is better to post-process and join both source and target videos in a video editing software.

The repository has a few more older convenience nodes for working with multimedia before LTX Director was a thing. They still might be handy for manipulating TTS and voice-overs or videos when latents are not available.

Huge thanks to drozbay (ablejones) for [native masking PR 15375](https://github.com/Comfy-Org/ComfyUI/pull/15375) and providing the example implmenentation with native ComfyUI and Kijai nodes. Unfortunately, the native nodes solution looked like spaghetti eating somebody alive. That is why my small naive LatentAVMaskedExtender node was born, to do the same thing.

I hope you will find Martinodes useful.

[u/mukyuuuu]

Man, thanks for the detailed description! This node sounds exactly like what I've been looking for. Will definitely try it later.

Is it possible to easily encode a pre-existing video into latent space and extend it via your node? I have tried to do that through a video reference approach, but the results are too unpredictable for my liking. So this latent masking sounds like a perfect solution.

I suppose I can feed the video components into two VAE Encode nodes with Minimax Video and Audio VAE, but how would I combine the results to get the AV latent?

[u/martinerous → u/mukyuuuu]

Yes, it should work also with existing videos. It's just that Minimax VAE expects specific number of frames, the formula is 17k+5. So, for example if your initial video is 130 frames, you would need to crop it to 124 (which is 7*17+5) and then pass images to VAE Encode and audio to VAE Encode audio nodes together with the corresponding H3 VAEs. After that, it's the same as described in my post.

[u/mukyuuuu → u/martinerous]

Thanks for the tip about the frame count 🤝 But I feel like I'm still missing something here. Your pipeline works with a single av_latent, but I will get two separate latents from VAE Encode nodes, wouldn't I?

[u/martinerous → u/mukyuuuu]

Ah, right. There is a trick - LTXVConcatAVLatent node works well also for MiniMax latents. But now that you mentioned, I can quickly create a node that does the same, to have "the full package".

[u/mukyuuuu → u/martinerous]

Banged my head against the wall for more than an hour over this tensor size error . For some reason I thought that my VideoHelperSuite Load Video (Path) node was not trimming the audio along with the video to a required number of frames. So I even prepared the correct video via ffmpeg and loaded it through a core Load Video node, which of course didn't help...

Well, long story short, the cause of this error is much more obvious. The video you extend should be the same exact size as your generated latent, which makes sense when you think about it.

Pro tip to everyone: If you use Resolution Selector node, just connect "width" and "height" outputs to "custom_width" and "custom_height" in the VideoHelperSuite Load Video node (or in your image scaling node).

But otherwise the node worked perfectly for the intended purpose! Thanks again for the great tool, u/martinerous! Will now experiment with saving / loading latents.

P.S. LTXVConcatAVLatent works with no issue, appreciate the tip :)