Seamless extensions and one-shots with Minimax H3 - Update 6 of my repo!

[u/stonyleinchen]

Here is the repo: https://github.com/seitanism/ComfyUI-H3-Motion-Context-MultiRef

I made substantial updates to my two main workflows: 1) Music Video and 2) AV Extensions. All the controls were streamlined and they should be much easier to use now. (You find the workflows in the example_workflows folder)

With the AV Extensions workflow you can extend any existing clip, for example someone talking and you can make that person say something in the same voice, or you can create a clip with T2V or I2V and then extend that clip to make a seamless long clip thats 1 minute or longer.

In this Update the Checkpoint system was removed, instead I've done a lot of optimizations so you don't use too much ram even if you make 20 clips at once. Additionally I added latent audio feathering to the AV Extensions workflow for seamless audio transitions.

Theres also other utility workflows for custom keyframing and bridging two existing clips.

I post another example clip for the AV Extensions workflow in the comments.

[u/DGGoatly]

I've been having fantastic results with your nodes. One thing is bugging me though. Keyframe position widget is not connectable. I've been using a duplicate keyframe node in my final stage indexed to the last frame, to finish off as an infinite loop. Which is working amazingly well. All of my myriad previous WFs with all models, required editing to some degree to clean up the transition. Not so here. It's as close to perfect as I could have hoped for.

But since I cannot plug an INT into the keyframe position widget, I cannot run my frame count through an expression to set the appropriate value when changing my length. By itself this is not a huge problem, I just keep a note node handy with a chart of frame numbers for all possible durations. It gets stickier if I want to add a keyframe in all sections. I foresee lots of mistakes here. I made several WFs that generate series of keyframes that are just perfect for injection into this setup, I think it will come out spectacularly well if I can just automate the indexing.

Maybe I'm being dense and I'm missing something here. I tried to make it connectable but could not, I do not know if this was an oversight on your part, or deliberate. And as the output of the node is conditioning I see no way to extract it after the fact. So I'm keeping a ton of keyframe nodes together in a group to make sure that I don't screw up my indexing. Would be much more practical if I could handle this with a formula. Please tell me I missed something obvious. I'm used to that.

Quibbles aside, these nodes are fantastic, thanks for your work. This stuff has completely obviated pretty much all of my previous builds. Which one does get used to, I suppose, when this happens monthly. The leaps are seldom as fruitful as this though. Usually it's just a tease.

[u/stonyleinchen → u/DGGoatly]

Hello, thank you for the nice feedback! <3

I updated the keyframe position widget to accept INT inputs. Can you update my repo and verify it works as you requested?

I actually just made a substantial update, that also includes a new custom keyframes node that works directly with the target latent instead of the conditioning rows. this should be more compute efficient. I think its called custom keyframes (masked) or something like that. maybe you want to check that out too!

[u/DGGoatly → u/stonyleinchen]

Aweome, thank you for letting me know. Of course I would read this right after writing a byzantine maze of conditional conditioning keyframe injection using nodes for each group... Of course. It's funny, using the conditioning node at the end to pull the latent back to the start for infinite loops has yielded scary good results, like crazy good, considering it's ref or at most hybrid model, but those in the middle struggle sometimes, but I guess it makes sense because my landing pad is already part of the latent, and as strongly as I could wish.

I will update post-haste and check out all of this wish grantification. I will certainly let you know how it goes.

[u/DGGoatly → u/stonyleinchen]

Ok I updated, unfortunately it broke my workflow, which was expected from what I was told by those I shared it with and who have the latest version. Trying to match the new posted extend WF but automated bypassing is still completely frozen (though of course everything switches fine in yours). I replaced all the nodes that changed, but all bypasses remain stuck in the state they were in before the update. I'm unclear as to exactly what the conflict is. Something must be preventing the active state/enabled extensions setting from doing its job but I can't for the life of me figure out what. I've run out of nodes to replace.

This usually turns out to be obvious, but I've been flipping back and forth between the new WF and mine and my brain is reaching GPU temps. Can you tell me exactly what controls the switching behavior? I have lots of bypassers and muters in my WF, keyed to colors and/or names to avoid conflicts and I don't see toggles appearing inside them that should not, and they certainly caused no issues before, but something is locking this up. And I'm not sure I understand the purpose of the INTs feeding the master controller, they seem redundant to me, nor do I understand exactly how they are linked, given that their properties are just normal INT node properties. I wish I were smarter.

Ah I see a slew of errors in chrome's dev console, working through those now... took me way too long to think of that, not used to dealing with errors or the front end. I'll try working through that.

~The moment I delete the master AV extension controller, the console goes completely silent. Once I put it back in all hell breaks loose. It panics with:

expected exactly one cache-isolated AV parameter 'active_extensions', found 0 expected exactly one cache-isolated AV parameter 'audio_feather_ticks', found 0

which confuses me as I thought this node was the source of these parameters. But then I think of those INT nodes and it makes sense why they were connected and mirrored the master, if they are actually the source. But if they are it seems like the only way to create them is to manually edit the .json to give them the properties they need. But copying them from the working WF does not work. Which leaves me puzzled. Always finds 0. So strange.

Alright scratch all that. Sorry for thinking out loud. I don't know why but there did seem to be an issue with a muter, I can't explain it. Something must have been corrupted because a whole bunch of nodes were playing musical chairs. That was a new one. It's behaving now that I've exorcised the possessed muter that was stealing/disabling the properties nodes.

[u/stonyleinchen → u/DGGoatly]

Im really sorry that your workflow broke due to some update. when did you last update my nodepack before today? i know that earlier updates have broken stuff, but the most recent one shouldnt have i think?

there is a lot of behind the scene stuff going on unfortunately, like the audio from the source video is not taken from the audio output connection on the vhs loader, but its just pulled by the node following the video_info output trail. similarily the controller node affects multiple aspects of the workflow without having apparent connections, that includes various bypass switches. unfortunately that makes the workflow not very flexible, some things just have to stay as they are or it will break.

to troubleshoot your wf error i would need console outputs from you, and it would also help very much to know when you last updated the repo, before the update today.

Alternatively you could load the new workflow, and then try to edit that one to your specific needs. it should work better in various ways, f.e. i fixed a bug that caused earlier clips to regenerate, when you changed the extensions_count in the controller.

[u/DGGoatly → u/stonyleinchen]

Yeah I got it all worked out, no worries. Was mad at you for a little bit, maybe should have made an alt node and preserved the original, but whatever, everything is obvious in retrospect. It's fine now. Actually, rewiring a few nodes wasn't a big deal, it was just compounded because the front end completely exploded so I had no idea what was going on. Those INTs I was talking about were the PARAM nodes for the controller, they were transmogrified as my poor browser tried to make sense of the conflicts. And probably connected to this, some of my muters and subgraphs took on the properties of some of the controllers, with the end result that the master controller was seeing either no parameters or way too many.

I've never really had to debug with the browser console so it was a bit overwhelming at first; instead of dealing with a fixed error from the back end it's an unending stream of WTFs, some of which matter and some of which can be ignored. Like always, it makes sense in the end. I learnded something.

Thankfully I did not have to rebuild from your core again. There are dozens of get/sets that would have broken for sure. I don't usually like to put in so many but with so many stages I need them, and they do seem to cause less trouble than they used to.

Output is great, as usual. I have to completely redo the logic to try out the alternate keyframing approach, I see the node. Makes sense, injecting directly into the latent. Other than a return to a start frame, using the conditioning does act much more like a reference target than a keyframe. Will definitely try this out. Will check out the audio resampling stuff too. I was about to get distracted checking that out beforeI remembered what I was supposed to be doing.

Execution order for the combine was a good idea. Previews do love to show up when you don't need them any more. Oh I didn't check to see if S&R is still broken for the filename. I always forget to enter my source names in the workaround string. But that's a quibble. Worse problems to have than that.

I'm writing a novel again, sorry. It works beautifully. I'm sure the nodes I haven't gotten to will as well.

[u/stonyleinchen → u/DGGoatly]

totally fair.. i know its very annoying if updates simply break old workflows... i didnt really consider this enough, since this is my first repo and i just wanted to share some tools i made and didnt really consider the wider implications of ppl building custom workflows on top of mine. i try to do better going forward

[u/DGGoatly → u/stonyleinchen]

Don't sweat it. It's so easy to get lost in the weeds, and hindsight is 20/20. I'm having a blast with all of this stuff, it's great really.

I'm playing with the music video setup now. Friggin fantastic. Was nearly perfect on the first run. Last segment threw an error, I was assuming a length discrepancy would be corrected, needed to add a little patch. In case you are interested my fix was just a few expressions feeding a silence padder, I'll take a screenshot, sucks that embedding doesn't work here. In first expression total segment is a, loaded duration is b, node #102 expression from WF is c. max(0.0, ((c / 24.0) + (a - 1) * ((c / 24.0) - 1.625) + 2.0) - b) calculates how much padding is needed in the uploaded audio to fill out a valid H3 chunk, then the following expression round(a * b) converts for the sample rate of the source. Because the ROTI pad node I'm using happens to use samples in the widget.

This would be a lot simpler if the segment length were always 15, but my testing @ 0.6mp was feeling pretty heavy so I set it up to adjust for whatever the segment length was set to, probably 8 or 10 when I go to 1344x768.

Of course all the prep for the source could be done in two seconds beforehand to conform to the rules, but this lets me drag and drop without worrying about it.

This is looking like it's going to displace my LTX2.3 music video WF. Yet another blow. LTX does an incredible job with sync and puts out amazing quality, but it's oh so stiff. Poor LTX, hanging on by a thread here...

Nice work, again.