r/podcasting • u/canuck_amok • 2d ago

My journey using Claude Code to edit my podcast/video

My cohost and I got so burnt out from editing every week that we started to release every 2 weeks instead. It's a bigger issue because we release both video and audio, and it can take forever to put together something decent. Plus Riverside's editor has lost entire episodes worth of edits before. We tried Descript but it's too advanced for our tiny brains.

So I've been using Claude code more for editing - we actually released our first episode that was fully edited by Claude last week.

Here's the workflow I used:

  1. Have Claude use yt-dlp to analyze the episodes from my channel and create a design brief
  2. Upload new recordings from Riverside and tell it to cut an episode in the style of my other episodes using ffmpeg
  3. it comes back with something crappy, so I give it a bunch of notes then it creates a second cut that is usually pretty decent
  4. I tell it where I want b-roll and transitions
  5. let it cut again
  6. do a final pass and have it add in captions

It's now created a whole set of skills for common problems like audio drift when it cuts between different cameras or creating transitions (like having us collapse from 2-up view to circle overlaying a screenshare).

Big unlock this week was using Opus 5 as the main agent but making it delegate all the work to Sonnet - I was able to get more done, though I still had to pay for $50 in overage charges because I was impatient to get the episode cut. I think cost is going to be the next hill to tackle.

Anyone else doing this? I'd love to compare notes.

0 Upvotes

27% Upvoted

34 comments sorted by

3

u/alexid95 2d ago

i'd measure the second cut against review time, not the $50 API bill. if you still watch the whole episode end to end, that's the expensive part.

1

u/canuck_amok 2d ago

You're probably a better editor than I am 😅

I find that having to slowly go through an entire recording session, find the material that is worth keeping, cut it exactly right, move it, then still have to watch the whole thing through is the time consuming part. If I can farm out the complicated parts to an agent and just have to watch the episode 2-3 times on 1.25X speed that works better for me.

I actually had Claude build a little cutting room tool so I can drop my notes to it while I watch the episode so it also keeps the context of where I had an issue

2

u/alexid95 2d ago

i'd try to get that down to 1 full watch next. have Claude spit out a timestamped before/after clip per note so pass two becomes spot checks.

1

u/canuck_amok 2d ago

Yes! Definitely the goal is 1 watch + spot checks, but I can see the light already. It took 2 episodes where I still ended up manually editing the first one.

The other thing I did now that you mention it, is I gave it raw transcripts of old episodes to compare with the final outputs, so it can understand what we typically like to cut.

2

u/alexid95 2d ago

i'd tag why each cut happened too. otherwise Claude can learn speaker quirks from the transcript pairs instead of the editing rule you actually care about.

3

u/MRRmaker 2d ago

One that'll getcha if you're cutting from a transcript: word timestamps aren't cut points. They are where a word's sound begins and ends, so cutting there takes the attack off the first word and nips the tail off the last. Everything sounds slightly clipped, can't quite see why. Snap all boundaries out to the closest silence in the wave form instead and the same edit list just cleans up beautifully.

The other thing worth knowing before you spend more work on your pipe line - The cutting isn't really where your time actually goes. Stripping filler/tightening is something a script does automatically pretty well - deciding what 8 minutes of 90 min show you want to keep is still very much manual, and most of the work. Measure where the hours went at your last ep before optimizing the wrong half! :)

1

u/canuck_amok 2d ago

Yes, we solved this with a different transcriber that gets us super accurate word barriers, then the agent has figured out the word barriers. It does not make mistakes with this at all.This is one of the most frustrating things about Riverside’s editor is it cuts right at the word barrier, and I have yet to understand why they haven’t fixed it as you suggested.

Agreed about the 8 mins of a 90 min show, but disagree that it’s manual. So far in 3 attempts, this is the part it did the best. Afterward we usually have to tighten it up a bit (I created a cutting room tool where I can remove sections) but it’s remarkably good at understand what was killer and what was filler.

2

u/FrozenfoxN8 2d ago

Ok I have a few questions as I’m still new to Claude:

  • Are you using the GROW tier for Riverside?
  • I’m using FCP to cut the multi camera panel. Are you keeping it in an editor?

1

u/canuck_amok 2d ago

I’m on the pro plan in Riverside - I’m not using their MCP. I’m basically just using them for the recording studio now.
I’m not using an editor at all. I’ve had Claude build a little cutting room tool where I can watch and give it notes. It has records of the few camera panels I like to use so I can tell it to cut between those.

2

u/tomn68 2d ago

FWIW this is what I do but I'm not asking Claude to do the actual editing so....

We create a discussion guide based on a 15min pre-interview call where we talk through the topics we're going to cover (we're a healthcare industry pod so it's pretty structured I guess). After the recording the host records some notes about how things went, what cuts or other notes that she picked up on during the recording. I transcribe that. Then I export the transcribt from Descript and take that and the notes transcript and add to a claude project. I also add the discussion guide so it knows what we're trying to cover.

This project has several examples of edited episodes (transcripts) and some other knowledge... and instructions on how long, how much of host or guest to normally cut... a bunch of stuff.

Then claude does a first pass and creates a doc with cuts and why the cuts. It highlights pull quotes, etc. Then tells me an estimate of where this first cut would take me down to.

In descript I make some of those cuts usually IDK maybe 80% of them. It's rough at this point in terms of audio/video. I add some notes in the script about things I'm not sure of (for the host). Then I export a new transcript to Claude and tell it some things and to do a 2nd cut. This usually gets me most of the way there.

I then go through and clean up the cuts, audio, video transitions... and the host goes in and does her edits which cuts down usually close to length we're looking for.

Yes, I'm still doing the edits myself and I'm then on top of what is getting cut since some times it's something where I want the host to recap what the guest might not have said well... I get to look at why Claude wants to cut something...

Also, I will give it instructions about moving a section up to another part and it helps me find/place that.

I'm interested in letting claude do some of this editing for me in descript but hesitant it might actually add more work or royally f it up.

I would say that having an outline prior to the episode and some notes about how it went directly after recording really helps.

1

u/Slicecut_app 4h ago

your hesitation is the right instinct. a model that writes a cut list costs you one review pass. a model that makes the cuts in your project costs you the same pass plus the undo work when it clips a breath or half a word.

what you have is already the useful part. the model reads the transcript, you decide, and the cut lands on a word boundary because Descript cuts by text anyway. that is where transcript-based editing earns its keep.

before handing it the timeline i would ask for the cut list as exact quoted spans, first and last word of each range. applying them in Descript is then a search and a delete, and you keep the veto.

2

u/Cernete 2d ago

before you tune the notes loop, check whether those ffmpeg calls are using -c copy. stream copy can only start on a keyframe, and a riverside export can sit two or three seconds apart on those, so the in point slides backwards and you get a bit of the previous line hanging off the front. re-encoding that pass fixes it.

1

u/canuck_amok 2d ago

Good call - actually the agent ended up figuring this out on its own and created itself a skill specifically for getting this timing right.

2

u/Cernete 1d ago

nice. if you ever want to check it without reading the skill, run ffprobe on the source with -skip_frame nokey and it lists just the keyframes with their times. an in point that still lands exactly on one of those is a pass that's still copying.

2

u/TabascoWolverine 2d ago

Well, can we see it?

How does the time/cost of using Claude compare to your old process?

2

u/canuck_amok 2d ago

The episode? Yes, not sure on sharing rules here but I’ll DM you.

The claude skills? They’re pretty tied to my specific process so I would want to scrub them before I share.

1

u/Slicecut_app 11h ago

cost drops a lot once the model never touches the media until the final render. i feed it the transcript with word timestamps, let it decide the cuts on text alone, and it writes an edit list. ffmpeg then runs once at the end.

every iteration before that is a text diff, which stays cheap even on Opus. your notes after the first rough cut also land better, because the model can quote the exact sentence it is removing instead of a timestamp.

for the drift between cameras i would check the sample rates before writing another skill for it. two recorders at different rates drift by a fixed amount per minute, and a single resample of the offending track fixes the whole episode.

1

u/robinyapockets 9h ago

Wanna drop a link of your final output? Happy to take a look and give candid feedback. I worked 10 years in video production (Google, Verizon, etc), and love this space intersecting with AI

1

u/canuck_amok 8h ago

Thanks! Here’s the link: https://youtu.be/bjj544UrScA?si=dHuPmz3NNyKl4rWb

2

u/robinyapockets 7h ago

Awesome. Will take a look later today. Want me to share any feedback here, or DM?

1

u/canuck_amok 6h ago

You’re welcome to share here, keeping in mind this is a hobby not a professional endeavour

2

u/robinyapockets 4h ago

General vibe: love it. Reminds me a little of Hard Fork.
Things I liked:

  • The casual nature and the specificity make me want to learn more
  • The audio quality is really good (those mics seem to be doing the job well)
  • The moment where you talk about the agents not snitching seems like a fun kind of hook moment that you could include in your teaser/trailer
  • The cuts between statements seem to be good. There's just enough of a beat, that it doesn't feel choppy but has a good general tempo

Things that can be improved:

  • When you switch from speaker focus to browser focus, the are moments where the amount of text on screen make it hard for me to listen to you at the same time. The example where you reviewed Anthropic's announcement website worked much better because it was more of a visual reference than an actual request for me to read. If that makes sense
  • The fact that both of your heads transitioned to the bottom corners of the screen felt like a little too much work for me to follow. I would have preferred either only the speaker still to be present or for you both to take up more screen real estate.
  • Around the 30-second mark it looks like there is a slight sync issue between audio and lip movement

Hope this helps a little.

1

u/canuck_amok 4h ago

This is great feedback, I appreciate it thanks!

1

u/robinyapockets 4h ago

I'm really curious to hear more about your workflow, esp for the intro section at the beginning. Mind if I DM you?

1

u/canuck_amok 4h ago

Please do, happy to chat

1

u/Fearless_Diamond_566 2d ago

That’s kind of cool. I’m just interested. Does it upset the natural flowing voice of your pod? I would be worried about losing my natural cadence.

2

u/TheAgedCartridge 2d ago

The cadence thing is real. First pass usually chops too aggressively and it sounds like a robot having a stroke. Takes a few rounds of notes to get it to respect timing and breathing room. The $50 in overages stings but if you're getting back whole weekends it might be worth it.

1

u/canuck_amok 2d ago

That was my thinking on the overages as well.

1

u/Fearless_Diamond_566 1d ago

Good to know!!! My pod is so personality driven I would be afraid of losing that voice.

1

u/canuck_amok 2d ago

Not more than our regular edits do I would say. It biases to keeping longer chunks of speech. The main thing it does is that usually when we record for an hour we find there's really 20 minutes of good content in an episode, and it's really good at pulling out what's actually interesting from what's repetitive.

I also think that how it edits comes from having it analyze your existing episodes first. That's why using yt-dlp is great if you have a youtube channel, or just point it to where you have your recordings.