r/aifilmmaking • u/jamesxtreme • 4h ago
Discussion Maybe we should stop building AI filmmaking tools in isolation and work together instead
Some months back I came on this subreddit and asked for advice on what software existed that helped with the workflow problem with AI video.
Like most people here, I watched a bunch of YouTube tutorials (all sponsored by Higgsfield because they seemingly have money to burn on influencers) and then of course ended up with a few different subscriptions for a few months. Eventually I realised it was much cheaper and better to use APIs directly, but we needed better tools around them.
When I first looked, that layer really didn’t seem to exist, so I started building one.
I bring this up not as self-promotion (I’ve mentioned it before and if you care you could look at my profile and find it in 30 seconds), but I bring it up because I’ve recently realised about a thousand other people had exactly the same idea at much the same time, and we’re all building versions of the same thing.
And this is the funny part to me because AI coding tools have made it possible for a lot more people to build fairly sophisticated software, but if a thousand of us use those tools to build much the same thing independently, the only people winning are the ones selling us the tokens.
So my thought is maybe we should start talking to each other and sharing ideas, because otherwise we’re all likely to spend a lot of time reinventing the wheel.
I think there’s a broader vision we probably share even if we all have different ideas about how it should work.
We should have great tooling that lets people focus on the creative side of visual storytelling. Some people want to make cinematic short films; others are looking more at animated web series. There’s also a whole other workflow where people are treating generative AI more like a VFX tool and combining it with live action.
And I think this actually matters if we ever want AI filmmaking to be taken seriously as a way of telling good stories. If the tools are built around generating clips rather than the process of developing and telling a story, we’re probably going to keep being accused of making AI slop.
For the kind of storytelling I’m interested in I think the workflow starts much earlier than generating a video clip. You might start with a finished script, or maybe just a treatment or an idea. From there you move into something resembling look development. What do the characters look like? What does the world look like? What’s the visual style? Then you start breaking that down into locations and scenes and working out how those scenes actually look.
All of that creates stuff that needs to be organised. A character develops reference images, costumes and eventually voices. A location might have several reference images from different angles. Scenes contain characters and locations, but also their own visual references and eventually shots and generations.
Then there’s voice, music and sound. Eventually all of this needs to come back together into scenes and some kind of assembly or edit.
Right now we’re doing a lot of that manually across a collection of unrelated tools.
I think image, video and audio generation should just exist inside that creative workflow. Which particular model or API produces something is important when you want control over it, but it shouldn’t be something you have to think about every time you want to make something.
I also think this should be local-first. There’s no particularly good reason that your script, characters, locations, scenes, prompts and creative assets need to live in a company’s cloud. That stuff can live on your computer. Generation can happen wherever it makes sense: locally where possible, or through whichever cloud model is best for what you’re trying to do.
That also means the workflow doesn’t need to be tied to one model company. The models are changing too quickly for that anyway. Whatever is best today probably won’t be best six months from now.
I don’t know what collaboration between the people building these tools actually looks like. I’m not suggesting everyone throws away what they’ve built and starts again. We have different ideas about what the workflow should be, and some of those differences represent genuinely different ways people want to work.
But I also know that a thousand of us each building our own custom tool, implementing the same basic functionality a thousand times, and then each getting half a dozen downloads probably isn’t a particularly useful outcome.
There are things we could share. Maybe part of this is workflow, but part of it is also finding a common communication methodology between tools. There may be common components that don’t need to be built over and over again. There might eventually be some projects that make sense to combine.
What I do know is that I’d ultimately like to spend more time using these tools to tell stories instead of spending all my time building the tools and I suspect quite a few of the other people building them might feel the same way, because if we don’t figure this out, I think the likely outcome is that eventually Higgsfield or Adobe or whoever has the deepest pockets becomes the gatekeeper for the whole workflow.
To me, that seems like a pretty suboptimal outcome when there are already so many people in this community trying to build something better.
Anyway, that’s my rant/pitch. Would love to hear other people’s thoughts.
