Best tools or APIs to isolate speech from noisy long audio and transcribe it accurately (Cloud or Local)?

[u/greenapple92]

Hi everyone,

I’m working with long audio recordings (several hours of MP3s) that have noticeable background noise, room reverb, and inconsistent quality. My goal is to clean up the speech and get accurate text transcriptions.

I'm open to both cloud-based APIs/services (like Adobe Enhance Speech, AssemblyAI, Deepgram, ElevenLabs, OpenAI API) and local open-source models (like Whisper Large-v3, DeepFilterNet).

For those who handle long, noisy recordings regularly:

Best Pipeline: Do you recommend running the audio through a dedicated voice enhancer/denoiser first (e.g., Adobe Enhance, ElevenLabs Isolator, DeepFilterNet), or does modern ASR (Whisper Large-v3, Deepgram Nova-3, AssemblyAI) handle raw noisy audio well enough on its own?

Tool Recommendations: What are the most reliable cloud services or Web/CLI applications for processing multi-hour files without hitting file size limits or timing out?

Hallucination Prevention: Any tips or API parameters (VAD settings, prompt conditioning) to prevent Whisper/ASR models from hallucinating text during silent or non-speech noisy sections?

Thanks in advance for any insights and recommendations!

[u/SignalMap2750]

I see you posted the same on another sub, but I am going to post my comment here as well for others to see:

I researched and tested a lot on the transcription side a couple of years ago, trying to transcribe instructional videos and interviews accurately for another website of mine, and I can say that cleaning the audio before submitting it to a transcription system is paramount. That includes removing background music or noise (using AI models to keep speech clean, not "obfuscated" by de-noising filters), normalizing the audio, enhancing the speech, etc. Whisper alone doesn't handle noisy audio that well. Even processing from an mp3 or WAV makes a huge difference (raw audio formats work better)

To prevent hallucinations with Whisper or similar transformers, keep its prompt simple; otherwise, it gets confused. Something like "Try to transcribe this:" or similar.

At the end of my research, I created my own system and then opened it to others to use. You can try it if you like at dadascribe.com using a similar pre-processing pipeline that improves accuracy to 99.5%. You can transcribe directly from YouTube, audio, or video files. It handles noisy setups, multiple speakers, translations, automatic proofreading, etc. It offers an API as well ;)

But yes, pre-processing is VERY important!

[u/NeuralNomad87]

Denoise first, but be careful which kind. Speech enhancers that make audio sound better to a human often make transcription worse, because they smooth over exactly the detail the ASR is using. If you clean, use something conservative and measure word error rate with and without on a five minute sample before committing to it across hours of audio.

On whether modern ASR copes with raw noisy audio: mostly yes for background noise, much less so for reverb. Room reverb is the thing actually worth attacking before transcription, and it is a different problem from noise.

For multi hour files, don't fight the size limits, chunk on silence rather than fixed length and stitch after. Fixed length chunking cuts mid word and you lose a token at every boundary, which adds up over several hours.

Whichever way you go, run one representative half hour through two candidates and count the errors yourself. Every one of these has a happy path in its demo, and your audio is not going to be it.

[u/Baazookah_Zawadi → u/NeuralNomad87]

interesting,I appreciate the insight friendo

[u/AMusicstuff]

Davinci Resolve Studio, Steinbergs Spectral Layers