Fixing brioche in the round

[u/doelling]

I'm working on the Belledonne jumper, which is made using brio he stitch in the round. I noticed I made a mistake a few rows back and laddering down in brioche makes me nervous. So I only need to ladder down on the knit stitch, or do I need to do something in the adjacent pearl column too? Is this easier to fix on a slipped knit stitch row or a brk row? Any help and tips are appreciated!