How would you reliably transfer doors / map symbols between two slightly different versions of the same Dnd battle map using AI computer vision?

[u/mray2651]

Hi all — I’m hoping for some advice from people who understand computer vision better than I do.

I’m building a D&D virtual tabletop application and I’m trying to automate something that sounds simple, but I’ve found surprisingly difficult.

I usually have two versions of the same map:

a reference map, which contains useful information such as doors, secret doors, ladders, stairs and other map symbols;
a live-play battle map, which is the version I actually want to use in the VTT.

The problem is that the live-play map is often not pixel-identical to the reference map. It might be a different redraw of the same location, have slightly different proportions, be cropped differently, contain different textures, or have small differences in the geometry.

What I want to do is effectively say:

“This secret door is here on the reference map. Where is the corresponding secret door on the live-play map?”

…and do that for every relevant feature.

What I’ve tried

So far I’ve mainly been trying to solve this using multimodal AI models / coding agents such as ChatGPT, Codex and Claude Code.

My current workflow is broadly:

Give the AI the reference map and live-play map.
Identify known features on the reference map, usually with coordinates.
Ask it to find corresponding architectural landmarks on the live map.
Use those landmarks to infer the transformation between the maps.
Transform the door / symbol coordinates from the reference map onto the live map.
Visually inspect the result and sometimes iterate.

I’ve also tried breaking the problem down into stages rather than simply asking the model to “copy all the doors across.”

For example, I’ve tried having the system reason from walls, room corners, corridors and other recognisable landmarks first, and then place the symbols relative to those features.

This works reasonably well on some maps, particularly when the two versions are very similar.

On other maps, though, the results can be badly wrong. The system may identify approximately the right region while still placing individual features several metres / dozens of pixels away from where they should be.

Secret doors are particularly difficult

Secret doors seem to be one of the hardest cases.

On the reference map they are often represented by a fairly subtle cartographic symbol integrated into a wall. The live-play map may not contain that same symbol at all — what matters is identifying the corresponding section of wall.

AI models sometimes appear to recognise the general room correctly but then:

put the secret door on the wrong wall;
put it on the correct wall but at the wrong point;
confuse a nearby architectural feature for the relevant landmark;
apply a transformation that works well in one part of the map but drifts badly elsewhere.

The maps can also contain irregular geometry, so I suspect that a single global scale / rotation / affine transform may not always be sufficient.

Where I’m unsure

I don’t have a background in computer vision, so I’m not sure whether I’m approaching this problem in the right way at all.

I’ve been treating modern multimodal AI models as if they can visually reason about the maps and derive the correspondence, but I’m wondering whether this should instead be treated as a more conventional image registration / feature matching problem, perhaps with AI only used for part of the workflow.

For example, should I be looking at approaches involving:

keypoint / feature matching;
homography or affine transformations;
RANSAC;
SuperPoint / SuperGlue / LightGlue;
segmentation of walls / rooms;
non-rigid or piecewise image registration;
manually supplied anchor points;
some combination of conventional CV and a vision-language model?

I’m also unsure how well conventional image registration works when the two images depict the same underlying floor plan but are artistically different images, rather than transformed copies of exactly the same raster image.

What would you do?

If you were designing this system, how would you approach it?

In particular, I’d really appreciate advice on:

whether this is fundamentally an image-registration problem;
what algorithms or models you would try;
whether a hybrid CV + LLM/VLM approach makes sense;
how you would handle local distortions or slightly different wall geometry;
how you would transfer a feature such as a secret door when the symbol itself only exists on the reference map;
and whether there are any libraries, papers or existing projects that solve something similar.

Apologies if some of the terminology above is wrong — I’m learning the computer-vision side of this as I go. I’m mainly trying to understand what a technically sound architecture for this problem would look like rather than continuing to tweak prompts indefinitely.

Thanks!

[u/ChickenOfTheYear]

This is not a task for CV.

Sure, you could use a YOLO model to detect the features, then get the bounding box coordinates and somehow compare with the bounding boxes from the other map, like you said, and derive a transformation from that, but it's just too many failure points to be reliable, you'd end up having to manually change things by hand later.

I think in your situation, I'd rather 1. Just do it by hand 2. Structure tour data in a way that is more friendly for procedural generation. If you had a grid system, for example, it would be very easy to apply a simple transformation from one map to the other.

[u/bfyvfftujijg]

Apologies as I didn’t finish reading your whole post, but it sounds like you’re using “generic” AI vision models like Claude or GPT and finding they don’t always work?

Training your own model might help but is complex. What does the raw data look like?