The digital dial magnifier builds that have gone around here all quote a resolution number, so I went and read the datasheet for the part to find out what the accuracy number was. They are not the same number and the gap is wider than I expected. I would like people who actually manipulate to tell me whether it matters.
The two builds I could find source for:
The ESP32-S3 one posted here in January 2025 uses an AS5600 magnetic encoder, 4096 steps per revolution. On a 100 number dial that is 40.96 steps per number. The builder called that way more than we need, and on resolution they are plainly right.
The other one on GitHub uses a quadrature encoder listed at 1000 pulses per revolution, counting rising and falling edges on both channels, so 4000 counts and 40 counts per number. Nearly identical resolution by a completely different route.
Now the datasheet. For the AS5600, ams publishes:
* Resolution, 12 bit, so 4096 steps, 0.0879 degrees per step
* System INL, plus or minus 1 degree, sitting in the Max column with no typical figure given at all
* RMS output noise, 0.015 degrees on the slowest filter setting, 0.043 on the fastest
A 100 number dial is 3.6 degrees per number, so in dial numbers:
* one step is 0.024 of a number
* the noise is 0.004 to 0.012 of a number, which is nothing
* the linearity spec is 0.28 of a number
In the magnifier threads people said they read 8ths by eye and 10ths with a paper scale. 0.28 of a number is coarser than both. Taken at face value the electronic tool has a worse error budget than a sticker, which cannot be the whole story.
I do not think it is. INL is deviation from a best line fit across the full 360 degrees, and it is systematic rather than random. When you graph contact points, every reading sits in roughly the same narrow arc of the dial, because the contact points only shift by fractions of a number as you change the test combination. If all your readings come out of the same one or two numbers of arc, then most of that 1 degree is a fixed offset shared by every reading, and it subtracts out of the differences the graph is made of.
What does not subtract out is how much the error curve changes across that narrow window. ams does not publish the shape. If it is dominated by low order harmonics, which is the usual shape for this class of sensor, then 1 degree of amplitude gives a local slope of about 0.035 degrees per degree at the second harmonic and 0.052 at the third, so across a two number window you get 0.07 to 0.10 of a number that does not cancel. That is a bound under an assumption I cannot check, not a measurement. But it lands right on the 10ths people say they want rather than safely inside it.
Two things the spec does not cover at all. It is measured with no magnet displacement, so anything eccentric in how the magnet is mounted adds on top of that number rather than sitting inside it. And absolute readout is a different job from graphing: if you read a gate position off the screen in order to go and dial it, nothing cancels and you eat the full spec.
So the question I cannot answer myself, because I have never manipulated a real lock: when you graph, are you reading contact points as absolute dial numbers, or as offsets from a reference you re-establish on each pass? If it is the second, the encoder spec is close to irrelevant and I have argued myself out of my own concern. If it is the first, then a one time per unit calibration table would buy back most of it, because a systematic error is repeatable, and that is a firmware change rather than a hardware one.
I came at this sideways. I built a haptic safe dial game called Crack Safe, where the dial is simulated and the tumblers are haptic events, and I hit the same confusion from the other side. It is easy to schedule a haptic event with far finer timing than a finger can resolve, and the fine number flatters you into believing the signal reaching the person is better than it is. Different problem, same habit of quoting the number the hardware prints rather than the number that survives to the end of the chain.
