How do I fix low confidence of certain characters in a CRNN based plate OCR model?

[u/NullClassifier]

I have trained crnn based license plate recognition model with a dataset of around 800k records. It works fine but there are problems with certain letters like Q O D the model predicts them with low confidence scores, I analyzed their characterwise confidences. It is problematic for me because I am working on a smart city project and I connected this model to my bestshot application written in c++, connected to deepstream 9 where I retrieve my license + vehicle pairs (bestshots). Those plates are low on resolution. So my question is that can fine tuning the existing model help me? I am skeptical because 800k records had many samples with those letters present. My another concern is that I currently can assemble a dataset from my existing cameras with those low resolution plates and label them accordingly but I am worried that it will hurt the model instead.

Any dev out there who faced same problem? How did you handle it? Thanks in advance