BEYOND BENCHMARK ACCURACY: EVALUATING WRITER DEPENDENT DOMAIN SHIFT IN HANDWRITTEN 0-999 DIGIT RECOGNITION

Abstract

Handwritten digit recognition systems can achieve high accuracy under the defined evaluation while showcasing weaker performance when applied to handwriting of unseen writers. In this research paper we investigated writer dependent domain shift in handwritten 3 digit recognition and tested whether the training duration influences cross writer generalization. A digit specific convolutional neural network was implemented on cross writer experiments where two writers were used for training and the third writer was excluded for testing. The model was tested using multiple durations and random seeds. Then we evaluated on five completely new independent writers. Even though the model achieved 95.3% accuracy on the benchmark evaluation, the exact three digit accuracy fell down to 30.2% in one of the most challenging configuration. During the cross writer experiment the performance was varied due to writer’s identity. Increasing the training duration only produced little improvements. Then Independent writers evaluation was conducted which resulted the accuracy of 66.78% on exact three digit, whereas the writer level performance ranged from 42.78% to 86.11%. The individual digit position accuracy was higher at 86-87%, which indicates that many complete number error resulted from mistakes in one or more digit positions. These finding establishes that receiving high baseline accuracy does not alone equate the robust writer independent generalization. These findings highlight the importance of evaluating handwriting recognition systems on unseen writers.