In conclusion, one might be tempted to asking the question: "was the computational costs really worth it since the results of the neural network softmax regression model were close to that of the convolutional neural network model?". In this case, it is worthy to note two things: (1) A 1% difference in accuracy is a big deal in the field of machine learning and there are numerous research groups trying their best to squeeze out as little as a 0.5% improvement in machine learning models. (2) The testing of these models is based on properly cleaned, and normalized datasets. In the real-world, this is not the case. The CNN does well at learning complex features and abstracting higher levels of information from input data - this has been proven in literature. Finally, regardless of the final accuracy of either model, the CNN has better potential in computer vision and image recognition tasks.

References

[1]     Cohen, G., Afshar, S., Tapson, J., & van Schaik, A. (2017). EMNIST: an extension of MNIST to handwritten letters. arXiv preprint arXiv:1702.05373.
[2]     Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images.
[3]     Coates, A., Ng, A., & Lee, H. (2011, June). An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (pp. 215-223).
[4]     Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., & Ng, A. Y. (2011, December). Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning (Vol. 2011, No. 2, p. 5).
[5]    
[6]     LeCun, Y., Jackel, L. D., Bottou, L., Cortes, C., Denker, J. S., Drucker, H., ... & Vapnik, V. (1995). Learning algorithms for classification: A comparison on handwritten digit recognition. Neural networks: the statistical mechanics perspective261, 276.
[7]     LeCun, Y., Boser, B. E., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W. E., & Jackel, L. D. (1990). Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems (pp. 396-404).
[8]     Hou, Y., & Zhao, H. (2017, October). Handwritten Digit Recognition Based on Improved BP Neural Network. In Chinese Intelligent Systems Conference (pp. 63-70). Springer, Singapore.
[9]     De Chazal, P., Tapson, J., & van Schaik, A. (2015, April). A comparison of extreme learning machines and back-propagation trained feed-forward networks processing the mnist database. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on (pp. 2165-2168). IEEE.
[10]     Revow, M., Williams, C. K., & Hinton, G. E. (1996). Using generative models for handwritten digit recognition. IEEE transactions on pattern analysis and machine intelligence18(6), 592-606.
[11]     Sabourin, M., Mitiche, A., Thomas, D., & Nagy, G. (1993, October). Classifier combination for hand-printed digit recognition. In Document Analysis and Recognition, 1993., Proceedings of the Second International Conference on (pp. 163-166). IEEE.
[12]     Liu, C. L., Nakashima, K., Sako, H., & Fujisawa, H. (2003). Handwritten digit recognition: benchmarking of state-of-the-art techniques. Pattern recognition36(10), 2271-2285.
[13]     Liu, C. L., Nakashima, K., Sako, H., & Fujisawa, H. (2004). Handwritten digit recognition: investigation of normalization and feature extraction techniques. Pattern Recognition37(2), 265-279.
[14]  Teow, L. N., & Loe, K. F. (2002). Robust vision-based features and classification schemes for off-line handwritten digit recognition. Pattern Recognition35(11), 2355-2364.
[15]     Jeong, C. S., & Jeong, D. S. (1999, December). Hand-written digit recognition using Fourier descriptors and contour information. In TENCON 99. Proceedings of the IEEE Region 10 Conference (Vol. 2, pp. 1283-1286). IEEE.
[16]     Gorgevik, D., & Cakmakov, D. (2004, August). An efficient three-stage classifier for handwritten digit recognition. In Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on (Vol. 4, pp. 507-510). IEEE.
[17]     Zhang, B., Fu, M., Yan, H., & Jabri, M. A. (1999). Handwritten digit recognition by adaptive-subspace self-organizing map (ASSOM). IEEE transactions on neural networks10(4), 939-945.
[18]     Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and object recognition using shape contexts. IEEE transactions on pattern analysis and machine intelligence24(4), 509-522.
[19]     Leem, S., & Kim, S. (2016). High Speed Digit Recognition for Serial Numbers of Home Appliances.
[20-21]     Corr, P. J., Silvestre, G. C., & Bleakley, C. J. (2017). Open Source Dataset and Deep Learning Models for Online Digit Gesture Recognition on Touchscreens. arXiv preprint arXiv:1709.06871.
[22]     Su, J., Vargas, D. V., & Kouichi, S. (2017). One pixel attack for fooling deep neural networks. arXiv preprint arXiv:1710.08864.
[23]     Mathur, G., & Rikhari, S. (2017). A Review on Recognition of Indian Handwritten Numerals.
[24]     Parseh, M. J., & Meftahi, M. (2017). A New Combined Feature Extraction Method for Persian Handwritten Digit Recognition. International Journal of Image and Graphics17(02), 1750012.
[25]     Alom, M. Z., Sidike, P., Taha, T. M., & Asari, V. K. (2017). Handwritten Bangla Digit Recognition Using Deep Learning. arXiv preprint arXiv:1705.02680.
[26]     El Hindi, K., Khayyat, M., & Abu Kar, A. (2017). Comparing the machine ability to recognize hand-written Hindu and Arabic digits. Intelligent Automation & Soft Computing23(2), 295-301.
[27]     Chandio, A. A., Jalbani, A. H., Laghari, M., & Awan, S. A. (2017). Multi-Digit Handwritten Sindhi Numerals Recognition using SOM Neural Network. Mehran University Research Journal of Engineering & Technology36(4), 8.
[28]     Toulgaridis, N., Bougioukou, E., & Antonakopoulos, T. (2017, May). Architecture and implementation of a Restricted Boltzmann Machine for handwritten digits recognition. In Modern Circuits and Systems Technologies (MOCAST), 2017 6th International Conference on (pp. 1-4). IEEE.
[29]     Patel, A., & Kalyani, T. V. (2016, February). Support Vector Machine with Inverse Fringe as Feature for MNIST Dataset. In Advanced Computing (IACC), 2016 IEEE 6th International Conference on (pp. 123-126). IEEE.
[30]     Schaetti, N., Salomon, M., & Couturier, R. Echo State Networks-based Reservoir Computing for MNIST Handwritten Digits Recognition.
[31]     Mohapatra, R. K., Majhi, B., & Jena, S. K. (2015, December). Classification performance analysis of MNIST Dataset utilizing a Multi-resolution Technique. In Computing, Communication and Security (ICCCS), 2015 International Conference on (pp. 1-5). IEEE.
[32]     Rafiei, M. H., & Adeli, H. (2017). A new neural dynamic classification algorithm. IEEE transactions on neural networks and learning systems28(12), 3074-3083.
[33]     Tuba, E., Tuba, M., Simian, D., & Street, I. R. (2016). Handwritten digit recognition by support vector machine optimized by bat algorithm. In 24th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision,(WSCG 2016) (pp. 369-376).
[34]     Deng, L. (2012). The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine29(6), 141-142.