Structure Optimization in Deep Neural Networks with Synaptic Pruning Based on Connection Appraisal

Computer and Knowledge Engineering

مقالات آماده انتشار، پذیرفته شده، انتشار آنلاین از تاریخ 07 تیر 1404   

نوع مقاله: Machine learning-Sadoghi

شناسه دیجیتال (DOI): 10.22067/cke.2025.88039.1113

نویسندگان

Aghil Ahmadi؛ Reza Mahboobi Esfanjani^*

Department of Electrical Engineering, Sahand University of Technology, Tabriz, Iran.

چکیده

Deep neural networks typically require predefined architectures, which can lead to overfitting, underfitting, high computational costs, and storage overhead. Dynamic structure optimization through pruning can reduce network redundancy, but it often results in performance degradation. In this study, we propose a novel pruning method inspired by biological synaptic pruning that adaptively optimizes deep neural network structures. The proposed method continuously monitors the contribution of each connection during training using a dynamic efficiency criterion that evaluates the relative importance of each connection within its layer. Connections are not removed immediately; only those consistently falling below a predefined threshold are pruned, ensuring stability and robustness. Validation is conducted by simulation on an industrial distillation column dataset under noisy conditions and the MNIST benchmark dataset. The results demonstrate improved accuracy, enhanced generalization, and faster learning, with an average pruning rate of 53%. Compared to conventional and state-of-the-art pruning techniques, our method achieves superior performance in terms of compression rate and accuracy while effectively mitigating overfitting.

کلیدواژه‌ها

Deep Neural Networks؛ Synaptic Pruning؛ Distillation Column؛ Connection Evaluation

مراجع

[1] Z. Allen-Zhu, Y. Li, and Y. Liang. (2019, Dec.). Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers. Advances in Neural Information Processing Systems. [Online]. 32, pp. 1–13.

[2] C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, and S. Yang. (2017, Jul.). Adanet: Adaptive Structural Learning of Artificial Neural Networks. Presented at International conference on machine learning (ICML). [Online].

[3] Y. Chauvin. (1990, Feb.). Generalization Performance of Overtrained Back-Propagation Networks. Presented at European Association for Signal Processing Workshop. [Online]. pp. 45-55. Available: https://doi.org/10.1007/3-540-52255-7_26

[4] G. G. Towell, M. W. Craven, and J. W. Shavlik. (1991, Jun.). Constructive Induction in Knowledge-Based Neural Networks. Presented at Proceedings of the Eighth International Conference (ML91), pp. 213-217. [Online]. Available: https://doi.org/10.1016/B978-1-55860-200-7.50046-5

[5] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, and Y. Wu. (2019, Dec.). GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. Advances in Neural Information Processing Systems. [Online]. 32, pp. 1–13.

[6] V. Sze, Y. H. Chen, T. J. Yang, and J. S. Emer. (2017, Dec.). Efficient Processing of Deep Neural Networks: A Tutorial and Survey. Proceedings of the IEEE. [Online]. 105(12), pp. 2295–2329. Available: https://doi.org/10.1109/JPROC.2017.2761740

[7] S. A. Janowsky. (1989, Jun.). Pruning Versus Clipping in Neural Networks. Physical Review A. [Online]. 39(12), p. 6600.

[8] G. Chechik, I. Meilijson, and E. Ruppin. (1999, Jun.). Neuronal Regulation: A Biologically Plausible Mechanism for Efficient Synaptic Pruning in Development. Neurocomputing. [Online]. 26–27, pp. 633–639. Available: https://doi.org/10.1016/S0925-2312(98)00161-1

[9] A. Pascual-Leone, A. Amedi, F. Fregni, and L. B. Merabet. (2005, Jul.). The Plastic Human Brain Cortex. Annual Review of Neuroscience. [Online]. 28(1), pp. 377–401. Available: https://doi.org/10.1146/annurev.neuro.27.070203.144216

[10] C. A. Mangina and E. N. Sokolov. (2006, Apr.). Neuronal Plasticity in Memory and Learning Abilities: Theoretical Position and Selective Review. Psychophysiology. [Online]. 60(3), pp. 203–214. Available: https://doi.org/10.1016/j.ijpsycho.2005.11.004

[11] M. V. Johnston, A. Ishida, W. N. Ishida, H. B. Matsushita, A. Nishimura, and M. Tsuji. (2009, Jan.). Plasticity and Injury in the Developing Brain. Brain and Development. [Online]. 31(1), pp. 1–10. Available: https://doi.org/10.1016/j.braindev.2008.03.014

[12] G. Hinton, O. Vinyals, and J. Dean. (2015, Mar.). Distilling the Knowledge in a Neural Network. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.1503.02531

[13] G. Chechik, I. Meilijson, and E. Ruppin. (1998, Oct.). Synaptic Pruning in Development: A Computational Account. Neural Computation. [Online]. 10(7), pp. 1759–1777. Available: https://doi.org/10.1162/089976698300017124

[14] S. Han, J. Pool, J. Tran, and W. Dally. (2015, Dec.). Learning Both Weights and Connections for Efficient Neural Network. Advances in Neural Information Processing Systems. [Online]. 28, pp. 1–13.

[15] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz. (2019, Jun.). Importance Estimation for Neural Network Pruning. Presented at Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11264–11272. [Online].

[16] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. (2016, Nov.). Pruning Convolutional Neural Networks for Resource Efficient Inference. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.1611.06440

[17] Y. LeCun, J. Denker, and S. Solla. (1989, Dec.). Optimal Brain Damage. Advances in Neural Information Processing Systems. [Online]. 2, pp. 598–605.

[18] B. Hassibi and D. Stork. (1992, Dec.). Second Order Derivatives for Network Pruning: Optimal Brain Surgeon. Advances in Neural Information Processing Systems. [Online]. 5, pp. 164–171.

[19] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. (2019, Jun.). Filter Pruning via Geometric Median for Deep Convolutional Neural Network Acceleration. Presented at Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 4340–4349. [Online].

[20] R. Yu, A. Li, C. F. Chen, J. H. Lai, V. I. Morariu, X. Han, M. Gao, C. Y. Lin, and L. S. Davis. (2018, Jun.). NISP: Pruning Networks Using Neuron Importance Score Propagation. Presented at Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 9194–9203. [Online].

[21] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. (2016, Aug.). Pruning Filters for Efficient ConvNets. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.1608.08710

[22] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. (2018, Aug.). Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.1808.06866

[23] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. (2014, Jun.). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research. [Online]. 15, pp. 1929–1958.

[24] L. Wan, M. Zeiler, S. Zhang, Y. LeCun, and R. Fergus. (2013, Jun.). Regularization of Neural Networks Using DropConnect. Presented at International Conference on Machine Learning (ICML(, pp. 1058–1066. [Online].

[25] X. Sun, X. Ren, S. Ma, and H. Wang. (2017, Jun.). MeProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.1706.06197

[26] P. J. Angeline, G. M. Saunders, and J. B. Pollack. (1994, Jan.). An Evolutionary Algorithm That Constructs Recurrent Neural Networks. IEEE Transactions on Neural Networks. [Online]. 5(1), pp. 54–65. https://doi.org/10.1109/72.265960

[27] X. Yao and Y. Liu. (1996, Mar.). Evolving Artificial Neural Networks Through Evolutionary Programming. Presented at Proceedings of the 5th Annual Conference on Evolutionary Programming, pp. 257–266. [Online].

[28] J. C. Park and S. T. Abusalah. (1997). Maximum Entropy: A Special Case of Minimum Cross-Entropy Applied to Nonlinear Estimation by an Artificial Neural Network. Complex Systems. [Online]. 11, pp. 289–307.

[29] E. Vonk, L. C. Jain, and R. Johnson. (1995, Dec.). Using Genetic Algorithms with Grammar Encoding to Generate Neural Networks. Presented at Proceedings of the IEEE International Conference on Neural Networks, vol. 4, pp. 1928–1931. [Online]. Available: https://doi.org/10.1109/ICNN.1995.488965

[30] I. Ioan, C. Rotar, and A. Incze. (2004, Sep.). The Optimization of Feed-Forward Neural Network Structure Using Genetic Algorithms. Presented at Proceedings of the International Conference on Theory and Applications of Mathematics and Informatics (ICTAMI), Thessaloniki, pp. 223–234. [Online].

[31] F. Zhao, T. Zhang, Y. Zeng, and B. Xu. (2017, Oct.). Towards a Brain-Inspired Developmental Neural Network by Adaptive Synaptic Pruning. Presented at Proceedings of the International Conference on Neural Information Processing, pp. 182–191. [Online]. Available: https://doi.org/10.1007/978-3-319-70093-9_19

[32] A. Ahmadi and B. Mashoufi. (2012, May). New Optimized Approach for Artificial Neural Networks Training Using Genetic Algorithms and Parallel Processing. International Review on Computers and Software. [Online]. 7(5), pp. 2232–2238.

[33] J. Frankle and M. Carbin. (2019, Mar.). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.1803.03635

[34] N. Lee, T. Ajanthan, and P. H. Torr. (2019, Feb.). SNIP: Single-Shot Network Pruning Based on Connection Sensitivity. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.1810.02340

[35] C. Wang, G. Zhang, and R. Grosse. (2020, Apr.). Picking Winning Tickets Before Training by Preserving Gradient Flow. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.2002.07376

[36] V. Sanh, T. Wolf, and S. Ruder. (2020, May). Movement Pruning: Adaptive Sparsity by Fine-Tuning. arXiv preprint. [Online]. Available: https://doi.org/10.48550/arXiv.2005.07683

[37] S. Han, J. Pool, J. Tran, and W. J. Dally. (2015, Dec.). Learning Both Weights and Connections for Efficient Neural Networks. Advances in Neural Information Processing Systems. [Online]. 28, pp. 1135–1143. Available: https://doi.org/10.48550/arXiv.1506.02626

[38] R. Morini, M. Bizzotto, F. Perrucci, F. Filipello, and M. Matteoli. (2021, Feb.). Strategies and Tools for Studying Microglial-Mediated Synapse Elimination and Refinement. Frontiers in Immunology. [Online]. 12, no. 640937. Available: https://doi.org/10.3389/fimmu.2021.640937

[39] R. P. Guidorzi, M. P. Losito, and T. Muratori. (1982, Oct.). The Range Error Test in the Structural Identification of Linear Multivariable Systems. IEEE Transactions on Automatic Control. [Online]. 27(5), pp. 1044–1054. Available: https://doi.org/10.1109/TAC.1982.1103068

[40] E. Abdul Jaleel and K. Aparna. (2015). Identification of Ethane-Ethylene Distillation Column Using Neural Network and ANFIS. Presented at Proceedings of the 5th International Conference on Advances in Computing and Communications (ICACC). [Online].