Enhancing Channel Selection in 5G with Decentralized Federated Multi-Agent Deep Reinforcement Learning

Computer and Knowledge Engineering

مقاله 1، دوره 7، شماره 2 - شماره پیاپی 14، اسفند 2024، صفحه 1-16   

نوع مقاله: Original Article

شناسه دیجیتال (DOI): 10.22067/cke.2024.88900.1119

نویسندگان

Taghi Shahgholi¹؛ Keyhan Khamforoosh^{* 2}؛ Amir Sheikhahmadi²؛ Sadoon Azizi³

¹Department of Computer Engineering, Sanandaj Branch, Islamic Azad University ,Sanandaj, Iran

²Department of Computer Engineering, Sanandaj Branch, Islamic Azad University,Sanandaj, Iran

³Department of Computer Engineering and IT, University of Kurdistan,Sanandaj,Iran

چکیده

The increasing popularity of vehicular communication systems necessitates efficient and autonomous decision-making to address the challenges of vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications. In this paper, we present a comprehensive study on channelization in Cellular Vehicle-to-Everything (C-V2X) communication and propose a novel two-layer multi-agent approach that integrates deep reinforcement learning (DRL) and federated learning (FL) to enhance the decision-making process in channel utilization.

Our approach leverages the autonomy of each vehicle, treating it as an independent agent capable of making channel selection decisions based on its local observations in its own cluster. Simultaneously, a centralized architecture coordinates nearby vehicles to optimize overall system performance. The DRL-based decision-making model considers crucial factors, such as instantaneous channel state information and historical link selections, to dynamically allocate channels and transmission power, leading to improved system efficiency.

By incorporating federated learning, we enable knowledge sharing and synchronization among the decentralized vehicular agents. This collaborative approach harnesses the collective intelligence of the network, empowering each agent to gain insights into the broader network dynamics beyond its limited observations. The results of our extensive simulations demonstrate the superiority of the proposed approach over existing methods, as it achieves higher data rates, success rates, and superior interference mitigation.

کلیدواژه‌ها

C-V2X Optimization؛ Multi-Agent Learning؛ DRL-based Channel Access؛ Federated Learning Integration

مراجع

1. Obiodu, A. Raman, A. K. Abubakar, S. Mangiante, N. Sastry and A. H. Aghvami. (2022, Feb.). DSM-MoC as Baseline: Reliability Assurance via Redundant Cellular Connectivity in Connected Cars. IEEE Transactions on Network and Service Management. [Online]. 19(3), pp. 2178-2194. Available: 10.1109/TNSM.2022.3153452
2. Zhang, M. Peng, S. Yan and Y. Sun. (2019, Dec.). Deep-Reinforcement-Learning-Based Mode Selection and Resource Allocation for Cellular V2X Communications. IEEE Internet of Things Journal. [Online]. 7(7), pp. 6380-6391. Available: 10.1109/JIOT.2019.2962715
3. Yang, Y. Ju, L. Liu, Q. Pei, K. Yu and J. J. P. C. Rodrigues, “Secure mmWave C-V2X Communications Using Cooperative Jamming," in GLOBECOM 2022 - 2022 IEEE Global Communications Conference, Rio de Janeiro, Brazil, 2022, pp. 2686-2691
4. -X. Zheng et al. and K. K. W. (2022, Jun.). Physical-Layer Security of Uplink mmWave Transmissions in Cellular V2X Networks. IEEE Transactions on Wireless Communications. [Online]. 21(11), pp. 9818-9833. Available: https://doi.org/10.1109/TWC.2022.3179706
5. S. Bahbahani, E. Alsusa and A. Hammadi. (2022, Nov.). A Directional TDMA Protocol for High Throughput URLLC in mmWave Vehicular Networks. IEEE Transactions on Vehicular Technology. [Online]. 72(3), pp. 3584-3599. Available: 10.1109/TVT.2022.3219771
6. Xiang, H. Shan, Z. Su, Z. Zhang, C. Chen and E. -P. Li. (2022, Oct.). Multi-Agent Reinforcement Learning-Based Decentralized Spectrum Access in Vehicular Networks with Emergent Communication. IEEE Communications Letters. [Online]. 27(1), pp. 195-199. Available: https://doi.org/10.1109/LCOMM.2022.3214792
7. Sehla, T. M. T. Nguyen, G. Pujolle and P. B. Velloso. (2022, Mar.). Resource Allocation Modes in C-V2X: From LTE-V2X to 5G-V2X. IEEE Internet of Things Journal. [Online]. 9(11), pp. 8291-8314. Available: https://doi.org/10.1109/JIOT.2022.3159591
8. Bagheri et al. and K. Moessner. (2021, Mar.). 5G NR-V2X: Toward Connected and Cooperative Autonomous Driving. IEEE Communications Standards Magazine. [Online]. 5(1), pp. 48-54. Available: https://doi.org/10.1109/MCOMSTD.001.2000069
9. Twardokus and H. Rahbari. (2023, Mar.). Towards Protecting 5G Sidelink Scheduling in C-V2X Against Intelligent DoS Attacks. IEEE Transactions on Wireless Communications. [Online]. 22(11), pp. 7273-7286. Available: https://doi.org/10.1109/TWC.2023.3249665
10. Sehla, T. M. T. Nguyen, G. Pujolle and P. B. Velloso. (2022, Mar.). Resource Allocation Modes in C-V2X: From LTE-V2X to 5G-V2X. IEEE Internet of Things Journal. [Online]. 9(11), pp. 8291-8314. Available: https://doi.org/10.1109/JIOT.2022.3159591
11. -H. Wu, R. -H. Hwang, C. -Y. Wang and C. -H. Chou. “Deep Reinforcement Learning Based Resource Allocation for 5G V2V Groupcast Communications,” in 2023 International Conference on Computing, Networking and Communications (ICNC), Honolulu, HI, USA, 2023, pp. 1-6.
12. Lin, J. G. Andrews, A. Ghosh and R. Ratasuk. (2014, Apr.). An overview of 3GPP device-to-device proximity services.  IEEE Communications Magazine. [Online]. 52(4), pp. 40-48. Available: https://doi.org/10.1109/MCOM.2014.6807945
13. Chen et al. and R. Zhao. (2017, Jul.). Vehicle-to-Everything (v2x) Services Supported by LTE-Based Systems and 5G. IEEE Communications Standards Magazine. [Online]. 1(2), pp. 70-76. Available: https://doi.org/10.1109/MCOMSTD.2017.1700015
14. Ran, “Study on Evaluation Methodology of New V2X Use Cases for LTE and NR,” Dubrovnik, Croatia, 2017.
15. H. C. Garcia et al. and T. Şahin. (2021, Feb.). A Tutorial on 5G NR V2X Communications. IEEE Communications Surveys & Tutorials. [Online]. 23(3), pp. 1972-2026. Available: https://doi.org/10.1109/COMST.2021.3057017
16. Shahgholi, A. Sheikhahmadi, K. Khamforoosh, and S. Azizi. (2021, Feb.). LPWAN-based hybrid backhaul communication for intelligent transportation systems. Architecture and performance evaluation. EURASIP Journal on Wireless Communications and Networking. [Online]. (35). Available: https://doi.org/10.1186/s13638-021-01918-2
17. Yang, L. Zhao, L. Lei and K. Zheng, “A two-stage allocation scheme for delay-sensitive services in dense vehicular networks,” IEEE International Conference on Communications Workshops (ICC Workshops), Paris, France, 2017, pp. 1358-1363. Available: https://doi.org/10.1109/ICCW.2017.7962848
18. Liang, S. Xie, G. Y. Li, Z. Ding and X. Yu. (2018, Feb.). Graph-Based Resource Sharing in Vehicular Communication. IEEE Transactions on Wireless Communications. [Online]. 17(7), pp. 4579-4592. Available: https://doi.org/10.1109/TWC.2018.2827958
19. Chen, B. Wang and R. Zhang. (2018, Oct.). Interference Hypergraph-Based Resource Allocation (IHG-RA) for NOMA-Integrated V2X Networks. IEEE Internet of Things Journal. [Online]. 6(1), pp. 161-170. Available: https://doi.org/10.1109/JIOT.2018.2875670
20. Bai, W. Chen, K. B. Letaief and Z. Cao. (2010, Dec.). Low Complexity Outage Optimal Distributed Channel Allocation for Vehicle-to-Vehicle Communications. IEEE Journal on Selected Areas in Communications. [Online]. 29(1), pp. 161-172. Available: https://doi.org/10.1109/JSAC.2011.110116
21. Liang, J. Kim, S. C. Jha, K. Sivanesan and G. Y. Li. (2017, May.). Spectrum and Power Allocation for Vehicular Communications With Delayed CSI Feedback. IEEE Wireless Communications Letters. [Online]. 6(4), pp. 458-461. Available: https://doi.org/10.1109/LWC.2017.2702747
22. Ye, G. Y. Li and B. -H. F. Juang. (2019, Feb.). Deep Reinforcement Learning Based Resource Allocation for V2V Communications. IEEE Transactions on Vehicular Technology. [Online]. 68(4), pp. 3163-3173. Available: https://doi.org/10.1109/TVT.2019.2897134
23. Wang, X. Zheng and X. Hou, “A Novel Semi-Distributed Transmission Paradigm for NR V2X,”  in IEEE Globecom Workshops (GC Wkshps), Waikoloa, HI, USA, 2019, pp. 1-6.
24. Mosavat-Jahromi, Y. Li, L. Cai and L. Lu, “NC-MAC: Network Coding-based Distributed MAC Protocol for Reliable Beacon Broadcasting in V2X,” GLOBECOM 2020 - 2020 IEEE Global Communications Conference, Taipei, Taiwan, 2020, pp. 1-6.
25. Zappone, M. Di Renzo and M. Debbah. (2019, Jun.). Wireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both?. IEEE Transactions on Communications. [Online]. 67(10), pp. 7331-7376. Available: https://doi.org/10.1109/TCOMM.2019.2924010
26. Wang, H. Ye, L. Liang and G. Y. Li. (2020, Mar.). Learn to Compress CSI and Allocate Resources in Vehicular Networks. IEEE Transactions on Communications. [Online]. 68(6), pp. 3640-3653. Available: https://doi.org/10.1109/TCOMM.2020.2979124
27. Lu, Yan, Ping Wang, Shuai Wang, and Wangding Yao, “A Q-learning based SPS resource scheduling algorithm for reliable C-V2X communication,” in 5th International Conference on Digital Signal Processing, 2021, pp. 201-206.
28. Busoniu, R. Babuska and B. De Schutter. (2008, Feb.). A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). [Online]. 38(2), pp. 156-172. Available: https://doi.org/10.1109/TSMCC.2007.913919
29. Gu, W. Chen, M. Alazab, X. Tan and M. Guizani. (2022, Jul.). Multiagent Reinforcement Learning-Based Semi-Persistent Scheduling Scheme in C-V2X Mode 4. IEEE Transactions on Vehicular Technology. [Online]. 71(11), pp. 12044-12056. Available: https://doi.org/10.1109/TVT.2022.3189019
30. Cao and H. Yin, “Resource Allocation for Vehicle Platooning in 5G NR-V2X via Deep Reinforcement Learning,” in 2021 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), Bucharest, Romania, 2021, pp. 1-7.
31. Yuan, G. Zheng, K. -K. Wong and K. B. Letaief. (2021, Jul.). Meta-Reinforcement Learning Based Resource Allocation for Dynamic V2X Communications. IEEE Transactions on Vehicular Technology. [Online]. 70(9), pp. 8964-8977. Available: https://doi.org/10.1109/TVT.2021.3098854
32. He, L. Wang, H. Ye, G. Y. Li and B. -H. F. Juang, “Resource Allocation based on Graph Neural Networks in Vehicular Communications,” GLOBECOM 2020 - 2020 IEEE Global Communications Conference, Taipei, Taiwan, 2020, pp. 1-5.
33. Wang, X. Jiang, Y. Zhou, Z. Li, D. Wu, T. Tang, A. Fedotov and V. Badenko. (2022, Jun.). Multi-agent reinforcement learning for edge information sharing in vehicular networks. Digital Communications and Networks. [Online]. 8(3), pp. 267-277. Available: https://doi.org/10.1016/j.dcan.2021.08.006
34. Wu, Z. Liu, F. Liu, T. Yoshinaga, Y. Ji and J. Li. (2020, Jun.). Collaborative Learning of Communication Routes in Edge-Enabled Multi-Access Vehicular Environment. IEEE Transactions on Cognitive Communications and Networking. [Online]. 6(4), pp. 1155-1165. Available: https://doi.org/10.1109/TCCN.2020.3002253
35. Gu, X. Yang, Z. Lin, W. Hu, M. Alazab and R. Kharel. (2020, Sep.). Multiagent Actor-Critic Network-Based Incentive Mechanism for Mobile Crowdsensing in Industrial Systems. IEEE Transactions on Industrial Informatics. [Online]. 17(9), pp. 6182-6191. Available: https://doi.org/10.1109/TII.2020.3024611
36. Liang, H. Ye and G. Y. Li. (2019, Aug.). Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning. IEEE Journal on Selected Areas in Communications. [Online]. 37(10), pp. 2282-2292. Available: https://doi.org/10.1109/JSAC.2019.2933962
37. Tian, Q. Liu, H. Zhang and D. Wu. (2021, Jun.). Multiagent Deep-Reinforcement-Learning-Based Resource Allocation for Heterogeneous QoS Guarantees for Vehicular Networks. IEEE Internet of Things Journal. [Online]. 9(3), pp. 1683-1695. Available: https://doi.org/10.1109/JIOT.2021.3089823
38. Chen et al. and Y. Zhang. (2020, Jan.). Age of Information Aware Radio Resource Management in Vehicular Networks: A Proactive Deep Reinforcement Learning Perspective. IEEE Transactions on Wireless Communications. [Online]. 19(4), pp. 2268-2281. Available: https://doi.org/10.1109/TWC.2019.2963667
39. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, 2017,pp. 1273-1282.
40. Niknam, H. S. Dhillon and J. H. Reed. (2020, Jun.). Federated Learning for Wireless Communications: Motivation, Opportunities, and Challenges. IEEE Communications Magazine. [Online]. 58(6), pp. 46-51. Available: https://doi.org/10.1109/MCOM.001.1900461
41. Qi, Q. Zhou, L. Lei and Kan Zheng, “Federated reinforcement learning: Techniques, applications, and open challenges,”  2021.
42. Valente, C. Senna, P. Rito and S. Sargento. (2023, Feb.). Embedded Federated Learning for VANET Environments. Applied Sciences. [Online]. 13(4), p. 2329. Available: https://doi.org/10.3390/app13042329
43. Wang, X. Fangmin, H. Zhang and C. Zhao. (2022, Apr.). Joint resource management for mobility supported federated learning in Internet of Vehicles. Future Generation Computer Systems.[Online]. 129, pp. 199-211. Available: https://doi.org/10.1016/j.future.2021.11.020
44. Guo, F. Tang and N. Kato. (2022, Sep.). Federated Reinforcement Learning-Based Resource Allocation in D2D-Enabled 6G. IEEE Network. [Online]. 37(5), pp. 89-95. Available: https://doi.org/10.1109/MNET.122.2200102
45. Kandukuri and S. Boyd. (2002, Jan.). Optimal power control in interference-limited fading wireless channels with outage-probability specifications. IEEE Transactions on Wireless Communications. [Online]. 1(1), pp. 46-55. Available: https://doi.org/10.1109/7693.975444
46. Li, L. Lu, W. Ni, A. Jamalipour, D. Zhang and H. Du. (2022, May.). Federated Multi-Agent Deep Reinforcement Learning for Resource Allocation of Vehicle-to-Vehicle Communications. IEEE Transactions on Vehicular Technology. [Online]. 71(8), pp. 8810-8824. Available: https://doi.org/10.1109/TVT.2022.3173057
47. M. Meredith, “Technical Specification Group Radio Access Network; Study on LTEBased V2X Services; (Release 14),” 3rd Generation Partnership Project, 2016.
48. Guo, L. Liang and G. Y. Li. (2019, Feb.). Resource Allocation for Low-Latency Vehicular Communications: An Effective Capacity Perspective. IEEE Journal on Selected Areas in Communications. [Online]. 37(4), pp. 905-917. Available: https://doi.org/10.1109/JSAC.2019.2898743
49. Zeng et al and Y. Wu, “Multi-D3QN: A multi-strategy deep reinforcement learning for service composition in cloud manufacturing,” International Conference on Collaborative Computing: Networking, Applications and Worksharing. Cham: Springer International Publishing, 2021, pp. 225-240.
50. Yang, X. Xie and M. Kadoch. (2019, Jan.). Intelligent Resource Management Based on Reinforcement Learning for Ultra-Reliable and Low-Latency IoV Communication Networks. IEEE Transactions on Vehicular Technology. [Online]. 68(5), pp. 4157-4169. Available: https://doi.org/10.1109/TVT.2018.2890686
51. Chen, J. Hu, Y. Shi and L. Zhao. (2016, Sep.). LTE-V: A TD-LTE-Based V2X Solution for Future Vehicular Network. IEEE Internet of Things Journal. [Online]. 3(6), pp. 997-1005. Available: https://doi.org/10.1109/JIOT.2016.2611605
52. L. Liang, S. Xie, G. Y. Li, Z. Ding and X. Yu. (2018, Apr.). Graph-Based Resource Sharing in Vehicular Communication. IEEE Transactions on Wireless Communications. [Online]. 17(7), pp. 4579-4592. Available: https://doi.org/10.1109/TWC.2018.2827958