Spatiotemporal Sequence Prediction Based on Spatiotemporal Self-Attention Mechanism
DOI:
https://doi.org/10.15837/ijccc.2024.6.6771Keywords:
Spatiotemporal prediction; Self-attention mechanisms; Graph Convolutional Networks; Transformer architecturesAbstract
This paper introduces the GCN-Transformer model, an innovative approach that combines Graph Convolutional Networks (GCNs) and Transformer architectures to enhance spatiotemporal sequence prediction. Targeted at applications requiring precise analysis of complex spatial and temporal data, the model was tested on two distinct datasets: PeMSD8 for traffic flow and KnowAir for air quality monitoring. The GCN-Transformer demonstrated superior performance over traditional models such as LSTMs, standalone GCNs, and other GCN-hybrid models, evidenced by its lower Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE). An ablation study confirmed the importance of each component within the model, showing that removing elements like GCN layers, Transformer layers, attention mechanisms, or positional encoding detrimentally impacts performance. Overall, the GCN-Transformer model offers significant theoretical and practical contributions to the field of spatiotemporal data analysis, with potential applications across traffic management, environmental monitoring, and beyond.
References
Tu, Q., Geng, G. & Zhang, Q. (2023). Multi-Step Subway Passenger Flow Prediction under Large Events Using Website Data, Tehnički vjesnik, 30 (5), 1585-1593. https://doi.org/10.17559/TV-20230227000384
Chang, D., Wang*, Y. & Fan, R. (2022). Forecast of Large Earthquake Emergency Supplies Demand Based on PSO-BP Neural Network, Tehnički vjesnik, 29 (2), 561-571. https://doi.org/10.17559/TV-20211120092137
Christalin Nelson, S., Tapan Kumar, M., & Prakash G.L. (2022). A Novel Optimized LSTM Networks for Traffic Prediction in VANET, Journal of System and Management Sciences, 12(1), 461-479.
Wu, Z., Huang, M., Xing, Z., & Yang, T. (2024). Improving Short-Term Traffic Flow Prediction using Grey Relational Analysis for Data Filtering and Stacked LSTM Modeling, INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL, 19(1). https://doi.org/10.15837/ijccc.2024.1.6149
Zheng, C., Fan, X., Pan, S., Jin, H., Peng, Z., Wu, Z., Wang, C., & Yu, P. S. (2023). Spatiotemporal joint graph convolutional networks for traffic forecasting, IEEE Transactions on Knowledge and Data Engineering, 1-14. https://doi.org/10.1109/TKDE.2023.3284156
Wang, S., Li, Y., Zhang, J., Meng, Q., Meng, L., & Gao, F. (2020). PM2.5-GNN, Proceedings of the 28th International Conference on Advances in Geographic Information Systems. https://doi.org/10.1145/3397536.3422208
Geng, X., Li, Y., Wang, L., Zhang, L., Yang, Q., Ye, J.,& Liu, Y. (2019). Spatiotemporal multigraph convolution network for ride-hailing demand forecasting, Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 3656-3663. https://doi.org/10.1609/aaai.v33i01.33013656
Wang, S., Cao, J., & Yu, P. S. (2022). Deep learning for spatio-temporal data mining: A survey, IEEE Transactions on Knowledge and Data Engineering, 34(8), 3681-3700. https://doi.org/10.1109/TKDE.2020.3025580
Nohekhan, A., Zahedian, S., & Haghani, A. (2021). A deep learning model for off-ramp hourly traffic volume estimation, Transportation Research Record: Journal of the Transportation Research Board, 2675(7), 350-362. https://doi.org/10.1177/03611981211027151
Guo, S., Lin, Y., Feng, N., Song, C., & Wan, H. (2019). Attention based spatial-temporal graph convolutional networks for traffic flow forecasting, Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 922-929. https://doi.org/10.1609/aaai.v33i01.3301922
Hoque, J. M., Erhardt, G. D., Schmitt, D., Chen, M., Chaudhary, A., Wachs, M., & Souleyrette, R. R. (2021). The changing accuracy of traffic forecasts, Transportation, 49(2), 445-466. https://doi.org/10.1007/s11116-021-10182-8
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat. (2019). Deep learning and process understanding for data-driven Earth System Science, Nature, 566(7743), 195-204. https://doi.org/10.1038/s41586-019-0912-1
Yu, B., Yin, H., & Zhu, Z. (2018). Spatio-temporal graph convolutional networks: A Deep Learning Framework for traffic forecasting, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, . https://doi.org/10.24963/ijcai.2018/505
Fu, X., & Zhang, L. (2021). Spatio-temporal feature fusion for real-time prediction of TBM operating parameters: A deep learning approach, Automation in Construction, 132, 103937. https://doi.org/10.1016/j.autcon.2021.103937
Xu, G., Lin, K., Li, X., & Ye, Y. (2022). SAF-Net: A spatio-temporal deep learning method for typhoon intensity prediction, Pattern Recognition Letters, 155, 121-127. https://doi.org/10.1016/j.patrec.2021.11.012
Bhardwaj, N., Pal, A., Bhumika, & Das, D. (2024). Adaptive Context based Road Accident Risk Prediction using Spatio-temporal Deep Learning, IEEE Transactions on Artificial Intelligence 1-12. https://doi.org/10.1109/TAI.2023.3328578
Zhao, S., Zhao, K., Xia, Y., & Jia, W. (2022). Hyper-clustering enhanced spatio-temporal deep learning for traffic and demand prediction in bike-sharing systems, Information Sciences, 612, 626-637. https://doi.org/10.1016/j.ins.2022.07.054
Modi, S., Bhattacharya, J.,& Basak, P. (2022). Multistep traffic speed prediction: A deep learning based approach using latent space mapping considering spatio-temporal dependencies, Expert Systems with Applications, 189, 116140. https://doi.org/10.1016/j.eswa.2021.116140
Pan, Z., Liang, Y., Wang, W., Yu, Y., Zheng, Y., & Zhang, J. (2019). Urban Traffic Prediction from Spatio-Temporal Data Using Deep Meta Learning, Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. https://doi.org/10.1145/3292500.3330884
Yang, H., Jiang, J., Zhao, Z., Pan, R.,& Tao, S. (2024). STVANet: A spatio-temporal visual attention framework with large kernel attention mechanism for citywide traffic dynamics prediction, Expert Systems with Applications, 254, 124466. https://doi.org/10.1016/j.eswa.2024.124466
Nikpour, B., & Armanfard, N. (2023). Spatio-temporal hard attention learning for skeleton-based activity recognition, Pattern Recognition, 139, 109428. https://doi.org/10.1016/j.patcog.2023.109428
Ma, C., Yan, L., & Xu, G. (2023). Spatio-temporal graph attention networks for traffic prediction, Transportation Letters, 1-11. https://doi.org/10.1080/19427867.2023.2261706
Chen, Y., Zheng, L., & Liu, W. (2022). Spatio-Temporal Attention-based Graph Convolution Networks for Traffic Prediction, 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). https://doi.org/10.1109/SMC53654.2022.9945522
Shi, X., Qi, H., Shen, Y., Wu, G., & Yin, B. (2021). A Spatial-Temporal Attention Approach for Traffic Prediction, IEEE Transactions on Intelligent Transportation Systems, 22(8), 4909-4918. https://doi.org/10.1109/TITS.2020.2983651
Zeng, H., Peng, Z., Huang, X., Yang, Y., & Hu, R. (2022). Deep spatio-temporal neural network based on interactive attention for traffic flow prediction, Applied Intelligence, 52(9), 10285-10296. https://doi.org/10.1007/s10489-021-02879-1
Murugesan, R. K., Madhu, K., Sambandam, J., & Malliga, L. (2023). Covid-19 Forecasting Using CNN Approach With A Halbinomial Distribution And A Linear Decreasing Inertia Weight-Based Cat Swarm Optimization, INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL. 18(1). https://doi.org/10.15837/ijccc.2023.1.4396
Chao, Z. (2023). Machine learning-based intelligent weather modification forecast in smart city potential area, Computer Science and Information Systems, 20(2), 631-656. https://doi.org/10.2298/CSIS220717018C
Wang, S., Song, A., & Qian, Y. (2023). Predicting smart cities' electricity demands using k-means clustering algorithm in smart grid, Computer Science and Information Systems, (00), 13-13.
KNEŽEVIĆ, D., BLAGOJEVIĆ, M., & RANKOVIĆ, A. (2023). Electricity Consumption Prediction Model for Improving Energy Efficiency Based on Artificial Neural Networks, Studies in Informatics and Control, 32(1), 69-79. https://doi.org/10.24846/v32i1y202307
Additional Files
Published
Issue
Section
License
Copyright (c) 2024 Yuan Zhao
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
ONLINE OPEN ACCES: Acces to full text of each article and each issue are allowed for free in respect of Attribution-NonCommercial 4.0 International (CC BY-NC 4.0.
You are free to:
-Share: copy and redistribute the material in any medium or format;
-Adapt: remix, transform, and build upon the material.
The licensor cannot revoke these freedoms as long as you follow the license terms.
DISCLAIMER: The author(s) of each article appearing in International Journal of Computers Communications & Control is/are solely responsible for the content thereof; the publication of an article shall not constitute or be deemed to constitute any representation by the Editors or Agora University Press that the data presented therein are original, correct or sufficient to support the conclusions reached or that the experiment design or methodology is adequate.
