High-Dimensional Regression Analysis and Artificial Intelligence: Theory, Methods and Applications

In this book, a novel high-dimensional linear and nonlinear regression model is introduced to address, in part, the challenges of evaluating the stability and confidence of large-scale models' interpretability. The book begins by reviewing foundational concepts in regression analysis, and discussing the current state and challenges of AI interpretability. Through an in-depth exploration of regression models, the core principles of data-driven linear regression are explained. To enhance the explanatory power of regression models, variable-parameter regression models are further investigated and extended to variable-parameter nonlinear regression models. To handle complex relationships, the Gauss-Newton iterative method is incorporated, ensuring the stability of high-dimensional nonlinear regression. The Confidence Interval-based Credibility Evaluation (CICE) framework combines statistical indicators—such as interval width, center deviation, and accuracy—into a single score to assess the stability and reliability of explanations, validated through case studies in engineering, finance, and time series prediction. Overall, the book presents a coherent framework for interpretable AI, integrating regression modeling, confidence region construction, and credibility evaluation to enhance interpretability and statistical accountability, fostering more trustworthy AI systems. Chapter 1 introduces the fundamental concepts and theoretical developments of both regression analysis and AI explainability, highlighting their interconnections. Chapter 2 reviews essential probability theory and mathematical statistics, covering random variables, measure spaces, probability distributions, parameter estimation (including least squares and maximum likelihood methods), and asymptotic theory, which serve as the foundation for analyzing model consistency and convergence. Chapter 3 focuses on the effects of correlated errors in linear regression, establishing parameter convergence conditions to ensure the consistency and asymptotic normality of covariance estimators. Chapter 4 introduces variable-parameter regression models and systematically studies M-estimators and generalized regression models within the framework of robust statistics. By addressing non-normal errors and outliers, these methods improve model adaptability. The chapter also establishes the robustness of the generalized regression model through theoretical analysis of covariance estimation. Chapter 5 introduces the Confidence Interval-based Credibility Evaluation (CICE) framework, which integrates multiple statistical indicators into a unified score to assess the stability and reliability of model explanations. Through real-world case studies in engineering, finance, and time series prediction, the effectiveness of CICE in detecting unstable interpretations and enhancing model transparency is demonstrated.

This book is the result of a longstanding collaboration between two leading scholars. Professor Asaf Hajiyev, a full member of the Azerbaijan National Academy of Sciences, is renowned for his work in probability theory, mathematical statistics, and stochastic modeling. He has authored over 150 scientific papers and several books, and currently heads the Department of Statistical Modeling at the Institute of Control Systems.

Professor Jiuping Xu, Distinguished Professor at Sichuan University and academician of several international academies, specializes in foundational modeling for computer science and management science. He has led over 80 national research projects and published more than 900 papers and 40 books. Together, their teams have developed innovative methodologies to evaluate the stability of AI interpretability using regression-based confidence regions. Their interdisciplinary partnership, active since 2010, combines theoretical rigor with practical insight to support robust and explainable decision-making in artificial intelligence.