ANALYSIS OF THE EFFECTIVENESS OF GENERATIVE AI MODELS FOR TEXT-TO-SQL TASKS IN BUSINESS INTELLIGENCE SYSTEMS

Abstract

The rising integrations of Generative Artificial Intelligence (AI) into Business Intelligence (BI) systems has transformed how organizations interact with data and enabled natural language based analytical reasoning through text-to-SQL generations. This study provides a comprehensive analysis of the effectiveness of recent Generative AI and Large Language Models (LLM) in executing text-to-SQL tasks. The study presents a comparative review of features and limitations of current and advanced text-to-SQL systems including MAC-SQL, SQL-PaLM, CHASE-SQL, and CHESS. Further, an overview of generative AI models with their architectural outline is also provided to highlight the evolution of large language model based approaches in structured data querying. Three state-of-the-art generative AI models has been selected including LLaMA 4, Qwen3 14B, Mixtral 8x7B to perform evaluations using custom evolution framework and a hybrid prompt template is also designed for text-to-SQL tasks. Evaluation metrics includes intent clarification accuracy, semantic clarification accuracy, SQL generation accuracy, execution response accuracy, and response latency. Further, the experimental results demonstrates that LLaMA 4 model acquired great overall performance across all major evaluation metrics. The findings confirms the effectiveness of Generative AI for text-to-SQL generation and highlights its potential in designing a next-generation, intelligent business intelligence systems. Future work will extend this research towards multi-agent and domain-related real-time business intelligence framework.