We’re sharing the new research and NBI.AI-1, the latest milestone in Narrative BI’s effort to democratize data insights.
The research by Narrative BI focuses on hybrid approaches to generating business data insights from structured data, combining the strengths of rule-based systems and Large Language Models (LLMs). Rule-based analytics systems are precise but lack adaptability. LLMs are great for pattern recognition but are too generic for specific business cases. We're introducing:
The research outlines a hybrid approach that leverages the robustness of rule-based systems with the adaptive power of LLMs. By doing so, we aim to improve the process of data extraction and uncover meaningful data insights from diverse data sources using advanced AI data analysis techniques. The hybrid method combines AI techniques with rule-based systems and supervised document classification, creating a powerful framework for business data analysis. LLMs play a crucial role by modeling linguistic characteristics and generating coherent responses, thereby uncovering personalized user interests, needs, and goals from user journeys and activities.
Key considerations for implementing a hybrid approach include ensuring high data quality, understanding the specific business domain, and having sufficient computational resources. The research highlights the importance of maintaining transparency and trustworthiness in the data extraction process.
The hybrid approach's effectiveness was benchmarked against purely rule-based and LLM data analytics methods. The results demonstrated that the hybrid model offers a balanced solution, leveraging the precision of rule-based analysis and the flexibility and depth of LLM-generated data insights. This integration enhances the quality of data insights generated, ensuring they are actionable and accessible to decision-makers.
The data used for the benchmarking was collected from 30 corporate Google Analytics 4 and Google Ads accounts via APIs for a time frame of approximately two years.
In this evaluation, we extracted and calculated business metrics (such as cost-per-click, new users, website sessions, etc.) for the required period using different methods: rule-based query builder, AI answer generator (built with ChatGPT API), and Hybrid Approach (AI Data Analyst based on the NBI.AI-1 model).
We evaluated different methods of extracting and presenting all relevant business data insights from the dataset, a crucial factor for comprehensive analysis.
User satisfaction can be influenced by factors like the accuracy of the information, the relevance, comprehensiveness, and readability of the reports provided. This metric is measured as the ratio of “likes” (reports marked as “helpful” by business users) to “dislikes” (marked as “not helpful”). The bigger the number, the higher the overall user satisfaction.
LLM systems are prone to data hallucinations: they confidently generate responses that look plausible, but that are entirely incoherent or inaccurate. We measured the percentage of data hallucinations by comparing responses to actual data in the dataset.
In conclusion, Narrative BI's hybrid approach offers a more dynamic and precise tool for business analytics. Our findings suggest that the hybrid approach not only enhances the precision of data insights but also improves their contextual relevance, making the AI data analysis process more comprehensive and actionable. By combining the strengths of rule-based systems and LLMs, the hybrid model addresses the limitations of each method when used independently. This research provides a foundation for developing more resilient and insightful practices of using Generative AI for data analytics, driving growth and innovation in today's data-driven business environment.
