Using ChatGPT for Data Analysis: Use Cases and Limitations

ChatGPT, a generative AI chatbot based on a large language model, has seen a rapid rise in popularity. Its ability to understand and generate human-like text has opened new opportunities for technological applications and real-life use cases. This article focuses on using ChatGPT for data analysis, an area where its capabilities are both promising and challenging.

ChatGPT, developed by OpenAI, is a conversational agent designed to understand and generate human language. It's powered by a large language model, which allows it to interpret and respond to a wide range of text inputs.

To use ChatGPT for data analytics, a ChatGPT Plus subscription is required since file upload is not available for free users. Advanced Data Analysis is a feature within ChatGPT that allows users to upload data directly to ChatGPT and ask questions about the data.

Using ChatGPT for data analysis can be a powerful way to leverage AI in understanding and interpreting data. Here are some practical applications:

• Data Querying and Interpretation: ChatGPT can handle requests for data analysis, providing summaries and interpretations based on provided datasets.
• Predictive Analytics and Pattern Recognition: Leveraging its language model, ChatGPT can identify trends in the historical data.
• Assisting in Data Visualization: It can generate textual data trends descriptions and charts, aiding in the visualization process.

We leveraged ChatGPT's Advanced Data Analysis feature in various contexts, where it demonstrated both versatility and its limitations. Here’s a summary of how it performed in different scenarios.

Tools: Custom GPT, Code Interpreter.

Dataset: Arbitrary dataset with structured advertising data.

In this example, we used Code Interpreter and requested a Custom GPT to analyze and generate insights from an arbitrary advertising dataset with structured data. ChatGPT successfully created visualizations and identified key metric trends, such as cost, revenue, click-through rate (CTR), and conversions. While it pinpointed top-performing traffic sources and offered actionable suggestions, the final recommendations were generally broad and not actionable.

Tools: Custom GPT, Code Interpreter.

Dataset: Google Search Console Data Export.

In another case, we asked a Custom GPT to suggest SEO improvements. When analyzing Google Search Console Data, ChatGPT offered a regional performance comparison and recognized date-wise trends. Its analysis of top queries, search appearance, and devices led to valid, actionable recommendations. However, the final recommendations tended to be at a high level.

Tools: Custom GPT, Code Interpreter.

Dataset: Mixpanel MAU data.

In this example, a custom GPT was used to analyze and generate insights from Mixpanel MAU data. In analyzing Mixpanel's Monthly Active Users (MAU) data, ChatGPT produced visualizations and trendlines for key metrics. Although it provided a general overview of the data, both the observations and the recommendations lacked depth, emphasizing the need for more detailed, context-specific insights.

Tools: Custom GPT, Code Interpreter.

Dataset: Arbitrary customer feedback dataset with unstructured text.

ChatGPT's performance in analyzing unstructured text from customer feedback highlighted its limitations in handling incomplete data. It struggled with fields that were not fully populated, only generating insights from fully completed columns. Nonetheless, the insights it did provide were valid and well-summarized, showing its potential in text analysis when provided with complete data.

In all of the scenarios mentioned, ChatGPT was able to process and analyze data to some degree. However, the observations made were typically often too generic, lacking in specificity and detail.

AI-powered tools like ChatGPT mainly use a conversation style for interaction, which isn't always the best way to work with data. SQL, for example, was made to be somewhat like English, and getting answers from data through structured query language or query-builders might be more practical and accurate. Moreover, when chatting with data in plain English, users need to know exactly what questions to ask. Along with this, there are other challenges:

• Lack of Accuracy: LLM models are trained to provide results (predict tokens) and incentivized to provide a generated output, even if the result is not accurate (“AI hallucination” phenomena).
• Handling Large Datasets: ChatGPT struggles with processing large or complex datasets.
• Dependence on User Input: The quality of output depends heavily on the user's ability to ask the right questions.
• Challenges and Limitations Data Privacy and Security: Handling sensitive data raises concerns about privacy and security.
• Unprepared Data Limitations: ChatGPT faces challenges with unstructured or unprepared data.
• Lack of Contextual Business Understanding: It lacks a deep understanding of specific business contexts.

In addition to these limitations, over-reliance on AI for data interpretation can lead to misinterpretation and overlook crucial nuances that human analysts or rules-based non-AI systems might catch.

Utilizing ChatGPT effectively in data analysis involves crafting precise prompts and understanding its scope and limitations.

Exploring other AI-powered data analysis tools, like Narrative BI, which offers customized solutions based on a proprietary dataset, can complement ChatGPT's capabilities.

Anticipated improvements in AI models could enhance data analytics capabilities, making AI interfaces more robust and insightful.

While ChatGPT offers exciting possibilities in data analysis, it's crucial to be aware of its limitations and the need for human oversight. The role of AI in data analytics is evolving, and ChatGPT is a significant, albeit limited, part of this landscape.

Unlike ChatGPT, trained for generic tasks, Generative BI solutions offer more specialized functions and deeper insights compared to chat-based AI.

Consider trying Narrative BI, a generative business intelligence platform, for a more tailored data analysis experience, especially if you're seeking insights beyond what generic models like ChatGPT can offer.