83 The AI Analyst Experiment. Hype or Reality?

All names in this article have been changed to maintain confidentiality and protect financial information. Any resemblance to real individuals is purely coincidental.

Can AI replace data analysts? The team, led by Alex Kolokolov, decided to put this to the test. They challenged AI with a real-world task: analyzing webinar data, calculating metrics, and generating a dashboard.

Alex Kolokolov is a business intelligence expert, bestselling author of “Data Visualization with Microsoft Power BI,” and the founder of the Make your Data Speak award.

Attempts to Upload the Data

The data from the CRM was stored in Excel across two sheets: one containing webinar details (topic, author, attendees, marketing expenses) and the other tracking course sales to webinar participants. The data was linked via Webinar ID.

The experts of the team uploaded the files to popular AI tools:

• Microsoft Copilot refused to work with .xlsx files, suggesting .pdf instead, which was not acceptable.
• Claude.ai does not support .xlsx but can read .csv, though it struggles with multiple tables.
• DeepSeek processed the data inconsistently.
• GPT-4 performed the best.

GPT-4 understood the data structure and answered questions but made errors in calculating ROI and conversion calculations, failing to account for the need to merge data from both sheets. We adjusted the prompt and reran the calculation, ensuring the correct format (1 webinar = 1 row).

Conclusion: AI does not always grasp table logic. Queries must be as detailed as possible.

Key Metrics Calculation

Now everything is ready to fully analyze the data. With AI’s assistance, there are a few key metrics have been selected and calculated:

• Which webinars convert best into course purchases.
• Best-performing hosts based on conversion rate from webinar to course purchase.
• Most popular webinar topics (not tied to specific hosts).
• ROI of each past webinar (not tied to specific hosts)

The first three tasks were solved correctly, but ROI calculations again presented issues—AI generated 50 rows instead of 10. After refining the prompt, the correct results were obtained.

Conclusion: AI forgets context in long dialogues. Always verify results manually and cross-check calculations using other tools, such as BI platforms.

Data Visualization

AI generated charts without major errors but overlooked details: axes were not labeled, and unnecessary grid lines were not removed. In text analysis, some “hallucinations” occurred—AI provided marketing advice that was not based on actual data.

Before & after:  

Conclusion: AI can be used for initial visualization, but manual verification is necessary. Plan ahead for what key information each chart should convey and check its clarity. The book “Data Visualization with Microsoft Power BI” can be a useful resource for this.

The Results

For now, an AI analyst is more like an intern—it can perform calculations but requires constant supervision. Errors occur at every stage, from data loading to calculations and visualization.

Key issues:

• File format limitations.
• Confusion in data relationships.
• Requires highly detailed instructions.
• Struggles with large data volumes.

Main takeaway: AI is useful, but critical thinking and verification are essential. It cannot yet operate autonomously. Analysts can rest easy for now.