Will AI take over the world and create havoc for the human race? It might, but before HAL shows up, the mess caused will be because of the flawed systems humans built. While cloud providers and consulting companies are rushing to sell AI, it’s worth asking the question: does it actually work and can you trust it? It’s easy to get seduced by seeing AI generate text or computer code to your prompt, but try something more complex, like using AI for content and data analysis, and a very different picture emerges. This article is about our experience and we hope it’s helpful as you’re thinking about how to use AI, perhaps in your content marketing or elsewhere in your company.
We started out by simply wanting to analyze the 100 best books of the century recently published by the NYT. We were interested in analyzing some simple statistics, like gender distribution and the number of books in translation. We didn’t get very far because the AI tool we used (ChatGPT v4o) never got it right.
We started off by asking some straightforward questions. The first response was not encouraging as it provided two different sets of numbers for gender distribution—first reporting 46 books by women and 54 by men, then in the same response changing to 40 books by women and 32 by men.
Next, we wanted to know how many of the 100 were fiction and nonfiction. The response was 81 fiction books and 19 nonfiction books. We presumed the information was correct and proceeded to ask more complex questions, like categorizing the books by theme. The list we got back included titles that were not among the 100 books in our original file.
After several more attempts with similarly erroneous answers and a prompt reset, we eventually got 76 fiction and 24 nonfiction titles. Next, we asked the AI tool to create a chart, but even after several tries, we kept getting the same response: an issue with generating the chart. When we asked it to export the data so we could create a chart ourselves, the output included an author who had been dead for a long time and certainly isn’t publishing posthumously: Jean-Paul Sartre.
Our experience with AI (specifically ChatGPT) being unable to correctly analyze data, “hallucinating” by erroneously inserting data not present in the original dataset, and failing to consistently generate charts makes AI an unreliable tool. For tasks beyond rewriting simple text, such as emails, or basic code for a website, it’s hard to see how the current iteration of AI could assist with more complex tasks in the near future. In other disciplines where AI is touted as a solution for data analysis, such as healthcare, accounting, or travel, it seems highly improbable that AI could be trusted anytime soon.
While we will continue to review the state of AI for its potential in content creation, the results we are seeing indicate that anything requiring research, understanding of data, and factual accuracy still demands significant human involvement at every step in the process. This is why our clients rely on us instead of defaulting to AI-generated content.
About the Author
Ren Agarwal is the CEO of StoryAZ Studio. With a passion for helping clients develop and implement successful marketing strategies, Ren brings a wealth of experience and insight into the ever-changing world of marketing and communications.