Over the holidays, my cousin demoed chatGPT for us. He had it write five fortune cookie messages that were slightly better than the fortunes that came with our sushi.
It was my first encounter with the AI chatbot released in November by OpenAI. Exciting things were coming my way. Seemed like it.
Later, I gave chatGPT a proper test. I wanted to see if it could write well, make me laugh, and accurately cover the keto diet, sleep science, and other areas of expertise for myself and my health and wellness content agency.
I was excited but trepidatious. If chatGPT could check all these boxes, it would rapidly upend content marketing and many other fields. It would be like the Model T displacing the horse and buggy or Netflix displacing Blockbuster.
To be clear, it wouldn’t displace the need for content strategy. You’d still need humans to chart a course. But a good enough AI would obviate the need for content writers.
I should note that AI already plays a role in content marketing. For instance, my team uses AI-driven tools like Grammarly (for plagiarism, grammar, syntax) and Clearscope to optimize content for search. But we still write the content ourselves.
I’ll also mention that chatGPT isn’t the only artificial language generator. (Other services like copy.AI are powered by similar software.) But chatGPT is considered, per the New York Times, to be “the best artificial intelligence chatbot ever released to the general public.”
So was this AI ready to take over content writing? Let me share what I found.
What I found
First of all, I was impressed with chatGPT’s writing ability. The sentences were sometimes long but never convoluted.
And with the right prompt, it could be engaging too. Asked to “give a witty, funny three sentence explanation of the keto diet,” chatGPT responded:
The keto diet is like a relationship: it’s all about commitment. You have to be willing to say goodbye to carbs and hello to fats. But if you stick with it, you just might end up losing weight and feeling better about yourself. Just don’t be surprised if you end up feeling a little cranky and hungry in the beginning.
The analogy isn’t perfect (why would you feel hungry at the beginning of a relationship?), but the prose made me smile. More standard prompts (“describe the keto diet in a paragraph”) generated satisfactory, Wikipedia-style responses.
The responses, however, seemed stock. In other words, I would tweak the prompts and get similar explanations (same wording) of how keto worked. There might be a workaround, but that could prove an obstacle to keeping content engaging and on-brand.
And things went south when I asked for deeper dives. For example, here’s a reference it gave when I requested a summary of intermittent fasting:
Horne, B. D., Muhlestein, J. B., & Anderson, J. L. (2018). Intermittent fasting: A review of human studies. Journal of Lipid Research, 59(3), 301-312. https://doi.org/10.1194/jlr.R081723
The study isn’t published. (Did the AI make it up?) The link goes nowhere.
When I admonished chatGPT, it admitted the error and gave another reference. This time, the paper existed, but the link was broken.
This is the main knock against chatGPT and its ilk. (And the creators at OpenAI openly admit it.) The AI often gets facts wrong, but it proceeds confidently nonetheless.
It can write reasonably well, but you can’t trust it. Which means it’s not ready to replace good science writers yet. The next version—maybe. Not this one.
But fortune cookie writers better start polishing their resumes.
By the way, check out my free report on building traffic and trust with health and wellness content marketing.
And if you’re interested in working with my team, email me at brian@brianjstanton.com and we’ll set up a free consult.
