In the fast-paced world of artificial intelligence, LLMs (Large Language Models) have become essential tools for automating complex tasks like content writing. But can these models really write texts that captivate and connect with their audience? To answer this question, we put three LLMs – Claude Sonnet 3.5, ChatGPT-4o, and Gemini 1.5 Pro – to the test by challenging them to create a welcome email for new customers of "Tess.
The goal was simple: generate a welcome email that was informative, inviting, and, above all, human. To evaluate the performance of the LLMs, we analyzed the emails based on four essential criteria for each of the models:
The Prompt:
We used the Tess AI Multi-LLM chat and provided the following briefing to the models:
The Results? A true battle of styles!
Each model left its own mark on the welcome email, showcasing its strengths and distinct styles:
Claude Sonnet 3.5:
Content Quality and Relevance: The Claude Sonnet 3.5 email provided relevant information about the platform and its benefits, but it fell short in detailing how users can start using the tool right away.
Personalization and Tone of Voice: The tone was positive and friendly, but personalization was limited to using the customer's name.
Structure, Organization, and Humanized Writing: The structure was organized, but the writing included "markety" phrases that felt unnatural.
Grammar, Spelling, and Effectiveness: The email was free from grammar and spelling errors, but the call to action was too vague.
ChatGPT-4o:
Content Quality and Relevance: ChatGPT-4o, in turn, followed the same approach, highlighting the platform's benefits, but it failed to present a practical guide for new users.
Personalization and Tone of Voice: ChatGPT-4o also showed good performance in tone of voice, using friendly and enthusiastic language, but personalization was limited to mentioning the client's name.
Structure, Organization, and Humanized Writing: The structure was organized, but some sentences felt "stiff," lacking natural flow.
Grammar, Spelling, and Effectiveness: The email was free from grammatical and spelling mistakes, and the call-to-action was more specific, but it could still be more targeted.
Gemini 2.0 Pro:
Content Quality and Relevance: The Gemini 2.0 Pro stood out in this area. The generated email included relevant and user-focused information, highlighting the platform's benefits and adding a clear call-to-action to access resources and the community.
Personalization and Tone of Voice: The Gemini 2.0 Pro opted for a more professional tone (despite the overuse of emojis) and less personal approach, prioritizing the brand's presentation and features. While this approach wasn't as close to the user, it better fits the platform's objective. However, the model failed to add personalization beyond using the customer's name.
Structure, Organization, and Humanized Writing: The email generated by Gemini 2.0 Pro stood out for having a clear structure, a more natural tone closer to the user, and for avoiding "marketing-speak" phrases. However, the excessive use of emojis may have hurt the email's formality, making it less professional.
Grammatical and Spelling Accuracy, and Effectiveness: The email was free of grammatical and spelling errors, and the call to action was clear and specific, including links to resources and the community.
Gemini 1.5 Pro stood out for delivering a more natural tone closer to the user, with an organized structure and a clear, targeted call to action. However, the excessive use of emojis and lack of personalization harmed the email's formality, making it less professional. Claude Sonnet 3.5 and ChatGPT-4o, while showing great performance, still exhibit "robotic quirks" such as stiff content and exaggerated adjectives.
This analysis shows us that LLMs are evolving quickly, but there’s still a long way to go to achieve natural and personalized human-like writing. Choosing the ideal LLM for your needs will depend on the type of text you need to generate and the target audience you want to reach.
Remember, the ultimate goal is to build strong relationships with your customers, and humanized and authentic writing still plays a key role in this process.
