AI Sycophancy and the Emperor's new clothes
For those of you familiar with the famous tale of The Emperor's New Clothes by Hans Christian Andersen, in the realm of generative AI, sycophancy is, in a way, not so different from the subjects feigning agreement with the Emperor's claim, just as AI might similarly feign agreement with users’ views. This behavior can render the AI either blissfully ignorant or, worse, a pitiful liar.
As defined by Caleb Sponheim in a Nielsen Norman Group article, 'Sycophancy refers to instances in which an AI model adapts responses to align with the user’s view, even if the view is not objectively true. This behavior is generally undesirable.'
In this article, I want to showcase a specific example of this problem that I identified with a very exciting AI chatbot (powered by GPT) that acts as a 'Fashion Advisor' for a major Berlin-based fashion and technology company. This example illustrates the risks that these phenomena can pose in certain generative AI products. Coincidentally, it also relates to the famous tale that opens this very article.
While in all fairness, this particular chatbot is still in Beta across a few markets, I think it is still valuable for the community to grasp the potential risks of sycophancy as it could potentially erode customer trust in the blink of an eye.
The Emperor’s Fashion advisor
On a very simple question posed as a regular customer (and a fashion ignorant) to this AI agent, I asked the AI Fashion Advisor through different interactions to tell me which colors should never be combined in the world of fashion. In response, the agent diligently provided some combinations that are generally to be avoided while always noting fashion's subjectivity. I believe that this is a great way of giving advice, so far so good.
See the example through 3 different instances:
However, through multiple independent interactions with the AI Agent, using different accounts and specifying different genders, without any shopping history or browsing activity, the agent still recommended that I purchase a combination of clothes with the previously mentioned colors. This occurred even when I explicitly mentioned my lack of fashion knowledge and my intention to use them for casual occasions.
Moreover, this specific interaction risks setting up potential customers for failure, especially since, once again, it involves a customer with no purchase or browsing history.
From my perspective, this is a clear example of sycophancy, not merely an AI chatbot's attempt to make sales. However, customers might perceive it differently. Considering the experience from a potential customer's viewpoint, they might order the suggested items only to discover it doesn't look good on them upon trying it on. This could lead to returns, causing losses for the business, and ultimately affect customer’s trust in the chatbot for future fashion advice.
Opinion: A temporarily sales boost at the expense of customer’s trust
AI shopping assistants have the potential to significantly drive sales by streamlining the shopping experience. They can do this by offering a tailored, hands-on process that addresses common pain points, such as navigating excessive product listings or deciphering technical specifications, especially when catering to a diverse customer base. By providing a directed and limited product offering pre-filtered to match the customer’s profile, these tools can simplify the shopping process and potentially enhance the sales funnel.
However, the use of AI-powered shopping assistants can sometimes lead to shortcuts that jeopardize customer trust, particularly when there is inconsistency between recommendations, customer needs, and the actual products offered. While sycophancy—as an unintended bias in today's generative AI—may not be deliberate, it can come across as intentional sales-driven deception. This risks reminding customers of the tale’s sellers attempting to sell an invisible dress to an emperor, ultimately undermining trust in the brand.