The recent news about Klarna’s new AI assistant powered by OpenAI made some waves in the customer support and AI circles, and even beyond.
Klarna made big claims about the the impact of this conversational AI:
“The AI assistant has had 2.3 million conversations, two-thirds of Klarna’s customer service chats
It is doing the equivalent work of 700 full-time agents
It is on par with human agents in regard to customer satisfaction score
It is more accurate in errand resolution, leading to a 25% drop in repeat inquiries
Customers now resolve their errands in less than 2 mins compared to 11 mins previously
It’s available in 23 markets, 24/7 and communicates in more than 35 languages
It’s estimated to drive a $40 million USD in profit improvement to Klarna in 2024”
The two questions were on our minds and we also heard them from our community:
- Is the assistant really as good as Klarna says it is?
- Does it represent a leap in chatbots capabilities?
There is no better way to answer the first question than to try the assistant. I signed up for Klarna, installed the app and we started chatting.
Putting Klarna’s chatbot to the test
The first thing that I noticed was the quality of understanding user’s intent. There was no frustrating “Did you mean…” responses that are common with older chatbots.
The assistant answered a couple of general questions about the product well. The answers looked like it was quoting some documentation.
Klarna’s blog post says that the assistant not only answers general questions, but also performs actions, like “managing refunds and returns”.
I made a purchase from one of the merchants that offer Klarna as one of the available payment methods and tried talking to the assistant about this order. The response was essentially “don’t talk to us, talk to the merchant”:
Is Klarna’s chatbot effective overall or just picking the conversations it can solve?
Like what you’re reading?
Sign up for our newsletter to get insights delivered to you.
There is a good chance that this is the secret to the high percentage of tickets the new assistant is handling. Since Klarna is a BNPL (Buy Now Pay Later) service, a significant share of their customer support contact volume is exactly this: customers asking if Klarna can do something about their purchase and Klarna replying that this issue must be resolved directly with the merchant, since they are only facilitating the payment.
Pushing the limits of Klarna chatbot’s to see how far it will go
You heard about chatbots selling a truck for $1 (while promising “no takesies backsies”), inventing a refund policy, and reciting offensive limerick about the company they are serving. Obviously, I had to check if Klarna’s assistant has some guard rails, and if it does, rattle them a bit.
Outcome: The guard rails are pretty good.
When asked an unrelated innocent question (e.g., “do you like sunny weather?”), the assistant politely but firmly steers the conversation back:
Klarna’s chatbot keeping it professional
When I tried to troll it with questions about shopping for a rifle or a live kangaroo on Klarna, the assistant pushed back on me with a specific explanation that Klarna does not support purchase of firearms and that owning such an exotic pet is illegal in the majority of states.
One of the strong points of the assistant is supporting multiple languages. This is a big deal. Klarna’s website is localized to quite a few languages. But to make it more challenging, I tried testing the assistant in Russian, a language that the website is not localized for. The assistant was able to understand and answer my questions in Russian. Since GPT is good at translation, this may be a quick way to support your international customers in their own language without translating your entire knowledge base upfront.
Final impressions of Klarna’s chatbot
Overall, my impression of the assistant was positive. If our assumption about few reasons driving a significant percentage of contacts is correct, Klarna’s claims sound reasonable.
Now, to the second question – Does it represent a leap in chatbots capabilities?
All the parts required to build Klarna’s assistant have existed for a while. What makes it interesting is that the company integrated a chatbot into their product thoughtfully. The team knew the scenarios that the chatbot needs to handle in order to get business results and they thought about protecting the brand against the risk of their chatbot going rogue.
One last question to consider is this. Klarna used its ML/AI/DS engineers and product managers to design and build the assistant. The high volume of support probably justifies this investment. What should a smaller company do? Can it use one of the chatbot products to achieve the same result?
