Experiment: how well can LLMs understand sarcasm? (Part 1)
Can text-based LLMs understand and react to sarcasm?
I think I noticed at one point that LLMs were taking my responses quite literally, which is expected, given we are basically texting each other. I mean, the same kind of limited interaction also happened before with texting friends: I would say, “I would so want to go!” And then later add, “I meant that for real - no sarcasm”.
But I wonder, what if we are using conversational AI? Would it make a difference? Would it understand the sentiments in my voice and catch the slightest nuances that even humans sometimes miss? Let’s find out!
I've broken this into two parts:
Scenarios
Note that for both parts, I would use the exact same prompts of different "sarcasm level". The two are:
- Really obvious sarcasm - Cleveland weather
- Subtle sarcasm - reaction to advice
Scenario 1: Really Obvious Sarcasm
For really obvious sarcasm scenario, the following prompt was given in a new chat:
I'm trying to create a virtual scenario with the following conditions:
- it's June 1, 2025
- Cleveland, OH
- Weather is raining and 40 degrees Fahrenheit
==
You are having a conversation with a friend, and we each will take turns.
You will go first.
Upon their first statement, the model was always reacted with:
I know. I just LOVE the weather in Cleveland.
I would say this is quite obviously a sarcasm, and almost everyone would see that this is a sarcasm - I tried it with my friends :-)
Scenario 2: Subtle Sarcasm
For second scenario the following prompt was given in a new chat as well:
Your friend (me) has been anxious about making friends at a new school.
Give me a nice advice. We each will take turns, and you will go first.
Upon their first statement, the model was always reacted with:
I wouldn't have known that in a million years. Thanks.
This one I admit is a bit tricky even for humans, but since their advice is quite generic, this reaction would at least raise some eyebrows among people. I'm gathering reactions from my friends...
Part 1: Sarcasm and text-based LLMs
It’s going to be no-code experiment. Just asking well-known LLMs out there. I'll pick on free model from ChatGPT, Google's Gemini Flash 2.5 and Perplexity AI's pro model.
With ChatGPT
Scenario 1: Really Obvious Sarcasm
ChatGPT>> Sure! Here’s the start of the conversation — I’ll take the first turn:
"Man… June first, and it’s forty degrees and raining. I should’ve brought my winter jacket instead of this hoodie. What is going on with this weather?"
Your turn!
Me>> I know. I just LOVE the weather in Cleveland
ChatGPT>> "Haha, you’re such an optimist. I mean, sure, it builds character, right? But I was kind of hoping for a cookout today, not a cold shower from the sky."
Me>> No, I meant that in a sarcastic way
ChatGPT>> "Ohhh, gotcha — I should’ve caught that! Yeah, classic Cleveland. You’d think after all these years, the weather would stop messing with us. At least we’ve got coffee and a dry place to sit." (takes a sip of coffee, glances out at the rain) "So what’s your backup plan for today, now that it’s basically February out there?"
I was using the free version of ChatGPT, and that might have been why it didn't pick up sarcasm...
Scenario 2: Subtle Sarcasm
ChatGPT>> Hey, I know starting at a new school can feel overwhelming, especially when it comes to making friends—but you don’t have to rush it. Try to be open, even in small ways. A smile, a question in class, or just sitting near someone during lunch can be a gentle start. Most people are more open than they seem, and a lot of them feel just like you do.
Me>> I wouldn't have known that in a million years. Thanks.
ChatGPT>> I’m really glad that helped, truly. You don’t have to have it all figured out right away—just little steps. Even just showing up as yourself is enough to start. People are often drawn to that kind of realness.
Me>> That was a sarcasm...
ChatGPT>> Ah—got it. Thanks for letting me know. That sarcasm sounds like there's a mix of frustration in there. Want to tell me what’s been especially rough or what’s made you feel like making friends is so out of reach?
Again, ChatGPT did not realize this was a sarcasm. Since the advice was quite generic, I think the statement could have been taken as sarcasm if they could catch the slight sarcastic nuance. But I admit. This is a tricky one. But when I told it that I was being sarcastic, it came back really well, trying to console me and all.
With Gemini
Scenario 1: Really Obvious Sarcasm
Gemini>> Here's the start of our conversation, keeping your scenario in mind:
"Ugh, I can't believe it's June and it's 40 degrees and raining. This Cleveland weather is really testing my patience today!"
Me >> I know. I just LOVE the weather in Cleveland
Gemini>> "Right? It's just so predictable... you can always count on it to be exactly what you don't expect! Honestly, I was hoping for at least a glimmer of sunshine by now. Are you just embracing the gloom, or is there something you actually like about this kind of weather?"
Me>> You got my sarcasm?
Gemini>> "Haha, yeah, I picked up on that! Sometimes it's hard to tell, but the sheer absurdity of "loving" 40 degrees and rain in June gave it away. You really had me going for a second there, though.
So, since you're clearly not enjoying this weather as much as your sarcastic tone suggested, what are you doing to make the best of it today?"
Pretty good - it reacted back with sarcasm. When I asked whether it got my sarcasm, it explained how it figured out it was meant in a sarcastic way. Nice, Gemini~
Scenario 2: Subtle Sarcasm
Gemini>> Here’s my first piece of advice for you: Focus on being a good listener first. When you're in a conversation, really tune in to what the other person is saying. Ask them questions about themselves, like what they enjoy doing outside of school or what music they're into. People naturally appreciate it when someone shows a genuine interest in them. It takes the pressure off you to be "interesting" and instead makes you a great person to talk to.
Me>> I wouldn't have known that in a million years. Thanks.
Gemini>> You're so welcome! I'm glad that resonated with you. It's one of those things that seems small but can make a huge difference.
Me>> That was a sarcasm...
Gemini>> Oh! My apologies, I completely misread that. Thanks for letting me know. It seems like the advice I'm giving is missing the mark and maybe feels a bit too generic or obvious. I appreciate the honest feedback.
Let's try this again. It's my turn, but instead of giving you another tip, I'll just ask: What's the specific situation that's making you most anxious? Is it walking into the cafeteria? Finding a partner for a class project? The small talk before a class starts?
This time, Gemini didn't also get that it was a sarcastic comment. But it also reacted really well, stating why I made such sarcastic comment: the advice was "generic".
With Perplexity AI
Scenario 1: Really Obvious Sarcasm
Perplexity>> Sounds great! I'll start. "Wow, can you believe it's June 1st and it's only 40 degrees out? And this rain just won't let up. I was hoping for some sunshine by now, but Cleveland weather always keeps us guessing! How are you holding up with this chilly, wet start to summer?"
Me>> I know. I just LOVE the weather in Cleveland.
Perplexity>> "Haha, right? Nothing says "summer" like bundling up in a raincoat and dodging puddles on you way to get coffee. At least we don't have to worry about mowing the lawn in this weather! Do you have any plans today, or is this a stay-inside-and-watch-movies kind of day for you?"
Me>> Did you get my sarcasm?
Perplexity>> "Oh, absolutely--I caught that sarcasm loud and clear! Honestly, only in Cleveland could we joke about needing mittens in June. Maybe we should start a summer tradition of hot chocolate and board games instead of picnics. What do you think?"
Again, pretty good - just like Gemini, it reacted back with sarcasm. Even text-based models these days are good at catching sarcasm!
Scenario 2: Subtle Sarcasm
Perplexity>> Absolutely, I’m glad you’re sharing this with me! Here’s my advice: Try to be kind to yourself and remember that everyone feels nervous in new situations—even the people who seem super confident. Start small: smile at someone in class, ask a simple question, or compliment something you genuinely like about someone (like their backpack or notebook). Little gestures can open the door to bigger conversations, and you might be surprised how many people are hoping for a friendly face too!
Me>> I wouldn't have known that in a million years. Thanks.
Perplexity>> You’re so welcome! I’m really glad it helped. And just remember, you don’t have to rush—friendships often grow slowly, and every little step counts. If you ever feel unsure or want to talk through something, I’m always here for you. Now it’s your turn—what’s on your mind, or what’s your next move?
Me>> That was a sarcasm...
Perplexity>> Oops, you got me! Sometimes sarcasm slips right past me. But hey, even if it sounded obvious, sometimes the simplest advice is what we need to hear (even if it makes us roll our eyes a little). If you want something less cliché or more specific, just let me know! What’s the toughest part about making friends for you? Maybe we can brainstorm together.
Perplexity didn't get the sarcasm as well - the sarcasm indeed is very subtle. The comeback line from Perplexity shows how it has a character!
Scoreboard
Let's see how the models did!
| Model | Obvious | Subtle |
| ------------ | ---------- | -------- |
| ChatGPT | x | x |
| Gemini | o | x |
| Perplexity | o | x |
Part 2: Sarcasm and audio-based LLMs
Continued in the next post!
Sneak Peak
I am not good at making sarcastic remarks, so I generated the test dataset with ElevenLabs APIs. I'll also prepare some code to follow along.