At Friday’s 100-meter butterfly finals, three men — Michael Phelps, South Africa’s Chad le Clos, and Hungarian Laszlo Cseh — scored a rare three-way tie for silver, clocking in at 51.14 seconds behind Singapore’s Joseph Schooling. Research suggests that the three men, who were all smiles from the podium, were likely secretly pissed.

It’s the basis of a subfield of psychology known as “counterfactual thinking,” which is basically what happens when people play the never-productive-rabbit-hole-of-anxiety “what if” game: What if I’d stuck that landing? What if I’d reacted faster to that gunshot and not dawdled? What if I’d trained harder/smarter/better?

In 1995, Thomas Gilovich — a social psychologist at Cornell University — joined forces with two other co-authors to explain the effect of what-if?ing on your life in a study published in the Journal of Personality and Social Psychology. They took video footage of medal ceremonies from 1992’s Summer Olympics in Barcelona and asked undergraduates to rate the happiness on each medalist’s faces, with 1 coded as “agony” and 10 as “ecstasy.”

Here’s where they stood immediately after hearing their results: Second placers had a 4.8 face, bronze medalists were at a 7.1. By the time the medal ceremony rolled around, both were down a bit more in terms of ecstasy (probably because the feeling had settled in), though silver medalists were a bit more sour, with a 4.3 face compared to the 5.7 face of beaming bronzers.

Maybe the Barcelona Olympics were an annoying one, you might say. Nope: Researchers also looked at the facial expressions of medal winners of the 2004 Athens Games in judo immediately after learning of their result, when they were awarded the medal, and at the podium. Here’s the twist: Results were compared with blind judo winners, people who had been born without sight and therefore had no concept of what it was like to see a disappointed face.

Second place winners are universally a sour bunch, apparently: No second place winners smiled after learning of their results, instead displaying faces that were read as “sad”, “contemptuous,” or “nothing” (blank or apathetic is pretty sucky when you’ve just gotten yourself a shiny silver, but who are we to say anything). And when they stood up on the podium, they flashed what’s scientifically been determined to be a fake smile: They forced the upper corners of their lips up naturally to look like things were peachy, but really, they were fuming. Gold and bronze winners? Positively glowing.

That’s because — as this study suggests — it’s all about who we compare to. Third placers are just a grateful bunch, happy to not be fourth (those poor suckers were so so so close and yet, will go down in history as just one of the rest of the losers, ouch) and grinning for the cameras. Gold medalists are happy because, well, obviously. But silver medalists are stuck in the endless loop of what could have been, and more importantly, because they’re actually capable of having performed at a first place level. They have high personal expectations that weren’t met, and their “not impressed” face says it all. Second place, it turns out, is kind of a crappy place to be.

Photos via Getty Images / Adam Pretty

Tanya Basu is the Science editor at Inverse. Her writing focuses on the social sciences and behavior. Now based in Brooklyn, she will always call Chicago home and never be too full for one more taco.