Thinking Fast: When Intuition Isn’t All Bad, According to its Biggest Critic

We rely on our gut for everything from choosing options at a restaurant to making high-stakes career choices. Behavioral science, however, has given intuition a bad rap. Over the years, the literature on cognitive biases and heuristics has categorized the numerous ways in which our impulses lead us to commit frequent, predictable mistakes. 

Behavioral science is all about unpacking human decisions and understanding what is driving our choices beneath the surface. Since Daniel Kahneman’s early work on cognitive errors, we have celebrated deliberate choices as a panacea for the problems that stem from our imperfect decision-making faculties. The classic directives are to stifle our intuition and assess a choice in the context of objective outside information: bring in base rates, check your priors, consider underlying motivations, and then make a decision. 

However, we can’t always stop and think about our choices before they happen. Many of our biases save us time and energy, so that we can handle a world that is constantly in flux. 

The word choice evokes scenarios where one has time to consider several options, perform some sort of calculus, and then emerge with a preference. But what about the decisions we make that don’t really feel like choices at all? When we follow our instincts, it can feel like we’re being pulled towards a certain option, and the process of decision-making becomes one of traffic control. Should we follow this urge or rein it in? 

Titans talking past each other

Kahneman, Thaler, Sunstein, and other big names in behavioral science have shown us ways that our intuition sends us headfirst into suboptimal scenarios. There are other thinkers, however, such as Gary Klein, who see things from a different perspective. Klein has respect for experts that make snap judgments in difficult situations, while Kahneman is characterized as skeptical of their expertise. This has led to claims that behavioral science is stuck on the problem of whether we can really trust ourselves. Somewhat recently, however, Kahneman and Klein took some time to figure out where their disagreement actually comes from. The biggest finding? They don’t disagree all that much. The differences in opinion are mostly a matter of perspective. The hard, underlying questions are solved when we look at the environment in which our intuition is going to work. 

When these two titans of behavioral science sat down to work out their differences, they realized they were looking at the same problem from different directions. Klein’s focus is on expert decision-making: what he’s trying to understand is how certain individuals use intuition successfully. He seeks to identify what qualities the experts have that the rest of us don’t, that allow them to often make the right call in situations that would leave the rest of us paralyzed. 

Kahneman, by contrast, cares about how the pros stack up to algorithms. If a human expert and a decision-making algorithm are presented with the same type of problem over and over again, algorithms will always win out in the end; after all, they’re optimized to make the best decision possible over the long run. Statistical thinking always wins in the end but that’s a different problem than the one Klein is trying to answer. 

Where intuition is valuable

As we’ve seen, it would be reductive to say that we shouldn’t use our intuition when making decisions. We have to do so, all the time. This still leaves us with the original problem of figuring out when we can trust our gut versus recognizing situations when we can’t go it alone. The truth is, intuitive thinking is necessary and often effective. The question of when and how to use our intuitions comes down to the nature of the decision-making environment. 

We must look out for two key criteria when assessing an expert’s ability to make gut choices: first, that their decision-making environment is one of high validity, and second, that they have adequate opportunities to practice their judgment. 

An environment of high validity is one where there are reliable cues that hint at the right answer. When we learn how to categorize objects or events, the repeated, consistent observation of certain features is essential to our ability to learn. Imagine, for instance, a young child learning what a “face” is. Every face has two eyes, a nose, and a mouth. In situations with high validity, reliable cues like these will be present, even if we’re not entirely aware of them on a conscious level.

Intuitive judgment doesn’t depend on consciously processing each cue. Rather, we use our procedural memory—the same cognitive system that helps us walk or ride a bike—to learn patterns in a given familiar situation. It may be possible in hindsight to understand how each muscle works in the act of walking, and the same goes for figuring out what prompted the right answer in intuitive judgments, but that isn’t what our body is doing when we make split-second decisions. In the moment, intuitive judgments are characterized by decisions that happen faster than one can rationalize. 

Second, the opportunity to practice one’s judgment comes from repeated cases where one can try out one’s luck at making snap judgments. Even highly predictable scenarios are hard to recognize the first time we encounter them. Cutlery should always be in roughly the same spot, no matter whose kitchen you’re in, but somehow I always have to open 3 or 4 drawers before I get it right whenever I’m in a new space. 

The difference between a firefighter (Klein’s favorite example of an expert decision-maker) and a stockbroker (one of the many “experts” Kahneman likes to skewer) is the validity of their environment. Both get to test out their intuition over and over again, but only the firefighter has reliable cues that can at least be distinguished in hindsight, such as temperature and the sound of the fire. The problem is different for the financial “experts,” as well as political and business leaders, who are faced with widely different situations each time they have to make an impulse decision. Since the stock market does not repeat itself predictably, the broker does not get the practice that is necessary to make reliable snap judgments. 

Conclusion

Breaking down the decision environment into these key criteria builds essential tools for applied behavioral scientists. By ensuring that decision-makers are equipped with the right context to make good choices, we can reliably improve outcomes for people according to their own goals. 

There are cases where we are clearly served well by our instincts. Athletes and emergency workers rely on fast-paced decision-making in order to do their jobs. Other times, our instincts give us unreliable advice. Behavioral science helps us understand when intuition can be improved by algorithms and other simple rule-based systems, and what our cognitive systems need in order to thrive. The field is at its best when we carefully pick apart the places where people need help to navigate complexity, and respect the brilliance of the human mind when it is doing just fine all on its own. 

Why Most Organizational Change Programs Fail

Imagine you work in the IT department at Super Productivity Corporation.1 You and your colleagues wish to adopt an automated process for IT help, based on an electronic help desk that works through an app on the employee’s computer.

You have very good reasons to be excited about the process: it is much faster than the existing phone-in process and will save time for everybody. After all, efficiency is a core value at the Super Productivity Corporation. For this reason, you don’t anticipate any problems—but, just to be safe, you still take the time to inform all employees about this change through emails, luncheons, and workshop demonstrations of how it’s used and what the benefits are. 

Launch day finally arrives. You and the others in the IT department arrive and sit back, ready to enjoy what is sure to be a nice, relaxing day at work. With the new system in place, the constant ringing of phones will be a thing of the past! But your hopes for a perfect day are shattered when a colleague comes knocking at the door to relay their technical issues, and in a somewhat agitated manner… 

The human side of change

In a business environment (as anywhere else), the successful implementation of an organizational change is dependent on the willingness of affected people to comply, and on their ability to easily do so. Studies show that the majority of major corporate change programs fail, often because human resource barriers such as lack of employee involvement and motivation are overlooked.2 

As human beings, it is in our nature to contemplate, question, and even resist change. If we are not able to consciously process the reasons for the specific change or accept that there is a need for things to be done in this new fashion, we often succumb to the status quo bias (our emotional preference for maintaining the current state of affairs) and consequently find ourselves resisting change—even when it might be helpful to us in the long run.

The human element of change is often overlooked, especially when the changes we want to see are themselves rather easy to implement. Think of moving to a new house: while the process of packing & unpacking is a hassle, the sofa, TV, and washing machine will operate and serve the same purposes in the new environment once offloaded from the van, no negotiation or pep talk necessary. But for the family members who are changing neighborhoods, schools, and routines, the process is much more difficult and involves cognitive and emotional elements. There is ambiguity, a sense of loss, fear of the unknown… 

Behavioral barriers to organizational change

The same applies to any change within a company or organization that affects employees, from the introduction of a GDPR protocol or pandemic-induced processes, to organizational restructuring following a merger or acquisition. Interestingly, it also applies to smaller-scale events such as the adoption of new, fairly simple technologies or tools.

Research shows that 55% of our workday behaviors are habitual.3 In other words, half of what we do is to respond automatically to cues in a way that has yielded some kind of reward for us in the past. This behavior is desirable most of the time as it helps us navigate the world around us, making life easier and work more productive.

There are instances, however, when routines inhibit organizational change and innovation—which is exactly what happened at Super Productivity Corporation. When the employees ran into an IT problem, out of habit, they picked up the phone and called the IT help desk. But following the launch of the new app, that habit did not produce the desired result, and they started feeling stressed and frustrated. 

When these negative feelings cascade throughout the organization, they can manifest as decreases in productivity and damage to employee morale. Organizations facing such a situation tend to find themselves spending substantial resources to temporarily recreate the old process and appease employees, while simultaneously trying to reintroduce the very simple (yet widely unpopular by now) new system. 

Going beyond habits, in our imaginary scenario, we can also think of a few other possible reasons for the failed transition to the helpdesk app by borrowing insights from further research.4 For instance, there could be an issue of information overload, which can lead to decision fatigue and reinforce employees’ tendency to cave into impulses and preformed habits (i.e. picking up the phone to report an IT problem).

In addition, there is often an issue of confirmation bias among the change architects. In the case of Super Productivity Corporation, the IT team, in all their excitement, may have ignored any contradictory or negative feedback from their colleagues, or brushed off employee concerns about implementing this change because they were so focused on the benefits it would bring.

Is there a better way? 

Fortunately, insights from behavioral science5 provide a roadmap for facilitating change. Research points towards three well-established paths for new habit formation, which are not perceived as costly by the people who undergo the change.

  1. Make the alternative behavior easy to adopt. Possible ways to do this include adjusting the levels of sludge (obstacles that cost effort and time and accentuate the intention-action gap) and salience (prominence or noticeability) of the available options. In effect, this should decrease the value of the habitual response, and as the new behavior becomes easy to adopt, the change itself (as envisioned by its architects) becomes plausible.

    A popular example is that of Google helping their employees develop healthier eating habits.6 To nudge employees towards healthier snack options, Google placed sugary sodas and chocolates behind an opaque sheet of glass (essentially out of sight) and put water, fruit, and nuts behind clear glass, at eye level. Over the seven-week test period, employees in the New York office consumed 3.1 million fewer calories from M&Ms, as well as 47% more water!
  2. Make it personal by connecting the change to individual employees in relevant and meaningful ways. Realistically, it is not always possible to make the new behavior easy, especially as technological innovation sometimes requires the adoption of protocols that are more challenging to follow (at least at the outset). In such cases, personalization helps. For instance, practices of normative feedback (providing performance scores relative to others) can increase people’s willingness to break away from habitual patterns in their quest to rank highly. This strategy relies on leveraging social norms which can help with behavior change.
  3. Make it about money. While a financial reward (or penalty) may not capture the whole story, it is still a powerful motivator. In the quest to change somebody’s habits, compensating them for their effort to engage in the new behavior can help offset their personal cost and be a powerful path to change. The inverse effect also works—that is, disincentives such as financial penalties applied to undesirable habits.  

Conclusion

To sum up, in line with the experience of Super Productivity Corporation, many companies fall short of implementing changes successfully. And if such small changes as switching to an IT helpdesk app can cause widespread frustration, imagine what happens during extensive restructurings and digital transformations. We can attest that communicating the benefits of the change and training employees are necessary elements of the change management process, but this is still not enough to ensure uptake.

Instead, the success of any change management process depends on how individual people will choose to behave, their willingness to change their habits, and whether or not they have resources that will support them in doing so. This is where behavioral insights come to complement business strategy and change management tools to help the organizational engine—that is, the people who work there—succeed in the desired transition. 

Can It Ever Be “Too Late” For a Career Breakthrough? Research Says No

A few months ago, in the heat of our hot vax, hot girl summer, I turned 30. Up until that point, it had hardly been the year I’d been dreaming of to ring in my next decade: I spent winter and spring cooped up alone in my apartment, trying out new hobbies to fill the time where I would normally be out and about. But just in time for summer, the stars aligned: social life opened up, and travel was possible again as my 30th birthday approached.

I started to consider: What comes next? When the summer slides into fall and we migrate indoors once again, what will become my focus? More existentially, will I have bid all my best years adieu? Or are they just around the corner? 

Looking for advice, wisdom, and inspiration grounded in science, I did a lot of reading. Conventional wisdom tells us to sharpen our focus in four realms in our lives: relationships, finances, health, and our careers. In terms of relationships, there is ample research on friendships, dating, and long-term romantic partners. There’s also extensive literature on how to establish healthy habits, and financial advice to help us save for later life.

But what about our careers? Is the window for a career breakthrough gone by the time our twenties run out? In other words, if I spent my twenties diving into different interests and concerns, but had not yet found “the one” (in many senses other than my career), had I missed the boat? I needed only to open my iPhone to be reminded of Whitney Wolfe Herd of Bumble, Daniel Ek of Spotify, or Evan Spiegel of Snapchat, who all had their big break before they turned 30.

A research paper published in September continues in a line of research on the “hot-streak” phenomenon. It turns out, reaching career greatness might not be so age-dependent as we might think, but instead dependent on a particular style of work.

Exploration vs. exploitation

A 2021 paper by Lu Liu and colleagues1 looked at the career trajectories of thousands of famous painters, film directors, and scientists. They found a unique pattern among those with mega fame. Across the three fields, those who became superstars had a period of exploration before their hot streak. After the hot streak of success began, the superstar became more focused on their pursuits, in a process the researchers called “exploitation”: milking an effective style/approach/methodology for all it was worth.

Neither exploration nor exploitation alone predicted a hot streak. It was the pattern of exploration followed by exploitation that was the winning combination—the key to the breakaway success of figures like Jackson Pollock in his “drip period,” or Peter Jackson in the era of The Lord of the Rings, among others. 

This finding is a blessing for millennials everywhere, who have taken to heart the wisdom that one should try everything once in their 20s—and who have not yet found their calling. However, a few questions remain: How do we know to stop exploring and start exploiting? Furthermore, how do we know when the period of exploration has turned into good old procrastination? And what role does productivity play?

If exploration plus exploitation is the secret sauce, when do I start either?

Received wisdom tells us that peak performance comes in our 30s and lasts until our 40s—when we have built up experience in our fields but still have “the energy and enthusiasm to sustain high productivity.”11 Conventional thought also implies that when we have reached our 30s, it’s time to settle down and get serious. In other words, we should stop exploring and start exploiting. 

However, this intuition does not match up with the evidence. When looking at people who have excelled in film, writing, music, and academia, the timing of superstardom appears to be random. A scientist’s highest-impact publication (the metric for scientific excellence) can happen at any time in their career, not necessarily when they were young and full of energy.10

Similarly, for creative careers in film, books, and music, luck is found to be an important driver of career success. Researchers have dubbed this the “random-impact rule.”4 For scientists, this principle applies across various disciplines and career lengths, regardless of the decade an individual came to fame or if they published in teams or alone10 (see note 3 for more on the role of teams).  

What is the catch? Productivity still had a defining effect on the timing of an individual’s highest-impact work.10 Looking at the achievements of scientists, researchers found that, on average, impact increased over time, and that this was a result of increasing productivity, not growing ability or excellence.

It appears, then, that if we can maintain high productivity and remain open to new opportunities, our big break might be just around the corner. 

When the going is good, get going?

When you are in a flow, and you have found a sense of career confidence and success, what do you do next? Must we produce as much as possible when the going is good?

Researchers have found that, in general, artists and film directors experience hot streaks lasting for around five years; for scientists, such bursts typically last four years.11 The same researchers also stressed that hot streaks were not characterized by increased productivity; individuals did not produce more during these periods, but what they did produce was more impactful than their previous or future work.

What’s more, it is not just the first big break that happens at random: A second and even third big break also happen randomly.10 That means, even at mid-life (assuming 80 is the average lifespan), there is still potential for another big break to happen virtually at any time (see note 1). 

In conclusion, when the going is good, try to maintain your flow, but do not stress about pushing out as much content as possible. It also tells us that an ebb and flow of excellence can be expected: We cannot always be on top. 

What is the advice to nearly 30-year-olds?

Is it all over at 30? The simple answer is, no, not for those who persevere. Dame Zaha Hadid, the renowned architect, produced her internationally recognized graduate thesis, “Malevich’s Tektonik,” at age 27—but it was not until 1997, at 47 years old, that Hadid received her first big commission to design the Cincinnati Art Museum.3 By the 90s, the technology required to bring her groundbreaking visions to life was finally available, and from there, she became the internationally famous architect we know today. 

No freshly minted 30-year-old has gotten there unscathed: As Scott Barry Kaufmann (2015) writes, “doing things differently sometimes involves doing things badly or wrong.” This is a message anyone in their early thirties (hopefully) learned through trial and error. What we can learn from the careers of outstanding artists, scientists, and filmmakers is that having spent one’s 30s exploring may not be a waste of time. Looking at figures like Jackson Pollock (see note 2) and Hadid, we see that even into our 30s, exploration can yield astounding success and superstardom. Furthermore, only recently have longevity researchers like Laura Carstensen of Stanford started to come out to say that full-time careers should/could start in one’s 40s.9

It seems, therefore, that there is no easy answer as to when one should stop exploring. However, there is a clear message that success can happen at any time in a person’s career, and the only true way to prevent a hot streak is to stop working altogether.11

What can we do for greater success, today?

If you’re not (yet) a famous painter, director, or scientist, fear not. This compelling research shows us that dedicated exploration, collaboration with differently skilled people, and consistent productivity are the ingredients for success. This research also does not discount the role of luck, however, in our careers and personal success (note 4). 

Perhaps you will not find this research to be particularly groundbreaking—of course, it has a lot to do with random chance. However, for artists, it highlights a thought that often occurs in retrospect. One article commenter I found explained this dilemma perfectly: “Many artists including myself struggle with never-ending exploration and producing a large body of work ending ‘in the drawer.’ While the exploitation step is obvious when to initiate and how long to sustain it is less so.”8

This message might therefore come at the right moment for you, fellow reader: anytime is a good time for a breakthrough. 

Notes: 

  1. Lui et al. (2021) also call out the career of Nobel Laureate and chemist John Fenn. Fenn won the Nobel Prize in chemistry at age 85, for work he started only after he was semi-retired, in his late 60s.
  2. In a similar vein, Mozart died at 35, having produced his most famous works in the couple of years leading up to his death. Jackson Pollock was 35 at the time of his drip period, and continued until he was 39. Peter Jackson was 40 when he directed the Lord of the Rings trilogy. 
  3. The impact of leveraging a team and our connections, both strong and weak ties, is measurable. During a hot streak, scientists were found to work with large teams, while in the lead up to their stardom they had experimented with new concepts with smaller teams.2 Furthermore, network scientist Barabási1 wrote: “How to reach the top of the corporate ladder? … replace the idea of a corporate ladder with a social bridge—we never work in isolation. We need to find the hubs that can accelerate our trajectories and reach out to them… no matter what our discipline is.”
  4. This article does not take the role of luck into account. However, researchers have commented on its importance, and it was written about in Daniel Kahnemann’s 2011 best-seller Thinking, Fast and Slow: “Luck plays a large role in every story of success; it is almost always easy to identify a small change in the story that would have turned a remarkable achievement into a mediocre outcome.”