Beyond Alexa and Towards Her: Psychological Principles for Designing Voice AI Interactions

The demo that Mira and the other two OpenAI employees demonstrated during the keynote was impressive and vastly different from the typical voice interactions we’re used to for three main reasons.¹

1. Real-Time Response Rates

Until now, every interaction we’ve had with a machine has been characterized by delays before and after we gave the command. Siri, Alexa, and pretty much all AI assistants require a significant pause after they’ve been prompted to process the command before outputting the result. 

OpenAI’s new voice model has vastly improved this time response, bringing what we recognize to be real-time interaction capabilities. This means that for the first time, voice interactions feel more immediate—and thus, more natural, tapping into our strong innate preference for fluid conversations.  

2. Enhanced Voice Recognition and Generation

OpenAI’s new voice model employs advanced algorithms that enhance both voice recognition and generation beyond anything that came before. 

On the recognition front, the model uses deep learning techniques to better process and understand speech patterns, improving its ability to handle various accents, dialects, and even speech impediments (to various degrees). Most importantly, for those of us who have tried barking orders at Siri while on the go with a high degree of failure, OpenAI’s model features have enhanced noise cancellation and contextual understanding. This allows for a much better interpretation of spoken languages in noisy environments and conversational contexts. 

Meanwhile, on the speech generation front, GPT-4o manages to sound remarkably human-like, complete with nuanced changes in tone, cadence, and emotion that adapt to the context of a given conversation. What is truly astounding is that the generated speech seems to exhibit natural prosody, along with good use of pauses and emphasis, making it sound very lifelike and quite expressive. 

These improvements in recognition and generation drastically reduce the gap between human and machine communication, leveraging our innate tendency to anthropomorphize and create an interaction that builds more trust.  

3. Multimodal Capabilities 

This model extends beyond just voice—it supports multimodal capabilities, meaning it can understand and respond to text and voice inputs interchangeably. This degree of flexibility enables a more versatile application of the model, opening up a whole world of possibilities for how it can be used. 

An impressive (and applicable!) example of this was demoed during the keynote by a student who shared their screen with ChatGPT and asked (via voice) to guide them through how to solve a math problem. The student could ask questions, dig deeper into items he didn’t understand, as well as scribble down notes to discuss them with the chatbot. 

Having a multimodal approach aligns well with cognitive load theory, allowing users to process information through multiple, distinct sensory channels which could potentially enhance their learning and problem-solving capabilities.  

These three components—real-time response rates, enhanced voice recognition and generation, and multimodal capabilities—are important building blocks in creating a chatbot experience that feels natural and intuitive for users. More specifically, the real-time responses create a sense of immediate engagement, the improved voice recognition and generation allow for more human-like interactions, and the multimodal capabilities provide flexibility in how users can communicate with the model. 

Most of the LLMs released in the last year have some sort of voice interaction capability. But keep in mind, these responses are not in real-time—the LLMs basically just transcribe what the user says before inputting it as text with a pretty long delay. OpenAI is the first organization to design voice interactions with their LLM that seem, at least to the user, to be real-time.
However, as the field matures and others follow suit, many of us here at The Decision Lab believe that experts who want to create interfaces that mimic human interactions need to go beyond this advancement. In particular, they need to take into account a more advanced level of conversational design and the cognitive load that the user experiences.

Conversational Design

Until now, models such as Siri or Alexa incorporated a very basic level of conversational design that mainly focused on accurate speech recognition, clear articulation, and responses that directly address the user’s query. Even with the text-based interface, new LLMs can go beyond that—they can leverage context retention across multiple turns and basic personalization, enhancing the natural conversation flow, especially when migrating from a text interface to a voice interface.  
We believe that the next frontier of these models will have advanced dialogue management abilities (the process of handling the conversational logic of a system) that ensure coherence, relevance, and most importantly, progression toward users’ decision-making goals. Meanwhile, what we’ve seen so far is that after a long series of interactions between a user and a chatbot, the chatbot seems to forget the initial context provided when the chat started, resulting in the user not being guided toward their decision-making goal.

Psychological Principles

At The Decision Lab, we're all about leveraging decision science principles to make AI interactions work better for people in real-world situations. When it comes to designing such effective interactions, understanding key psychological principles isn't just helpful—it's pivotal.

Such concepts can be leveraged to guide how information is presented, manage cognitive load (so users' brains don't feel fried), and create experiences that feel natural and engaging. Let's explore some of the key psychological principles that can be applied.

1. Cognitive Load Management: The model should be deliberate in how information is structured to avoid overwhelming the user. This can be achieved by breaking down complex information into smaller chunks, as well as revealing information gradually, as needed, instead of at all at once.
2. Anchoring and Adjustment: The model should “think ahead” by being mindful of how initial information can disproportionately influence subsequent decisions that the user makes.
3. Framing Effects: The model should consider how the manner in which options are presented can significantly impact the choice that the user makes.
4. Reciprocity: The model should try to replicate human interactions by building rapport through a give-and-take interaction style, while still being efficient in how they’re guiding the user to the final decision-making goal.

These psychological principles can be applied to "resolve" not only the last user query but the actual goal behind the query—or in other words, the underlying intent that prompted the question in the first place. For example, if a user asks about the weather in their city, the LLM should be able to infer that they're planning an outdoor activity and provide relevant suggestions along with the weather information. This should provide a more comprehensive and valuable interaction by anticipating the user's needs and providing more holistic assistance.

Let’s run through a scenario that demonstrates some of the psychological principles we discussed above ​​with a focus on how they apply specifically to voice interactions. 

Imagine that a user asks a chatbot to help them choose a health insurance plan appropriate for their situation. The chatbot’s goal, programmed by the insurance company, is to guide the user toward a decision that balances comprehensive coverage with affordability. Let’s call the chatbot… Bluey! 

Opening with a neutral tone

By starting with an open-ended question in a neutral tone, Bluey allows the user to direct the conversation, giving them a high degree of freedom and respecting their autonomy. This type of approach taps into a user’s self-determination, the psychological principle that explains why people are more engaged when they feel they have a say over the process. In a voice interaction, Bluey's invitation for interruptions acknowledges the fluid nature of spoken dialogue and encourages active listening.

Gathering Information with open-ended questions 

Through this line of questioning, the user is encouraged to reflect on their actual needs without pushing them in any particular direction. Since this is a voice interaction, Bluey's follow-up questions demonstrate active listening and help to create a more natural, conversational flow. This approach is particularly important in audio formats where users can't easily review previously stated information.

In addition, these questions make use of the availability heuristic, where people tend to heavily weigh their judgments toward more recent information. By making any recent medical experiences more salient, Bluey can help the user decide what kind of coverage they would need based on past expenses. Meanwhile, LLMs can extrapolate both types and amounts based on the user’s answer regarding the doctor and practitioner visit frequency.

However, there is also the danger here that the user will perceive the recent expenses as being more risky than the actual needs themselves. One way to prevent this is by helping the user go through their expenses, before asking them whether they believe their treatment is short-term or long-term. (This is exactly what Bluey did when inquiring about how long the user’s physical therapy will last!) This process helps to counteract the recency effect and promotes more balanced decision-making by…

1. Explicitly distinguishing between recurring and temporary health needs, encouraging users to look beyond their recent medical experiences and consider their health patterns in the long run.
2. Prompting users to critically evaluate the relevance of recent health events to their overall healthcare needs, reducing the tendency to overemphasize recent experiences when making insurance decisions.

Presenting options in a balanced way

By giving options regarding how information is presented, Bluey learns the user’s preferences—how they would like to receive the information as well as “process” it. This consideration takes cognitive fit into account, which suggests that performance improves when there's a good match between the way the problem is described and the task itself.

This is where a difference starts to emerge between information that is presented in a text format as opposed to audio. For example, while it’s much easier for a user to compare text next to each other such as in a table, during a verbal conversation, the way in which Bluey describes the options needs to take on a different format.

Let’s assume the user would want the information to be presented as a comparison—what would this look like in action?

After that, Bluey would be able to dig into each dimension of coverage one at a time, breaking the information up to make it more digestible.

This type of conversation demonstrates how Bluey keeps in mind the end goal of the conversation while modulating each prompt based on the user’s preferences and selected interface. It also showcases how chunking (breaking down information into manageable pieces), anchoring (starting with price information that serves as a reference point), and the primacy effect (presenting important information first) can play a role in creating successful voice-only interactions with LLMs. 

As voice interfaces for LLMs continue to evolve, it's crucial that designers, developers, and product managers focus not only on the cutting-edge technological advancements that help models resemble humans speaking to other humans but also on the psychological principles that are key to effective human-computer interaction. This way, we can go beyond what voice assistants like Alexa currently offer — but also, feel closer to them.

Here at The Decision Lab, we believe that by incorporating these principles, experts can create voice interfaces that are not just technologically impressive, but also intuitive, engaging, and truly helpful in guiding users through complex decision-making processes. Get in touch with us today to see how we can help you achieve just this.