Digital banner depicting the integration of AI and personalized design, featuring abstract elements like diary pages, circuits, and digital patterns in a modern, sophisticated style.

Unlocking personalized design with diary studies

read time - icon

0 min read

Jan 05, 2024

Backed by advances in AI and machine learning, personalization and user-centric design are becoming embedded in product directions across the board. Rather than tailoring products to specific demographics, companies have started tailoring products to specific individuals, addressing their unique preferences and needs.

Personalization is poised to unlock new levels of ease, convenience, and relevance for users – that is, if it’s done right. We only have to think of our encounters with targeted advertising to see how quickly personalization can shift to being insensitive, repetitive, or downright creepy – like that time an eerily accurate Facebook ad mirrored a recent online conversation I had with a friend.

More often than not, the success or failure of personalization pivots on how well the product adapts to its user – whether that be their emotions, attitudes, or goals. In this context, research methods providing static, one-off insights fall short of capturing the entire picture. Enter diary studies: the unsung hero that invites us into users’ dynamic lives, thus allowing personalization to be just that: personal.

Diary studies: Towards a dynamic understanding of user behavior

Traditional techniques – including interviews, user testing sessions, and focus groups – remain valuable, but only provide snapshots of user preferences at a specific moment. While some experimental methods like MaxDiff and Discrete Choice Experiments let us glimpse into how users respond differently to different inputs, these methods don’t tell the full story. How would this user respond if they were hungry? When they’ve been stuck in meetings all afternoon? Or when they’ve just caught up with a good friend?

Meanwhile, diary studies are a longitudinal research method where participants systematically record their thoughts, behaviors, experiences, or other relevant information over a set period of time. In the world of product development, diary studies let us see how users interact with and respond to a product (or even a specific feature) over time and across natural changes in their environment. 

Diary studies also allow us to cross the ditch from anticipated behavior to real-world actions. Instead of asking participants how they think they would respond to a certain recommendation, we can observe what really happens (at least, according to participants) and then gather multiple data points to see if this response changes over time. Since diary studies generally involve some element of self-reporting (although other methods of data collection are equally possible, but more on that later), they also capture a unique user perspective – which matters most when it comes to personalization. 

Behavioral Science, Democratized

We make 35,000 decisions each day, often in environments that aren’t conducive to making sound choices. 

At TDL, we work with organizations in the public and private sectors—from new startups, to governments, to established players like the Gates Foundation—to debias decision-making and create better outcomes for everyone.

More about our services

Harnessing new technologies: Diary Studies 101

While researchers agree that there is no gold standard for how to design a diary study, there is consensus that these studies are highly flexible to the question at hand. 

Diary studies can be organized and adapted into a variety of different experiments. For example, you can randomize your participants into different treatment groups and you have yourself a controlled study. Or, you can pair survey-style check-ins with interviews or observations for a mixed-methods framework that balances breadth and depth. 

Advances in tracking and AI technologies present new opportunities for diary studies. 

  • Get better measurements: Data from health tracking technologies can be used to gather measurements that supplement self-reported data. 
  • Chat with your participants: AI chat models can be programmed into your survey (for example, using OpenAI’s Chat Completions API) to pose responsive follow-up questions to participants, creating a more dynamic study experience. 
  • Create realistic recommendations: Likewise, AI text generation models can be used to transform preliminary, basic recommendations (“exercise for 15 minutes”) into recommendations that are closer to what a future product would generate (“We see you’ve got a lot on today, but a little goes a long way. How about squeezing in a 15-minute workout to lift your mood?”).

Diary studies in action: Examining exercise recommendations 

In many parts of life, what you say you’ll do matters a lot less than what you actually do. When it comes to comparing these intentions against real actions, diary studies are an ideal tool. Moreover, diary studies are adept at identifying opportunities to instill habits, offering a better understanding of the key moments where users can be nudged toward more consistent engagement. 

The persistence of the intention-action gap and the power of habits are both never more relevant than in the context of exercise. So, when faced with the task of testing personalized, evidence-based exercise recommendations with prospective users, we chose to use a diary study. Our study followed 30 active North American adults, with half of the group receiving personalized exercise recommendations, allowing us to measure the impact on their exercise behavior. 

For two weeks, we checked in with participants three times a day according to their usual exercise schedule: 

1) Early check-in: Understand what exercise they intended to do that day.

2) Pre-exercise check-in: Ask how they were feeling once the day got going and provide an exercise recommendation based on their response.

3) End-of-day check-in: Request a report on what exercise they did that day.

With this setup, we were able to answer questions like:

  • What were the key personal or contextual factors that made users more receptive to our recommendations? Our recommendations were particularly effective at changing users’ minds when they didn’t intend to exercise at all. This was especially true if users were feeling self-conscious or sore.
  • What recommendations were users most receptive to? Context is everything when it comes to answering this question. When users felt capable of taking on a new challenge, a recommendation to do a technically difficult exercise got more users sweating. Meanwhile, when users felt stressed, a recommendation to do a high-intensity workout was more likely to get them moving.
  • What contextual factors called for a more sensitive approach? People who had been inactive for a long time on a given day proved particularly difficult to budge. This group also tended to be stressed, self-conscious, and sore, suggesting that inactivity was the symptom of something bigger going on. Since recommendations to get active were not effective, our results suggest that this is the perfect opportunity for a self-compassionate approach that builds rapport with the user, such as reminding users that it’s okay to take a rest day or suggesting that they postpone their planned workout to another time.

Based on insights like these, the diary study provided recommendations targeted at making a product more responsive to changing moods, pressures, and sensitivities. Perhaps even more importantly, the results highlighted the contextual factors that a personalization system must know about (and therefore measure) to respond helpfully and appropriately to its target demographic. 

Getting a diary study right

While insightful and worthwhile, diary studies do require a certain amount of respect. Not only are you gathering potentially personal information, but you are also reliant on your participants remaining engaged. In our experience, the best insights are reached when diary studies are rewarding for both participants and researchers alike. With that in mind, here are a few must-haves for a well-run diary study:

Pay well

This is especially important if you are asking users to report on a sensitive topic. Previous research with our partner, Tremendous, shows that sensitive topics need to be compensated at a higher rate to engage participants, in particular for longer studies. Keep people engaged by spacing payments throughout the study, and show good faith by sending the first payment early on. 

Connect. 

Diary studies require users to stay engaged over longer periods of time and often call for closer support for participants. To build a connection with participants, try unveiling the human behind the Google Form by recording an onboarding video or offering a quick one-on-one catch-up with participants to answer any questions.

Remind people! 

Let’s face it – filling out a 5-minute check-in isn’t at the top of everyone’s mind. Support your participants by reminding them to fulfill their diary study tasks. Even better, allow participants to personalize their reminder schedule and method. Keeping tasks short will encourage people to respond to reminders straight away, rather than them putting them off, or even worse, completely forgetting them. 

Make the study relevant. 

Don’t underestimate the importance of reciprocity. Demonstrate that you are committed to making the experience positive and useful for participants in whatever way you can. As a starting point, this study suggests giving the gift of knowledge by providing personalized feedback reports to participants. 

Following these recommendations, our study on exercise recommendations achieved a 100% retention rate and glowing participant feedback. Beyond tolerating our daily intrusions, participants actually valued and enjoyed the experience; it provided accountability, motivation, and an opportunity to reflect on how they were truly feeling about their barriers to exercise. 

Dear diary…

While undoubtedly instrumental in the high engagement we experienced during that study, this positive participant feedback does raise an important point about diary studies. Recording how you feel and what you do is an inherently introspective exercise, and one that may not reflect “business as usual” for most people. Depending on the diary study, taking part is itself an intervention with potential impacts for participants. 

This is important to keep in mind both when designing the study and analyzing the results. Introducing a control group that undertakes a similar diary entry exercise but is not exposed to personalized input is one way of helping to tease apart these effects.

Personalization for all twists and turns

While personalization is often based on masses of big data, diary studies build a foundation for products that don’t just know users based on algorithms, but, vitally, understand them on a personal level. They shed light on the current and potential role of a product or service in users’ lives. Through diary studies, we learn how products can be more useful and better accompany users in overcoming day-to-day challenges and achieving their goals.

The Decision Lab is a behavioral consultancy that uses science to advance social good. We have created bespoke diary study designs using robust experimental methods to probe deeper into the experiences of real users and support the design of more useful and impactful products. If you are eager to explore how we can create tailored solutions that drive positive change in the world, contact us.

References

  1. Floyd, I., & The Decision Lab. (2023, October 24). How much research participants want to be paid. Tremendous. https://www.tremendous.com/blog/how-much-research-incentives-pay-participants/
  2. Janssens, K.A.M., Bos, E.H., Rosmalen, J.G.M. et al. A qualitative approach to guide choices for designing a diary study. BMC Med Res Methodol 18, 140 (2018). https://doi.org/10.1186/s12874-018-0579-6
  3. Kotamarthi, P. (2021, June 13). This Is Personal: The Do's and Don'ts of Personalization in Tech. The Decision Lab. https://thedecisionlab.com/insights/technology/this-is-personal-the-dos-and-donts-of-personalization-in-tech

About the Author

Caitlin Spence

Caitlin Spence is a Senior Associate at The Decision Lab. Before joining The Decision Lab she worked in Aotearoa New Zealand’s justice sector as part of a team using behavioural science to create more accessible and culturally aware systems. Caitlin is interested in using data and experimental design to understand how systems can be designed or changed to favour positive and equitable outcomes. She holds a Bachelor of Arts, majoring in Statistics, from the University of Auckland.

Read Next

Insight

The Mental Cost of Economic Insecurity

Explore the profound link between economic insecurity and mental health. Uncover how poverty affects well-being and the bidirectional impact between financial stability and psychological health, urging for holistic solutions to break the cycle of poverty.

Notes illustration

Eager to learn about how behavioral science can help your organization?