We’ve all faced decisions where both sides seem reasonable. One moment, you’re sure of the right path; the next, someone presents a compelling counterpoint that leaves you second-guessing.
Maybe you’re calling the shots on whether prospective employees must meet specific academic qualifications. Or you’re spearheading a public policy project that the entire community has an opinion on. It could even be something as basic as choosing new workplace software.
No matter the field, good decision-making requires diverse perspectives and reliable methods to synthesize them. Getting input from colleagues, community members, and even strangers online can help steer you in the right direction. But once you’ve gathered all that feedback, the real question becomes, what do you do with it?
In this article, I’ll share how TDL turns mountains of feedback into actionable insights. Our test case involved a professional board deciding whether a college degree was required for a specialized role. The result? An easier, clearer way to understand what’s most important to everyone involved. If you've ever felt overwhelmed by opposing ideas, spreadsheets, or counsel from all sides, this technique can turn chaos into clarity.
Gathering public input
Collecting community voices
In theory, we all understand the importance of entertaining different perspectives before making a critical decision. But in practice, when you receive a couple hundred (or more) comments, it’s easy to feel like you’ve taken on too much.
You know that feeling when your inbox is jam-packed, Slack notifications won’t stop, and you’re wondering if you’ll ever make it through the work piling up right in front of your eyes. That’s pretty much where we landed after reviewing over 250 people’s perspectives on the question: “Should a college degree be required for a particular specialized role?”
Some folks insisted that a formal degree is the gold standard for building trust and ensuring quality. Others pointed out that hands-on experience can be just as valuable, sometimes even more useful. And, of course, all sorts of people stood in between. However, when it comes to research, gathering feedback is easy; everyone has an opinion. The real challenge lies in making sense of it all.
Using AI to craft summaries
So how do you process so many opinions without getting lost in the weeds? We leveraged the power and efficiency of AI to classify each opinion, kind of like a personal research assistant. Every comment went into a ChatGPT-powered tool, which identified the core arguments and tagged them as either “in favor” or “against” requiring a college degree.
Using AI helped us avoid overlooking a great argument buried in an overwhelming set of opinions. And it’s not just a time-saver; this efficiency translates into real value. By reducing the hours spent manually coding feedback, teams can cut costs dramatically and refocus on strategic decisions rather than administrative tasks.
Of course, AI isn’t a silver bullet. While an algorithm can power through vast stacks of feedback—handy for everything from software rollouts to curriculum design—it can’t automatically catch all those subtle, human nuances. To keep things accurate and fair, we ran randomized quality checks. But even with that extra step, the time savings were significant.
Filtering and prioritizing arguments
After gathering all the arguments, from strong support for a mandatory degree to outright opposition, we asked ourselves: What themes keep showing up, and what ideas stand out? To find out, we used AI to sort each argument into existing categories or create new ones to capture unique ideas. Grouping related arguments helped us cut through the noise while ensuring each perspective got a fair hearing.
For each argument group, we captured four key data points:
- Frequency: How often an argument was tied to an overarching theme.
- Direction: Whether the argument supported or opposed requiring a college degree for the specific specialized role.
- Representative Quote: A direct excerpt from the original public letter illustrating the argument.
- Counterargument: An AI-generated response providing an opposing perspective.
Structuring the decision-making process
Tools for clarity
After sorting the feedback into clear categories, the next hurdle was figuring out which arguments held the most weight in the board members' eyes. We used two methods to help prioritize and evaluate the relevance of each argument:
MaxDiff Prioritization
MaxDiff (maximum difference) analysis is a choice-based survey method used to uncover which items matter most: product features, messaging concepts, employee benefits, or any other attributes. Instead of ranking a long list, respondents repeatedly choose the “best” and “worst” options from small subsets (typically 3–6 items). Across multiple rounds, these trade-offs reveal what people prefer and the strength of those preferences.1
Imagine you want to create a playlist ranking your top 40 songs in your music catalog. Instead of ranking all 40 at once, you present random groups of five tracks and simply ask: “Which song here is my favorite, and which is my least favorite?” After several rounds, some songs will consistently be picked as favorites, while others regularly fall to the bottom. Those response patterns translate into a clear, scaled ranking, so you know which songs resonate most and which are low priority.
This straightforward, engaging approach applies equally well to testing employee perks, feature sets, branding ideas, or any scenario where you must pinpoint valid preferences among many options.
Analytical Hierarchy Process (AHP)
After narrowing down the most critical arguments with MaxDiff, we turned to the Analytic Hierarchy Process (AHP). AHP is a structured decision-making framework that breaks down complex problems into a hierarchy of goals, criteria, and alternatives. This process uses systematic pairwise comparisons to generate numerical weights reflecting each element’s relative priority.2
Picture your board debating multiple strategic arguments. Instead of ranking them all at once, members compare two at a time: “Which is more compelling, Argument A or Argument B?” These matchups generate a clear, ranked list of which arguments carry the most persuasive power.
Using MaxDiff and AHP, we avoid letting the loudest voice automatically carry the day. Instead, each position gets a proper evaluation based on its own merits, which is exactly how it should be.
The board’s deliberation and next steps
Structured discussion
After running both MaxDiff and AHP, we were left with two clear lists, comprised of arguments for and against the degree requirement, ranked by how important they were based on our analysis.
The decision-makers gathered for a final round of debate. Some came in firmly believing in a degree requirement; others were of the idea that on-the-job expertise was more critical. But because we now had complex data on how many people shared each stance and how persuasive the board found each argument, the conversation took on a new level of clarity. They knew exactly what the other side of the argument valued most and the primary considerations that should be addressed regardless of the final decision. This helped the board members to engage in a more structured debate and ultimately arrive at a well-informed final decision.
behavior change 101
Start your behavior change journey at the right place
A model for informed decision-making
So how can you replicate this process in your endeavors? It’s simpler than you might think:
- Gather diverse opinions. Make it easy for people to chime in through surveys, town halls, or online forms.
- Use AI to categorize the feedback. AI can do the heavy lifting by sorting comments into themes quickly.
- Filter and prioritize using tools like MaxDiff. This helps highlight the arguments that people find most significant without drowning in a million details.
- Dig deeper with something like AHP. If you need more precision, pairwise comparisons can reveal how arguments stack against each other.
- Have a structured conversation. Use these insights to explore any unexpected twists before you lock in a final choice.
This approach is especially great for organizations with lots of input, whether that’s a charity balancing volunteer perspectives or a major corporation determining the next significant product innovation.
Where to go from here
If you’ve read this far, chances are you’re determined to make better decisions for your team, your community, or your customers.
Maybe you’ve been overwhelmed by too much feedback, or you’re dealing with a tough decision that has everyone split. This approach doesn’t just organize opinions—it brings structure to the chaos, helping you see what matters most. Ultimately, you’ll walk away with a decision you can back wholeheartedly, confident that a fair and transparent process shaped it.
If we know anything at The Decision Lab, it’s how to make informed decisions. If you're looking for professional help in navigating high-stakes decisions that will shape your organization's future, reach out—TDL can help you move forward with confidence.
