Evidence And Values In Policy And Research

Brooke: So the outcomes that we set for ourselves are not evidence-based or evidence-driven or anything like that, and they’re not supposed to be. The evidence only comes in once you have a motive. The first layer of kind of problematization around that ideal is that once you have an objective set, all you need to do is look at the evidence and it’ll tell you how best to get there. That first step of setting an objective, it’s not an evidence-relevant activity. Evidence can be used to identify instrumental means to reach a goal, but goal selection itself is an inherently normative, values-driven thing that the evidence just doesn’t drive.

Nathan: Right. And you can probably bring evidence in, to kind of shape the goal selection. If you’re looking at quality of life, there’s certain pieces of evidence or research that can kind of inform that evaluation, right?

Brooke: And this is where we start to get into a second layer of critique of that very, very kind of stringent, hyper-simplified evidence-based ideal of policy-making. That second thing is that even as we shape our values and our preferences, we keep in mind how effective things can be. Often, the narratives, not just between people, but even in the way that we conceptualize the outcomes that we want, are often strongly informed by the indicators we use.

Brooke: So for instance, when we talk about quality of life, often, the way that we think about quality of life will be strongly informed by the way that we assess it. I think everybody is pretty much on board with the idea that quality of life is something that should be promoted. What they don’t agree on is what makes a high-quality life. Even something as simple as quality versus quantity. Is an additional year at 10% less quality inherently more desirable than one less year, but all of the years between now and then being 10% higher quality? I think that there are big disagreements about that.

Nathan: How do you find answers to those problems without using evidence and research?

Brooke: That’s the thing. I don’t think that the harsh distinction between facts and values is one that we should get on board with. We should embrace this more complex kind of interactive relationship between facts and values, where even the way that we conceptualize our values will be informed by the types of facts, or evidence, that we are creating. And when I say creating, I don’t mean fabricating data. What I mean is we must choose a measurement protocol in order to create data. And in making those methodological choices, we’re doing just that. We have an active role to play in how the evidence is created.

Nathan: That reminds me of what I have read by Jürgen Habermas

Brooke: Oh, of course. Critical theory is all about this, right? There are some really interesting things going on right now in critical data theory and feminist data theory about how the datafication of the world is not kind of a neutral medium through which we view the world of experience. But actually, these media themselves actually have a perspective. They are a specific prism through which we view the world in which we live.

Nathan: I wonder, and TDL just published a recent article that mentions this sort of neoliberal idea that companies would just be better off if they could kind of cut through all their biases and hire the best people. Do you think that’s kind of an oversimplification where you’re not going to recognize the best people precisely because of the kind of structure in which we’re making those assessments about who’s valuable and who’s not?

Brooke: Yeah. I think that the oversimplification in that instance comes from the term “best”. Along which dimensions are certain people the best? If there is a very clear and unproblematic way to define that, then I agree that we can probably get this argument off the ground that all we need to do is de-bias the process and then we’re golden. But actually trying to define who the best candidate will be is an extremely difficult and fraught process. And in fact, I think that some of the most interesting things happen specifically when we get into productive arguments and productive disagreements about what “being the best” means in terms of hiring, in terms of fit, these kinds of things.

Nathan: Do you think evidence has to do with how we determine the best or is that fully something that’s normative and the evidence is selecting someone once we’ve decided what our goals are? How does an interaction between evidence and values play out in that kind of specific context?

Brooke: Here I think is a good opportunity to kind of pull open that complexity, that interactivity between norms and evidence. We might say, “Okay, well, I want to define ‘best’ along five dimensions, A, B, C, D, and E.” I can only do that by drawing from this kind of lexicon of measurable stuff that is out there. So, that lexicon, that armory of tools, which we can go and pick up in building our normative definition, is where the evidence is that informs values, that values are intimately connected to our ways of building evidence. Maybe it’s not evidence that builds up our normative values. It’s methodologies.

Nathan: So it’s the way that we collected our data that matters. 

Brooke: That’s right. So, methodologies are maybe a place that we should focus more on this discussion, that methodologies are intimately connected to both values because we need some way to concretize our values and to our evidence because we need some way to collect our evidence. In the absence of methodology, we will really struggle to define our values or to concretize them, and we’ll be completely at a loss for how to collect the information to try to identify who best fits these normative descriptions, like who’s the best candidate for this job position.

Nathan: So, let’s go one level deeper. Where do we determine our methodologies from? I mean, I know you said values. But in a concrete sense, if you’re going to conduct a survey, a lot of this comes from past experience, right? To use the language of your paper, what are some incision points, or touchpoints, where one can actually intervene and change the process?

Brooke: One of the most valuable interventions in terms of identifying what evidence will be relevant for a problem, helping them to collect that evidence, helping them to process it both in kind of a very technical data science type of way but also in this much softer, decision-making institutional process kind of way as well. One of the things that’s most important in that work is keeping visibility on this whole kind of cascade or flow from the type of outcome that you want, which guides the type of evidence you identify as being relevant, and in turn, influences the type of methodology that you select to go to collect evidence. Finally, the analysis of what you collect leads to the decision-making process that is informed by that analysis.

Brooke: For me, I think the incision point is really about keeping visibility on that whole pipeline. That’s something that in a lot of contexts breaks down. So the functions that I just talked about in that whole cascade, institutionally are often divided into very siloed ecosystems within an organization. The person who’s responsible for creating the data probably doesn’t have great visibility on the institutional process that is going to be informed by that data later on down the line. I think in some ways data scientists have been in a privileged position to do that. Specifically because for a number of years data scientists kind of escaped or eluded very concrete descriptions of what the role was. They were more or less labeled as these unicorns or wizards who just did everything that touched on data within an institution or an organization. 

Brooke: And in that respect, because they eluded that kind of description, because they eluded that pigeonholing, they were also allowed to have the freedom to walk all over the boundaries that constrained most people in an organization. In having a role that was allowed to just meander liberally across all of these boundaries, we created that kind of interconnection between the silos that allowed better decision making to take place. In that respect, good data scientists were good data scientists and really enabled organizations to make better decisions not just because of their technical mastery but also because of the unique role that institutions allowed them to occupy that normally they don’t allow anyone else to occupy, just like a transversal cut across the organization.

Nathan: Just kind of going against that sort of traditional Adam Smith sort of specification of labor thing.

Brooke: Exactly.

Nathan: Do you think the success of data scientists is a critique of that idea of specification of labor? Do you think that’s something that needs to be revised just when we look at organizational behavior and you have this CEO that’s really disconnected from the kind of nitty-gritty data collection sort of thing. Is that a problem given this sort of framework?

Brooke: Yeah, I think that it is. I think that it becomes a problem when our methodologies start to evolve very rapidly. Basically, as long as your methodologies are evolving very, very slowly, you don’t run into these challenges where a CEO, for instance, ends up with some kind of data report on their desk which might just be a one-page executive summary of insights. You don’t end up in a situation where a CEO can have that kind of product arrive on their desk and the entire process leading up to that product will be opaque.

Brooke: If the methodologies are the same ones we’ve been using for 50 years, the specialization of labor becomes less problematic. Because when the CEO ends up in their role, there may not be that much difference between the way that the process was happening when they had their hands kind of deep in the muck and the way that the process is happening now that they’re more hands-off in that kind of day-to-day execution.

Brooke: As methodologies start to evolve more rapidly, that type of system breaks down because the people who have taken this step back and are taking a wider strategic lens on what’s going on cease to have a good transparent visibility on what the day-to-day operation looks like at the ground level.

Nathan: One of my colleagues in computer science talked about this in the context of his field. Basically, if you don’t move up from being a coder within 5 or 10 years after getting your degree, your skills are obsolete, so you have to move into management before you kind of just stop being useful to the company. I think that’s a total example of the disconnect between the people running the show and the people on the ground using new tools that superiors don’t know how to use.