Data-driven decision-making

What counts is what gets measured?

Photo by Markus Spiske, courtesy of Pexels

We have an abundance of data. Just by using the myriad of apps on your smartphone or Googling questions in anticipation of countless solutions to your problem, mounds of data are being captured. A researcher can scrape data from public social media profiles, for instance, to analyze and study, producing theories about an array of phenomenon as well as research conclusions. (I have conducted such a study using Twitter data.) Do you have a question about a pressing issue? Look at the data.

With such an abundance of data, the logic goes, we should use it to make decisions in all aspects of life. Why wouldn't we? The more scientific our decisions, the better. Right?

Sometimes, yes, but not always. Data-informed decisions may not ultimately lead to the outcomes we desire, and data certainly miss worthwhile information.

Schools, since the institution of standardized testing, have collected hoards of data on academics and, in the case of international exams in particular, home life. The data are used to measure the success of schools in teaching what they aim to teach, as well as what might be out of the school's control, such as parents’ socioeconomic status. 

Literacy and numeracy rates for districts, states, and nations are derived from data, for example. Certainly, literacy is an appropriate proxy for aspects of success in life, specifically success in the workforce, but such data cannot tell you whether someone will live a fulfilling life. A fulfilling life is the result of more than just high letter grades and test scores—the most common data points used to determine whether a student is worthy of a credential. And a student's performance metrics may tell you nothing about their potential. There are many bright students who don't do well in school because they are bored with content that is too easy for them or they don't see the value in learning what's on offer.

The hyper focus on data-driven decision-making often misses what matters most in a student's life. One can get straight As, ace every test they take, yet still be unsuccessful and unhappy in life. They may still be unable to hold down a job, lack a sense of purpose, or live a life devoid of meaningful relationships. They can be chronically anxious, depressed, and stressed despite having received high marks on every measure that mattered to school officials and politicians.

Data-driven parenting is trendy. In some respects, such an approach to pregnancy and parenting makes sense. If the data do not show that eating sushi every now and then is harmful to an unborn child, why would we keep telling mothers that it is? Emily Oster is driving the data-driven parenting craze, and she uses the best studies available, those that use the gold standard of research design (when feasible), to guide parents in what to do and what not to do based on data.

But the data can't always capture what matters most. Sure, the physical health benefits to a breastfed baby decrease after three months, for instance, but what about the benefits derived from bonding with the mother while feeding. There is no rigorous test that can wholly capture that phenomenon. 

And yes, a C-section (versus a vaginal birth) should be avoided if possible, but the data show no long-term harm to mom or baby. Yet, certainly the inability of the mother to see her baby born or hold him or her immediately after birth has mental and emotional effects. It's difficult to know what those are because those data are challenging, if not impossible, to collect. The mother, especially a first-time mother, has no point of comparison and there are too many factors at play to isolate the cause of any future emotional impacts. The noise of other aspects of life disrupts claims that can be made from such data. Therefore, when a mother-to-be reads a data-driven parenting books, such as Expecting Better, these measures are largely absent.

Someone can likely point me to studies that claim to collect data on these questions of how test scores predict fulfillment in life or how breastfeeding impacts bonding between mother and child, but they don't show the sort of data that the gurus of data-driven decision-making hold in highest regard. They aren't raw numbers that can be plugged into a model. 

Making decisions based on data is all well and good, but what counts most in life cannot be measured and presented with a tidy bow wrapped around it.