Machine Learning and Program Evaluation

The recent Journal of Economic Perspectives had two articles that are very much of interest to quantitative program evaluators.  One of them dealt directly with the subject of causality, which may be discussed in a later blog.  However, the importance given to a new field of statistics known as machine learning is the subject of this blog.  This is the focus of one of the papers and is significant in the other.  For many, this would be seen as signaling the rising importance of a subject that practicing evaluators need to be more aware of.  There will be no attempt to review the two articles in a short blog.  But there will be an attempt to distill the possible implications of machine learning to evaluators.

It goes without saying that all new empirical techniques have the potential to be useful to program evaluation.  Still it is worthwhile to keep reminding ourselves of this, if we truly pride ourselves on our ability to use multiple lines of evidence.  Recently, there has been tremendous growth in the field of machine learning.  This is tied dramatic increases in the ability of  computers to process vast databases and very real breakthroughs in statistical techniques.  Both of these developments will change the field of program evaluation, likely for the better.

What is known as machine learning is actually a variety of statistical techniques, some old and some new.  Basically, they focus on doing two things.  First, they can take a database of micro records and divide them into unique groups.  This provides a framework to test whether the standard ways of segregating the data, such youth, Central Canada etc.,  are still valid.  Second, these techniques greatly improve the predictive accuracy of statistical equations.  This does not mean that analysts can better predict the future for the next 20 years but it does mean that analyst can better predict the impact on sales of scarce advertising dollars.  It is important to note that with this emphasis on predictive accuracy, there is not any focus on the  age-old evaluation problem of self-selection.  In fact, to my reading of this literature, there appear to be no new solutions to self-selection offered here except possibly in the important case of performance monitoring.

With the use of machine learning techniques within a performance monitoring context, evaluators can observe changes in output in real time.  This raises the very real possibility that evaluators may be able to make a convincing case for causality if they are able to predict changes in outputs or even outcomes in response to known changes in inputs .  If this is the case, evaluators may find themselves in the world of Michael Barber where performance results impact program delivery in real time.  This would give the profession far more policy influence than it has with results released every five years that are at least one year out of date.

As it stands in 2017, it appears that the machine learning is not a game changer for evaluators. However, be prepared for possible successes for large programs with continuous streams of data. Also, the ability of these techniques to identify unique groups of clients may produce some surprises. Overall, the world of performance reporting will continue to evolve, and the best predictions are that this evolution will be for the better.

From one computer to the next the truth is distilled.