Social Media as a New Source of Data for Program Evaluation

In many fields, analysts are discovering the use and value of what is known as “Big Data”.  In some senses, program evaluators are ahead of the curve, as they have been making use of administrative data for decades.  However, when it comes to the use of data from Social Media, evaluators have something to learn from other fields.  Evaluators also have to forge their own path, as our passion for capturing causality is not shared by others.

Why is Social Media useful?  A substantial portion of the population shares their experiences with the rest of the world through this media.  Both fortunately and unfortunately, a small percentage of these experiences relate to government programs.  Unfortunately, this amounts to finding a needle in a haystack.  Fortunately, it is possible to harvest  a new data source with sufficient effort.

It is true that the identification of program effects will be a challenge.  At times it may seem as if the evaluators are trying to compare needles in a haystack of participants with the haystack of non-participants.  Further challenges come in finding ways to work with enormous volumes of low-quality data without introducing biases.  And this can only be done after a way has been found to turn human discourse into data.

Is it worth effort?  Many evaluators will treat Social Media as a fad and ignore it.  For others, it will be a truly new source data.  The immediacy of the client experiences will provide insights not possible with other data sources.  Plus, they may find it will become possible to compare the experience of citizens in different countries.

Privacy issues will also present a challenge in the use of this data.  Governments will need to be able to prove that they are not spying on individuals.  Still on the flipside, governments are open to charges of arrogance if they do not listen to citizens as they express themselves through this media.
These are exciting times for evaluators.  I will be making my own contribution to this discussion at the 2016 Meeting of the Canadian Evaluation Society with a paper using Twitter Data.  Have a look at the most recent draft here.  I am sure that other will have different opinions.  We are guaranteed many lively discussions.

Sometimes it can be hard to find that needle

Other times it is easy