One of the more interesting and less-talked-about elements of big data is how it interacts with what was prevalent before it came along – namely, more traditional research methods.
There has been a seismic shift in how companies approach their fundamental business problems over the last few years from a research-based to a data-driven approach. At Deductive, we have seen first hand in TV analysis, typified by the shift in assessing TV audiences using panel data only, to a mixed-methodology that includes smart-TV, set-top box and OTT data.
The two approaches are fundamentally different. Panel data is generated by a strict research-based approach and provides a “gold standard” for audience measurement in most markets. It is a classic example of a small, managed data set.
Panel data now sits alongside other far larger, more unwieldy datasets that teach us more about how people watch TV, and which allow us to act directly on those insights. But these datasets require much more management, and work, to be useful for us.
In with the new
The relative merits and demerits of panel data and big data have been hashed out multiple times. But what’s particularly interesting is how we see some of the same trends – and debates – occurring in other industries as well.
- In healthcare, we are starting to see a shift from focusing on clinical trials to one where data gathered from all hospitals is used to improve services, while pharmaceutical companies are focusing on personalized medicine based on genetic data and data gathered through Internet-of-Things devices
- In the auto industry, using data to decide what makes a great driving experience and an attractive and exciting car has replaced the research that answered those same questions in the past. In finance, research to help banks, hedge funds and insurance companies make their complex decisions are replaced with much larger data sets, and in some cases it is artificial intelligence and not humans which are making the day-to-day buying, selling and loaning decisions.
- In the chemicals industry, companies have typically achieved their resource and planning allocation as a process based on research that would be implemented once or twice a year. This is now an ongoing process, based on the collection and understanding (through mining and analytics) of big data, resulting in vital improvements in efficiency and cost-saving that are being implemented much more often.
The most interesting startup disrupting an industry through the innovative use of data is probably Tesla in the auto industry. The company broke the mold by saying that its electric cars were going to be better, i.e., faster and better looking, as well as more environmentally-friendly, than their gas-driven competitors. They have since consolidated their success with driverless vehicles, a success based in part on Tesla's analysis of the data gathered about the driving habits of 60,000 of its vehicles sold with this data-gathering ability. And, by updating the software in its cars, Tesla can upgrade their features, e.g., adding driverless features, without the cars having to return to the manufacturers. And, because of Tesla’s success, the more established players in the industry have followed Tesla, both with electric cars and with their use of data, algorithms and artificial intelligence.
At Deductive, our data strategy team advises established companies on how to use data in innovative ways to remain competitive and prevent disruption from new and innovative data-driven startups. We can see resistance to this new data-driven approach, often because the data is not “managed” in the same way research panels are managed. The data needs gathering and significant pre-processing, cleansing and validation before it is useful. This is a far cry from the more "finished" product that is the output of a research panel.
Learning to get along
Looking at other how the old and the new co-exist in other verticals may give us some clues as to how these two opposing approaches will eventually co-exist in TV. To use our healthcare example, we haven’t seen an end to the use of clinical trials in healthcare – and we probably won’t. They are still heavily regulated, and the data that comes from said trials still drives whether a drug does or does not get manufactured. But big data from lots of hospitals is helping the healthcare industry solve problems and answer questions that clinical trials cannot - better and closer monitoring of individual patients, more and more granular inputs for modelling, population health management, to name but a few use cases. These applications are ancillary to clinical trials – not a replacement, and the same is true for TV. New data sets are supplementing the data analysis we can do. Tesla is probably the most extreme example of this - it is doing things with data that could never be done via a managed approach.
The other impact that is common across industries is a change in the amount of attention given to different parts of the data workflow. A huge amount of labor goes into the design and creation of a clinical trial, which results in a small, manageable and, hopefully, accurate data set. There is a parallel in TV here too. Much of the effort in creating a panel goes in the initial management and design stage, and then in the maintenance of said panel (as opposed to the data, which is an important distinction to make). A big TV dataset does not have this need, but a huge amount of work goes into managing the output of such data sets (something our data engineering team can attest too!).
The question is not whether one approach is superior, but what either approach can actually do. As has happened in other verticals, we expect new data sets in TV to continue to thrive, as they continue to help the TV industry to do things that old market research methods were never designed to do.