Michael McDonald, PhD - Across all of business there is a lot of interest in Big Data, Business Intelligence, Predictive Analytics, and other data related fields these days. Tools in these fields are particularly useful in the unconventional energy sector – they can help shale producers and servicers with everything from drilling more effective wells to responding to RFPs and maintaining infrastructure. Investors in the energy sector can make similar use of data to make more effective investment decisions.

This set of tools and techniques as a whole can be generically termed data analytics – and with major increases in computing power and software interfaces, 2017 may well be the biggest year yet for data analytics advances. Still for most novices in the field, there is a major misunderstanding around what data analytics can and cannot do.

Image Source: Defense Advanced Research Projects Agency (DARPA)

To begin with all data analytics processes start with a basic truism – garbage in, garbage out. If the data being analyzed is not accurate and representative of the world, then it’s not useful. This concept seems simple, but it is often forgotten. For instance, in a risk management function, people often think of data as being useful for extrapolating the likelihood of future events – but that is only true if we have data where the events we are worried about are actually occurring with the same frequency that they do in the world.

Take oil field servicers responding to Requests for Proposals (RFPs) for example – we can use a statistical model called a probit model to figure out the probability of a particular RFP being awarded given a variety of characteristics such as the servicer’s price, size, and competitive atmosphere. In order to model that effectively, we need to have data on the customer and the competitive market – size, profitability, industry, assets, etc . Once we have that data, we can figure out the likelihood of our bid being accepted given the prevailing situation in the industry. Equally importantly, data analysis can tell us statistically how confident we are in that outcome. In other words, we might be 82% sure that company XYZ would get a contract, while we are only 13% sure that firm ABC would get the award.

Yet in order to build this type of model, we need to have the right underlying data – that means having the right data on the firm, and having the right data on past deals that have occurred over a long period of time. In other words, building a data model requires investment of time and money – it is not a simple one-off process in many cases. Data analytics is powerful but only if we have the right tool for the job. Many industry insiders say that the single biggest problem that is holding back effective use of new data-related tools and technologies is the lack of data.

The second major issue with data analytics is that we need data which is properly cleaned and compiled. Most of the time the data used for analysis comes from different sources, some of which are high quality and others of which are low quality. That means that the datasets have to be cleaned and merged together into a single larger database. This is difficult and time consuming in many cases, especially with large datasets such as those used in the energy sector.

Investors in the energy sector may have a slightly easier time getting data, though there is often a cost to do so, and a single database by itself is rarely all that useful.

For instance, when trying to forecast future oil demand and shortfalls in a particular region, one needs to use data on demand which come from one source, data on existing stockpiles which come from a second source, and data on characteristics of producers which comes from a third source. The three sets of data have to all be merged together based on a single unifying factor like date of the returns. Once this is done, the data have to be cleaned to deal with issues like oil producers that close up shop, or bid-ask bounce in commodities pricing. When you get done with this process, you have a formula that allows you to make forecasts about oil pricing in particular regions – but again it requires time and investment to get accurate results.

In the energy sector, as in so many other industries there is often an element of institutional inertia which leads to less interest in new ideas. Those who do embrace new tech like data analytics early on are likely to be the ones that see the most benefit though. The key to such efforts though is investing in new data analytics capacity as a process rather than thinking of it as a one-time effort.

For those who have tried data analytics techniques with or without success in their firm, I welcome comments and feedback on how your efforts went.

Background Note:
Mike McDonald is a PhD in finance and a university professor in the subject. He also runs a consulting company doing work on quantitative investing, big data, and machine learning for a variety of financial firms, asset managers, institutional investors, and government regulators. Prior to getting his PhD, Mike worked for a major Wall Street bank and one of the top hedge funds. Comments, questions, and concerns are always welcome – email Mike at M.McDonald@MorningInvestmentsCT.com or visit his firm’s website at www.MorningInvestmentsCT.com