Shale, Big Data and the End of Saudi Reign at the Pump

America is becoming closer than ever to energy independence. In Texas alone—the 12th largest oil producer in the world if it were its own country—production is expected to surpass 3 million barrels a day by the end of the year, pulling it ahead of Venezuela, Kuwait, Mexico and Iraq. In just over two years, the state has doubled its crude output thanks in large part to shale, reversing a 23-year decline.

Albeit controversial, shale is leading the way to U.S. energy independence. Big Data can help lower shale exploration costs and make production safer. Leaders in the oil industry are looking at Big Data, particularly Prescriptive Analytics with Hybrid Data, to not only predict where to extract shale energy from but also how to produce it more effectively and safely.

Prescriptive Analytics is the third and final stage of Big Data Analytics. First, there’s Descriptive Analytics, which tells you what happened. Second, there’s Predictive Analytics, which tells you what will happen. Finally, there’s Prescriptive Analytics, which tells you what will happen, when, why, and how to improve this predicted future.

Shale oil and gas production presents a number of analytical challenges and opportunities because of the growing volume, velocity and variety of data that needs to be analyzed and interpreted to make mission critical investment and drilling decisions. This information includes a hybrid mix of structured and unstructured data such as images, sounds, videos, texts and numbers.

For Big Data to succeed in oil and gas exploration and production—especially shale—it has to collect and analyze data from a number of sources, such as:

  • Images from well logs, mud logs and seismic reports
  • Videos from fluid flow from hydraulic fractures
  • Sounds from fracking, collected by fiberoptic sensors
  • Texts from frac pumpers’ notes
  • Numbers from production data, artificial lifts

By combining information from all these sources, Prescriptive Analytics can unearth key insights, predict problems and opportunities and prescribe the best course of action.

This requires a combination of a number of disparate technical disciplines. For example, to analyze and interpret sound data we have to combine machine learning with signal processing, pattern recognition and speech recognition.

For images, you have to combine machine learning with pattern recognition, computer vision and image processing.

By combining these disparate scientific disciplines, we have a more holistic view of where and how to drill in a way that allows you to preempt future problems without creating new ones in the process of doing so.


Shale has its challenges from public perception to general costs and safety. And all of this will require intelligent, proactive, big-data-driven decisions. The key is getting the total picture first.

By Atanu Basu

Atanu BasuShale, Big Data and the End of Saudi Reign at the Pump

Related Posts