Dive Into a Lake of Data: Open Energy Data Initiative Increases Big Data Access for Everyone

---

When data is widely accessible and available to anyone, anywhere, it can transform communities—especially when that data concerns something as vital as energy.

The Open Energy Data Initiative (OEDI)—currently funded by the U.S. Department of Energy (DOE) Solar Energy Technologies Office (SETO) and developed by the National Renewable Energy Laboratory (NREL)—makes energy data widely available to anyone by providing access to big data via "data lakes," large collections of open-access energy data that are at anyone's fingertips in the cloud.

In the past, "big data"—data too large or too complex (or both) to be easily consumed by conventional means—could only be used by people with access to high-performance computing systems or supercomputers. OEDI makes vast amounts of energy data available for schools, companies just getting started, and data enthusiasts of all stripes. Each dataset includes access instructions and examples of how to work with the data.

Accessible from OpenEI, OEDI is a centralized location that can house data generated across the 17 national laboratories and all their partner organizations for research and development funded through DOE, with additional contributions from other federal agencies and private companies. Currently, over 200 providers from government labs, private industries, and universities contribute to OEDI. And that data is available to anyone with a laptop and an internet connection to access and use in the cloud.

OEDI's data repository runs the gamut, spanning high-resolution data for atmospheric conditions used every day by U.S. weather forecasters; real-world building data in hourly increments that shows buildings' energy consumption to serve as examples for engineers and architects; high-resolution data that shows how much wind or solar energy a given location can produce, not only for the United States but also many regions around the globe; databases about drilling and laser drilling; raw materials and supply chain analysis; and beyond, in over 2,000 datasets.

"Limited access to big data historically excluded startups, small organizations, and minority-serving institutions from research collaborations and community initiatives," SETO Director Becca Jones-Albertus said. "OEDI gives universal access to big data, fostering inclusive partnerships that enable better decision-making throughout the clean energy transition."

Open, Accessible, and Standardized

Traditional data repositories are not typically optimized for discovering or sharing data. Huge amounts of data further complicate data sharing by making the cost of storage and the time needed to transfer the data prohibitive, limiting access and reducing the potential for collaboration and innovation. Stored on the cloud, OEDI solves these issues by providing a centralized location for large energy datasets.

Making energy data free and widely accessible allows communities to work with researchers and the energy industry to find solutions that meet the communities' needs, fostering innovation that benefits everyone.

OEDI further supports accessibility by making data easier to use. Simple access is one thing, but if datasets are messy and difficult to work with, mere access means very little.

Converting raw data into standardized formats can be difficult and time-consuming. By cleaning and curating datasets, OEDI programmers aim to reduce the burden on individual data owners and project teams for data standardization, freeing up time and resources and improving the quality and consistency of standardized data.

OEDI data pipelines automatically detect certain raw data formats and standardize them, displaying the standardized data right alongside the raw data for scientific posterity. OEDI data standards adhere to the precedents from the U.S. General Services Administration's Open Data and FAIR (findable, accessible, interoperable, and reusable) data principles.

"With DOE support, NREL built OEDI from the ground up to share data," said Jon Weers, who leads OEDI for NREL. "Anyone can access the data through OEDI directly or through a network of data sharing partners like Data.gov, Google Datasets, and dozens of other sites."

Introducing Ask OEDI: Your New Research Assistant for Data Questions

Accessibility—making data available to the widest possible audience and reducing the learning curve for working with that data—is an essential function of OEDI. In support of FAIR data principles, NREL developed an artificial intelligence research assistant, AskOEDI, to help users find answers about datasets that go beyond simple searches.

AskOEDI is an integrated large language model with the metadata and supporting documents for OEDI datasets and can provide answers to users' questions using natural language processing and generative machine learning. Users can get answers to questions about specific datasets, including inquiries about the equipment, assumptions, and methodologies used in the origination of a dataset along with more abstract questions, such as the applicability of data to specific research fields. It only pulls answers from information vetted by OEDI and always cites sources, making it an ideal research assistant for OEDI data. AskOEDI increases the utility and discoverability of energy data by providing users with the means to quickly understand the nuances of a dataset without having to search through numerous associated publications.

"Many of the questions that used to be an email to the data owner can now be answered instantly," Weers said. "As long as they've been discussed in a linked publication or supporting document, AskOEDI can quickly find the answers to deeper questions on data such as 'How was this data collected?' or 'What assumptions went into the calculation used for this data column?' We're excited to see where this technology goes as we work in this emerging field of artificial intelligence."

Using Big Data for Big Impacts

OEDI is already impacting communities. The OEDI team supported communities in Puerto Rico in their transition to renewable energy in response to hurricanes Maria and Fiona. Working with DOE, the U.S. Census' The Opportunity Project (TOP), local governments, and local teams, OEDI researchers facilitated the development of five different innovative solutions to help Puerto Rico increase its resilience and transition to 100% renewable energy.

These solutions were developed by small teams of local collaborators focusing on community-driven solutions by applying and analyzing big data freely accessible through OEDI, which was recognized by Puerto Rico Governor Pedro Pierluisi during the TOP Summit.

"Open access to data supports local communities," Weers said. "In the wake of recent hurricanes, communities on the ground in Puerto Rico were able to use OEDI data to help them devise plans to rebuild and reinforce their energy infrastructure their way."

With OEDI, researchers from national laboratories to high school labs, from the United States and across the world, have greater access to valuable data to answer key energy questions. Over five years, OEDI has increased access to public data by 7,296%, to over 2.28 petabytes of data in 2024. Its data lakes now boast massive amounts of valuable data from solar, wind, geothermal, buildings, and marine energy programs to help anyone seeking to understand and contribute to innovative energy solutions.

Access the Open Energy Data Initiative on OpenEI.

Contact Jon Weers, NREL lead technologist and data systems architect.