DOE Data Days

The Department of Energy (DOE) has joined the larger scientific community in the promotion of data management as a means to higher quality, more efficient research and analysis, and as a critical component of data science. Tools and platforms to support data management and analysis are rapidly evolving and provide enormous opportunities. They also pose challenges that can be specific to DOE but are common across DOE mission areas and organizations.

The DOE Data Day workshop, abbreviated to D3, was born from this critical work. D3’s primary goals are:

Bring DOE institutions together to share their data management use cases, challenges, and solutions;
Identify potential synergies and efficiencies; and
Establish proactive channels for future collaborations.

The event crosses program boundaries and mission areas, with participants exploring best practices and the latest technologies to help DOE researchers leverage new techniques, respond to data security threats, and advance fundamental science in valuable ways.

2024

four people and a moderator on stage for the panel discussion

Read about this year's event via LLNL News: Data Days workshop gathers DOE national labs to discuss future of data management

Workshop dates: October 22–24, 2024

Hosted by: LLNL

Themes:
Cloud and Hybrid Data Management
Data Intensive Computing
Data Curation and Governance

Resources:

2024 Agenda (PDF 223KB)
2024 Photo album

Tuesday, October 22

Welcome and LLNL perspective
NNSA HQ perspectives and goals

Cloud and Hybrid Data Management

Wednesday, October 23

DOE Leadership Panel

Data Intensive Computing

Poster Session

Advancing Data Intensive Computing via Data Compression/Reduction at Extreme-Scale
PerSSD: Persistent, Shared, and Scalable Data with Node-Local Storage for Scientific Workflows in Cloud Infrastructure
Creating a Cross-Lab Curation Portal Featuring ML/AI Metadata Extraction
Driving DOE Forward: Scaling Data Governance and Stewardship for Strategic Success
Empowering the Marine Energy Community with AI-Ready Data from the Portal and Repository for Information on Marine Renewable Energy (PRIMRE)
Project Alexandria – Managing and Enabling Discovery of DNN R&D Data
Got Data? Discovery Is the Key
AI for Automated Citation Metadata Extraction in an Open Data Repository
OWL & SCROLL: Natural Language Models for Knowledge Preservation and Workforce Development
Bernie-AI and Beyond
Standards and Quality Control Processes for Earth Science Datasets
Data Governance and Stewardship for AI: An Effective Data Lifecycle Using Distributed Computing, Ontologies and Workflows for NeuroSymbolic AI
AI-Driven Knowledge Discovery Framework for Renewable Energy
Statistical Analysis of Convection in Variable stars Using Realistic Hydrodynamic Simulations
Breaking Free from the Human Chain: Automating Data for Impact
Capturing and Reporting NNSA Data Usage for the Berkeley Nuclear Data Cloud
Need-to-Know in a Data Virtualization Application
Building the Technical Ecosystem for the Data Archive (Darc): APIs, Analytics, Reflector
LANL Weapons Mission Technology Leverage of AI/ML for Big Data Ingestion, Indexing and Search
The World Data System: Representing the U.S. on the International Data Stage
Accelerating Deep Learning Training via Inter-Node Access Coordination Over Node-Local Storages
Preparing LANL’s Data for National Security AI Applications: The Ambitious Vision of the Mission Data Stewardship (Midas) Alliance
LLNL Data Curation: Current State, Issues, and Solutions

Thursday, October 24

Working Group Updates

Data Curation and Governance

2023

five panelists on a stage in front of an audience

Read about this year's event via LLNL News: Data Days brings Department of Energy labs together for discussions on data management and more

Workshop dates: October 24–26, 2023

Hosted by: LLNL

Themes:

Data Intensive Computing
Cloud and Hybrid Data Management
Data Access, Sharing, and Sensitivity
Data Curation and Metadata Standards
Data Governance and Policy

Report: 2023 Report (PDF 997KB)

Tuesday, October 24

Time	Topics, talks, and activities
7:00am	Check-in and hospitality HQ perspectives and goals
9:00am	Session 1: Data Governance and Policy DOE CDO enterprise data management: keynote by Robert King, DOE People manage data, systems help: institutional research data management An implementation plan for collectively improving DOE's geospatial science Ensuring DOE LM mission success through governance of environmental and geospatial data Creating institutional data governance and repositories Panel discussions
1:15pm	Session 2: Data Curation and Metadata Standards DOE data curation working group: keynote by Alex May and Olga Kuchar, ORNL Elevating nuclear weapon quality: data curation strategies for informed decision making ARM's data integration framework: how ARM focuses its resources on science, not data pre-processing Energy data eXchange: NETL's digital library and virtual laboratory supporting the fossil energy and carbon management research Interactive distillation of large single-topic corpora of scientific papers Building a resilient architecture for weapons program knowledge Panel discussion and breakouts
4:45pm	Adjourn

Wednesday, October 25

Time	Topics, talks, and activities
7:00am	Check-in and hospitality
8:15am	Session 3: Data Access, Sharing, and Sensitivity Globus and federated science data at DOE: keynote by Vas Vasiliadis, Globus PRIDE architecture foundations: architecture for digital engineering Verify database: centralized and secure technology verification Knowledge management and the Integrated Workflow Project Privacy preserving federated learning as a service: a key capability for building robust AI models for science Knowledge discovery for cross-disciplinary science via a knowledge graph Broadening the definition of data sharing: infrastructure and cultural needs for success Panel discussion and breakouts
1:00pm	Session 4: Data Intensive Computing Data management in the age of autonomous laboratories and generative AI: a perspective based on scaling biological applications: keynote by Aravind Ramanathan, ANL Comparative performance evaluation of large language models for extracting molecular interactions and pathway knowledge Workflows enablement and advanced environments: supporting your workflows Data-model-system co-design for scientific workflows in converged computing environments Building a world: the data challenges associated with low-latency environment generation for decision support Catalyzing scientific advancements: cutting-edge workflows and collaborative computing at the Advanced Photon Source Improving the understanding and representation of AMOC using advanced ML/AI techniques Panel discussion and poster session
5:30pm	Adjourn

Thursday, October 26

Time

Topics, talks, and activities

7:00am

Check-in and hospitality

8:15am

Session 5: Cloud and Hybrid Data Management

Using cloud to host large open-source datasets: keynote by Chad Rowan, NETL
Project Alexandria architecture
Improving ease-of-access to NNSA data stored in the Berkeley nuclear data cloud
Managing large scale drug discovery data in hybrid environment
Marine energy data, organized: the PRIMRE system
The MSD-LIVE data and computational platform
ESGF2-US data platform modernization
Panel discussion and breakouts

2:30pm

Adjourn

2022

Workshop dates: June 1–3, 2022

Hosted by: LLNL

Themes:

Cloud and hybrid data management
Data-intensive computing
Data access, sharing, and sensitivity
Data policy and ethics

Resources:

2022 Report (PDF 985KB)
2022 Agenda (PDF 208KB)
2022 Proceedings:

Presentations	Posters
Invited speaker: Data Policy as Translational Ethics (PDF 440KB) – D. Danks (UCSD) Invited speaker: Scientific Data Management for DOE (PDF 3.57MB) – M. Macduff (PNNL) Putting Data in the Spotlight: The Case for a Scientific Data Federation (PDF 728KB) – S. Boehm, S. Somnath, O. Kuchar (ORNL) Tsdat: An Open-Source Tool to Standardize Time-Series Data (.mp4 24.4MB) – C. Lansing et al. (PNNL) Data Flow: A Data Bridge to Air-Gapped Instruments (PDF 885KB) – G. Shutt, M. McDonnell, S. Somnath (ORNL) Invited speaker: Genomic Analysis and Learning at Scale: Mapping Irregular Computations to Advanced Architectures (PDF 6MB) – K. Yelick (LBNL) Indexing LANL's Historical Collections in Less Than 1,000 Years (PDF 1.2MB) – J. Maze (LANL) Data as an Ecosystem at ARM User Facilities (PDF 2.08MB) – M. Ihli et al. (ORNL) Materials, Aging and Compatibility Data Platform: A Case Study in Data Management (PDF) – S. Butterworth (LLNL) Making the Most of Data: Feature Engineering for Applied Supervised Machine Learning (PDF 2.6MB) – T. Martin (NETL) Invited speaker: Scientific Research Data Initiatives, Resources, and Support (.mp4 129.7MB) – M. Cooke (DOE Office of Science) DOE Office of Scientific and Technical Information Artificial Intelligence and Machine Learning (PDF 910KB) – M. B. West (OSTI) Building the Next Generation Earth System Grid Federation (ESGF2) (PDF 3.61MB) – F. Hoffman (ORNL), I. Foster (ANL), S. Ames (LLNL) APPFL: Open-Source Software Framework for Privacy-Preserving Federated Learning (PDF 2.75MB) – K. Kim et al. (ANL) PNNL DataHub: Building a Central Data Capability (PDF 4.1MB) – M. Hofmockel et al. (PNNL) Need-to-Know (NTK) Considerations for High Volume Data Access (PDF 263KB) – S. Byrnes (SNL) From ENSDF to NuDat: Disseminating Digestible Data Using Modern Web-based Technologies to Search, Filter, and Visualize Nuclear Data (PDF 3.43MB) – D. Mason, E. Ricard-McCutchen, A. A. Sonzogni (BNL) Managing Randomness to Enable Reproducible Machine Learning (PDF 426KB) – H. Ahmed, J. Lofsted (SNL) FAIRification for HPC Datasets and AI Models (PDF 237MB) – P.-H. Lin (LLNL) Towards a Big-Data Toolkit: Ensuring Data Governance & Ethical Considerations Are Applied to Large Datasets (PDF 241KB) – A. May et al. (ORNL) Ontology-Driven Discovery of Legacy Data in DMS (PDF) – C. Mathieu, K. Brunner (LLNL)	U on Up: Cloud Computing Across Classification Boundaries (PDF 823KB) – W. Rosenberger (LANL) Scientific Data Management: An Under-Recognized Grand Challenge (PDF 281KB) – S. Somnath et al. (ORNL) Survey of Scientific Data Needs at Oak Ridge National Laboratory (PDF 331KB) – S. Somnath, O. Kuchar (ORNL) Data Management System Enhancements for the APS Upgrade (PDF 1.01MB) – J. Hammonds et al. (APL) Unoccupied Aerial Systems: Development of a New ESS-DIVE Data and Metadata Reporting Format (PDF 818KB) – K. S. Ely, D. Yang, S. P. Serbin (BNL) Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets (PDF 51MB) – J. Bessac (ANL) et al. from Clemson University Distributed and Asynchronous Management of Science Workflows: Data and Computation Across DOE User Facilities (PDF 218KB) – S. R. Wilkinson et al. (ORNL) Using Semantic Relationships to Describe, Enhance, and Extend United States Nuclear Tests: July 1945 through September 1992 (PDF 396KB) – K. Brunner (LLNL) A Data Management Infrastructure for Multi-Lab Geoscience Projects (PDF 2.18MB) – K. Hodgkinson (SNL) et al. from LLNL, PNNL, LANL Towards a DOE Data Catalog: Ensuring Access, Sharing, and Protection (PDF 50.2MB) – A. May et al. (ORNL) The Energy Data Exchange: DOE Office of Fossil Energy Carbon Management’s Trusted Data Curation Platform (PDF 1.78MB) – C. Rowan (Maximus) et al. from NETL and Matric National Security Data Solution: Unsiloing Weapons Research Data (PDF 900KB) – V. Feagin et al. (LANL) Linking Legacies as a Basis for Categorizing Data in the Complex (PDF 519KB) – M. Ham (LANL) Data Enclaves for Scientific Computing (PDF 185KB) – A. Akram (UC Davis) et al. from LBNL The Challenges of Curating a Research Data Set from the World’s Most Complex Machine (PDF 129KB) – E. Andersen, J. Banning (PNNL) Toward a Petascale Data Hackathon for Exploring a Digital Twin of the Earth (PDF 4.22MB) – V. Anantharaj, S. Parete-Koon, T. Papatheodore (ORNL) Using Metadata Standards to Simplify Discovery and Navigation of Results (PDF 3.1MB) – H. Collier et al. (ORNL)

Presentations

Posters

Invited speaker: Data Policy as Translational Ethics (PDF 440KB) – D. Danks (UCSD)

Invited speaker: Scientific Data Management for DOE (PDF 3.57MB) – M. Macduff (PNNL)

Putting Data in the Spotlight: The Case for a Scientific Data Federation (PDF 728KB) – S. Boehm, S. Somnath, O. Kuchar (ORNL)

Tsdat: An Open-Source Tool to Standardize Time-Series Data (.mp4 24.4MB) – C. Lansing et al. (PNNL)

Data Flow: A Data Bridge to Air-Gapped Instruments (PDF 885KB) – G. Shutt, M. McDonnell, S. Somnath (ORNL)

Invited speaker: Genomic Analysis and Learning at Scale: Mapping Irregular Computations to Advanced Architectures (PDF 6MB) – K. Yelick (LBNL)

Indexing LANL's Historical Collections in Less Than 1,000 Years (PDF 1.2MB) – J. Maze (LANL)

Data as an Ecosystem at ARM User Facilities (PDF 2.08MB) – M. Ihli et al. (ORNL)

Materials, Aging and Compatibility Data Platform: A Case Study in Data Management (PDF) – S. Butterworth (LLNL)

Making the Most of Data: Feature Engineering for Applied Supervised Machine Learning (PDF 2.6MB) – T. Martin (NETL)

Invited speaker: Scientific Research Data Initiatives, Resources, and Support (.mp4 129.7MB) – M. Cooke (DOE Office of Science)

DOE Office of Scientific and Technical Information Artificial Intelligence and Machine Learning (PDF 910KB) – M. B. West (OSTI)

Building the Next Generation Earth System Grid Federation (ESGF2) (PDF 3.61MB) – F. Hoffman (ORNL), I. Foster (ANL), S. Ames (LLNL)

APPFL: Open-Source Software Framework for Privacy-Preserving Federated Learning (PDF 2.75MB) – K. Kim et al. (ANL)

PNNL DataHub: Building a Central Data Capability (PDF 4.1MB) – M. Hofmockel et al. (PNNL)

Need-to-Know (NTK) Considerations for High Volume Data Access (PDF 263KB) – S. Byrnes (SNL)

From ENSDF to NuDat: Disseminating Digestible Data Using Modern Web-based Technologies to Search, Filter, and Visualize Nuclear Data (PDF 3.43MB) – D. Mason, E. Ricard-McCutchen, A. A. Sonzogni (BNL)

Managing Randomness to Enable Reproducible Machine Learning (PDF 426KB) – H. Ahmed, J. Lofsted (SNL)

FAIRification for HPC Datasets and AI Models (PDF 237MB) – P.-H. Lin (LLNL)

Towards a Big-Data Toolkit: Ensuring Data Governance & Ethical Considerations Are Applied to Large Datasets (PDF 241KB) – A. May et al. (ORNL)

Ontology-Driven Discovery of Legacy Data in DMS (PDF) – C. Mathieu, K. Brunner (LLNL)

U on Up: Cloud Computing Across Classification Boundaries (PDF 823KB) – W. Rosenberger (LANL)

Scientific Data Management: An Under-Recognized Grand Challenge (PDF 281KB) – S. Somnath et al. (ORNL)

Survey of Scientific Data Needs at Oak Ridge National Laboratory (PDF 331KB) – S. Somnath, O. Kuchar (ORNL)

Data Management System Enhancements for the APS Upgrade (PDF 1.01MB) – J. Hammonds et al. (APL)

Unoccupied Aerial Systems: Development of a New ESS-DIVE Data and Metadata Reporting Format (PDF 818KB) – K. S. Ely, D. Yang, S. P. Serbin (BNL)

Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets (PDF 51MB) – J. Bessac (ANL) et al. from Clemson University

Distributed and Asynchronous Management of Science Workflows: Data and Computation Across DOE User Facilities (PDF 218KB) – S. R. Wilkinson et al. (ORNL)

Using Semantic Relationships to Describe, Enhance, and Extend United States Nuclear Tests: July 1945 through September 1992 (PDF 396KB) – K. Brunner (LLNL)

A Data Management Infrastructure for Multi-Lab Geoscience Projects (PDF 2.18MB) – K. Hodgkinson (SNL) et al. from LLNL, PNNL, LANL

Towards a DOE Data Catalog: Ensuring Access, Sharing, and Protection (PDF 50.2MB) – A. May et al. (ORNL)

The Energy Data Exchange: DOE Office of Fossil Energy Carbon Management’s Trusted Data Curation Platform (PDF 1.78MB) – C. Rowan (Maximus) et al. from NETL and Matric

National Security Data Solution: Unsiloing Weapons Research Data (PDF 900KB) – V. Feagin et al. (LANL)

Linking Legacies as a Basis for Categorizing Data in the Complex (PDF 519KB) – M. Ham (LANL)

Data Enclaves for Scientific Computing (PDF 185KB) – A. Akram (UC Davis) et al. from LBNL

The Challenges of Curating a Research Data Set from the World’s Most Complex Machine (PDF 129KB) – E. Andersen, J. Banning (PNNL)

Toward a Petascale Data Hackathon for Exploring a Digital Twin of the Earth (PDF 4.22MB) – V. Anantharaj, S. Parete-Koon, T. Papatheodore (ORNL)

Using Metadata Standards to Simplify Discovery and Navigation of Results (PDF 3.1MB) – H. Collier et al. (ORNL)

2020

Workshop dates: October 5–7, 2020

Hosted by: LLNL, virtual only

Themes:

Data curation and standards: legacy data, existing data, and future data
Data-intensive computing, high performance computing (HPC), and data science tools for DOE's computing communities
Data access, sharing, and sensitivity
Cloud, HPC, and hybrid data management

A companion hackathon covering a selected topic from the topic areas above was held prior to D3. There was no fee to participate in D3, and an abstract was not required to attend the workshop.

Resources: