November 1, 2023

Present

Geoffrey Fox, Wes Brewer, Gregor von Laszewski, Christine Kirkpatrick, Gregg Barrett, Ying-Jung Chen, Yuhan (Douglas) Rao, Piotr Luszczek, Mallikarjun Shankar,

Tentative Agenda

Any new members
Foundation Models, including the special talk last week Foundation Models for Weather Climate Valentine Anantharaj.mp4
White Papers
Using Benchmarking Data to Inform Decisions Related to Machine Learning Resource Efficiency mlcommons_data_energy_usage_paper
Benchmark Carpentry benchmark-carpentry
AI Readiness of MLCommons Science MLCommons Science FAIR Concept Paper
Other Benchmarks
AOB

Discussion

Arjun noted that Hugging Face tracked power
Yuhan Douglas Rao noted announcement from Met Office yesterday - AI to take weather forecasting by storm - Met Office
Collaborating with Turing Institute The Alan Turing Institute with headquarters at the British Library London
and that the CodeCarbon project is an interesting project on tracking the carbon emission of computing - CodeCarbon
Ne said there were 2 groups in NC State
TPC is Trillion Parameter Consortium: TPC-Hack-Start.pptx not to be confused with Trillion Pixel Challenge from ORNL
Could use Oak Ridge FORGE as basis of a benchmark; an open science LLM
Recently there is an Ocean LLM OceanGPT: A Large Language Model for Ocean Science Tasks | Papers With Code based on Llama-2 finetuned with 68K ocean papers
Geoscience LLM [2306.05064] K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization one million papers fine tuned on LLaMA-7B
HPC-GPT: Integrating Large Language Model for High-Performance Computing
BioBERT was original paper of this type BioBERT: a pre-trained biomedical language representation model for biomedical text mining and Med-PALM [2212.13138] Large Language Models Encode Clinical Knowledge just used a large LLM on medicine
Discussion of value of MoE Mixture of Experts
Gregor noted Table 1: Foundation Model Summary
Gregg Barrett and Arjun agreed that Science working group needed a differentiator for our activities and need to understand our challenges.
We need to align with NAIRR National AI Research Resource Task Force | NSF
We discussed meeting at SC23 but only did this informally
Christine mentioned publication identified for Power paper.

October 24, 2023 11am – 12pm (EDT)

Teams Link

Toward an open science foundation model for weather and climate

Valentine "Val" Anantharaj.Oak Ridge National Laboratory
Reliable forecasting of weather and assessment of risks at any location or region of the world requires the coordinated monitoring of weather and acquisition of data. A variety of data are routinely collected from diverse observing systems, including ground-based instruments and satellite platforms. The data are then integrated via numerical weather prediction systems using sophisticated data assimilation techniques on supercomputers to produce timely predictions and associated uncertainties that are used to make routine probabilistic forecasts and issue any associated warnings.

Recently, data-driven machine learning (ML) models have demonstrated their potential to augment the deterministic forecasts of NWP systems. A few of these ML models can also be broadly characterized as “foundation models” that have been developed based on AI Vision Transformer (ViT) architecture. The foundation models also offer capabilities to integrate heterogeneous data by fine-tuning a pre-trained model. This approach is very promising for weather and climate research and applications. Data from multiple modalities and resolutions can be employed for a range of new capabilities.

The speaker will also provide an overview of a recent collaborative initiative, led by NASA, IBM and ORNL, to develop an open science foundation model for weather and climate. The presentation will also outline and discuss the emerging opportunities for using foundation models leveraging the value chain of climate data from simulations and observations of the earth system while leveraging DOE exascale systems such as Frontier, and the Advanced Computing Environment (ACE) Testbed at the OLCF.

Recording of talk

https://drive.google.com/file/d/1P6eJDbzVxknKiDhCKqaMIbEa9deOxtW1/view?usp=sharing on MLCommons or at Oak Ridge https://drive.google.com/file/d/1Fhx6tNxzSjPJDDJGaFf57qIKR5kxAafh/view?usp=sharing

Present

Geoffrey Fox, Wes Brewer,Valentine Anantharaj, Tony Hey, Gregor von Laszewski, Piotr Luszczek, Aristeidis Tsaris, Murali Emani, Shantenu Jha, Dingwen Tao, Yuhan Rao, Amit Ruhela, Przemyslaw Porebski, and others for a total of 25.

Talk Highlights

Val described work with Oak Ridge, IBM, NASA, University of Alabama Huntsville, Chicago, Stanford, Colorado State, Its foundation model uses thes the IBM-NASA HLS (Harmonized Landsat Sentinel) Foundation model discussed earlier in our sessions NASA and IBM Openly Release Geospatial AI Foundation Model for NASA Earth Observation Data | Earthdata
Weather and Climate forecasting has quietly but dramatically improved over the last few decades
Now the Southern and Northern hemisphere predictions have similar accuracy and one sees the temperature inversion between low and high level atmospheres characteristic of climate change (now low levels are warmer, previously the opposite was true)
AI methods have come from NVIDIA (FourCastNet https://github.com/NVlabs/FourCastNet) Huawei (Pangu https://www.huawei.com/en/news/2023/8/pangu-weather-forcast) Google/Deep Mind (https://indico.dkrz.de/event/45/contributions/203/attachments/59/120/ESiWACE2%20talk_Hoyer%20virtual%20workshop%20on%20weather%20_%20climate%20modeling%20-%207%20October%202022.pdf and GraphCast [2212.12794] GraphCast: Learning skillful medium-range global weather forecasting), Micicrosoft (Climax, [2301.10343] ClimaX: A foundation model for weather and climate)
The Climax model is based on a Foundation model (pretrained on climate simulations and not observed data) but is not as well tested from fine-tuned results
Pretraining can be considered as an approach to fine tuning
GraphCast is particularly parameter efficient
Swin and Vision transformers are used.
FourCastNet was bad near poles but that has been improved
ClimaX illustrates the important idea of training with homogenized coarse resolution and then fine-tuning at higher resolution. This idea of multiple sophisticated fine-tunings based on a homogenized Foundation model is important. Fine-tuning can also introduce new variables
Surrogated and Digital twins are also being investigated
Nowcasting is predicting the next few hours.
Large language Model LLM Foundation models can also be linked to image-based Foundation models in a fine-tuning
Foundation models are an approach to Data Fusion and can have both simulations and observations as input
In Val’s project, Chicago is fine-tuning to understand clouds, and Colorado State is studying how many satellites are needed.
Private Industry interested in these developments
Ensemble methods (50 distinct simulations in traditional weather forecasting) need to be integrated with this.