September 18, 2024

Present

Geoffrey Fox, Armstrong Foundjem, Christine Kirkpatrick, Ying-Jung Chen, Juri Papay, Gregg Barrett, Gregor von Laszewski, Riccardo Balin, Victor Lu, Aïmérou Ndiaye

Tentative Agenda

Any New Members Introduction
Next Steps of our Working Group: Activities or Projects
White Papers
Status of Benchmarks (OSMIBench, MOM6, Kaggle Calorimeter)
Any Other Business

New Members

Armstrong Foundjem is still at Montreal Polytechnique. See Old New Members Link
Ying-Jung Chen is rejoining group and is now at Georgia Tech completing degrees. See Old New Members Link and https://www.linkedin.com/in/yj-elizabeth-chen/. She has left Descartes Lab and is now collaborating with Geoffrey Fox on time series for hydrology
Aïmérou Ndiaye https://www.linkedin.com/in/aimerou-ndiaye-867708162/ is an AI Engineer @ BAAMTU TECHNOLOGIES in Multilingual NLP & Speech Research. Baamtu https://www.linkedin.com/company/baamtu/ is in Dakar Senegal.

Discussion

We declined entrance to several bots read.ai, fireflies.ai, Bubbles notetaker, Jamilu’s notetaker, Tyler’s notetaker and Tyler’s ai task manager
We identified 6 projects for the group which are listed below. We will draft descriptions and form subgroups to discuss.

Working Group Activities or Projects

SOTA Generative AI Benchmarks for Science The Science WG's next initiatives should include the most interesting and challenging AI applications, as seen by users, providers, and vendors of AI infrastructure. Further, we expect benchmarks to be well documented so they are valuable for students and researchers wanting to apply the latest technologies to different applications. Also, we recognize that to have comparative value between machines, the benchmarks cannot keep changing and must have a clear historical tracking. On the contrary, a science discovery metric will likely require a continuously changing SOTA (state-of-the-art) technology. This suggests we identify leading edge (kernel) AI for science examples where we keep both a historical implementation and one that is at the leading edge. We intend these artifacts to be useful in many ways, both for new machine solicitations and for users needing performance estimates to estimate the time needed on AI resource allocations like NAIRR. In addition, the broad educational values have already been discussed. As we combine the goals of the HPC and Science working groups for this new set of benchmarks, we need a strong collaboration between these working groups. As well as giving documented examples, we can support other initiatives such as generating useful datasets from generative AI models, white papers and identifying key features determining performance described below. A natural starting point is the physics applications described by Nachman earlier this year Special Seminar May 23 2024 We can also include the Kaggle Calorimeter Challenge where we have three exemplar codes including Caloscore from Nachman . There is a lot of interest in diffusion models used extensively by Nachman but there aren’t clear documemented examples for the most innovative data unfolding applications Nachman described
SimAI model from Riccardo See Special Seminar July 24 2024
Key Features of Models as in analysis by Juri benchmark_predictions_v2.xlsx September 3, 2024 for DeepCam. Juri will come up with a template that can be discussed. We want to state so that it can be applied to other models for the same problem and be used to define and improve needed systems. We should look at Science and HPC working group examples
White papers led by Christine Kirkpatrick. Victor Lu and Armstrong Foundjem are interested in participating
Using Benchmarking Data to Inform Decisions Related to Machine Learning Resource Efficiency https://docs.google.com/document/d/1gOKA8BnlJnsTAELWFSmL7Fl7kJej_UrNH-FVXbZFxGI/edit?usp=sharing Christine and Gregor are working on Overleaf version. Victor Lu is interested in this
Benchmark Carpentry https://docs.google.com/document/d/15YIlAWOBA2_xjXkTnAZmaw003Jh4eqURVZYQHhdGYdQ/edit#heading=h.fa0u4qc1plw5 This is next priority
AI Readiness of MLCommons Science https://docs.google.com/document/d/1NbL-VdkrY9jzPxveOys2RCK8TdEJ7O5wgnxjAgzK-rE/edit?usp=sharing
Paper on Experiment management with Gregor, OakRidge and HPE. This is short term project and Gregor will send information around.
Time Series. Ying-Jung Chen and Geoffrey are working on time series for hydrology. See last meeting AI for Science Foundation Models and Patterns and the testing of multiple models in https://arxiv.org/pdf/2408.11990
OSMI and MOM6. Extensively discussed in previous meetings