Feb 10, 2021

The WG welcomed the new comers, Tom Gibbs (NVIDIA), Bala Desinghu and Steve Farrell
Christine Kirkpatrick (UCSD) presented FAIR principles to the WG. The slides can be found at mlcommons_fair_0221.pptx .
Benchmark Updates:
Benchmarks on Docker / Singularity
Code release to GitHub
Benchmark CLOUD: Need to be completed
Benchmark SEMDL: Tried on Cloud - but waiting for SciML
Benchmark UNO: Secured datasets through Globus, but yet to be evaluated.
A presentation on SciMLBench will be made to the WG in future meetings. See this recent talk by Tony Hey AI for Science - The Fourth Paradigm, eScience and Open Science.pdf
Tom Gibbs raised a point whether testing these benchmarks on ARM. This has not been thought through but Tom Gibbs suggested it will be a good move and it can be facilitated.
Arjun stressed on the packaging aspects, and Jeyan echoed on the packaging aspects indicating that it will be free from packaging aspects, such as docker/singularity aspects.
Arjun also stressed that there must be a way or format for people to develop benchmarks to a specific format.
DataPods and Open Storage Network (OSN): OSN will be happy to host the SciML datasets (and framework will pull it from relevant mirrors). See OSN_summary.pdf
Some datasets can be locally generated instead of being downloaded from servers.
FAIR ideas should be integrated as part of the MLCommons.

All to act on benchmarking
Jeyan to work on initial evaluation of the Cloud benchmark
Jeyan to act on a memo about Singularity and MLBox (with Vibhatha)
Jeyan to act on the release of the benchmark suite to the WG
Geoffrey will act on (10) to ensure that FAIR ideas are to be integrated as part of the MLCommons and MLCommons Science agenda