Skip to content
Science

April 20, 2022

April 20, 2022

Present

Jeyan Thiyagalingam, David Kanter, Arjun Shankar, Junqi Yin, Gregor von Laszewski, Juri Papay, Gregg Barrett, Farzana Yasmin Ahmad, Aristeidis Tsaris, Piotr Luszczek, Cade Brown, Geoffrey Fox
Apologies: Tony Hey, Christine Kirkpatrick

Tentative Agenda

NSF POSE solicitation

  • First solication around technlogy focussed solicitation
  • See linked draft (Geoffrey, Vijay and Gregor) POSE-MLCommons-Science(PMS).pdf
  • Also see Pathways to Enable Open-Source Ecosystems (POSE) (nsf22572) | NSF
  • It funds an organisation of ecosystem (open source) (only for management)
  • More towards sustainability of ecosystem /organisation / work
  • 1 year for $300,000 (covering, research software engineering, community building)
  • This funding would help the benchmarking initiatives going in Phase I. The phase ii will offer more funding
  • Timeframe: first phase starting around Jan 2023, if successful.
  • Multinational activities encouraged but interbational collaborators are not funded

MLCommons Community Meeting April 21 2022

Juri Papay agreed to make this 10 minute presentation https://docs.google.com/presentation/d/1fBvvCPtP32G725Qv2o5sofzPtqCzXLbl084kq4rvSiM/edit?usp=sharing for the science working group. Note Livermore update https://docs.google.com/presentation/d/1k9NmajLqB2X_fCwEAbDpoWhmRdZR-bFStS6_0n2slR4/edit?usp=sharing LLNL_D4DCT_MLCommons.pdf was not discussed.
Presentations for all working groups can be found at slides (2021.04.22 MLCommons Community Meeting.pdf) 2022.04.21 MLCommons Community Meeting.pdf and recordings (2022.04.21 1Q22 Community Meeting Recording ).

Benchmark Status given ISC's May 22 deadline

  • Juri: STEMDL and Cloud - will be ready for 22nd deadline
  • Gregor: Earthquake Benchmarks on 4 different systems, will be ready for 22nd deadline.
  • The earth science paper from which Earthquake benchmark was derived has been published in the GeoHazrds journal https://www.mdpi.com/2624-795X/3/2/11

Status of Paper given ISC May 22 deadline

What should we do at ISC BOF June 2 2022

We agreed on a WG Benchmark Talk / Release and announcement

AOB: Formalizing Benchmarks

  • Good discussion with David Kanter
  • Need to Establish timelines for submission or release cycles
  • There was a discussion of existence of a validation dataset that is hidden in Kaggle competitions. Note that like Kaggle we want better methods.
  • Gregg: In competitions like Kaggle, the metrics are validated by withholding the validation data. If the validation data is out there, someone will train on it.
  • Cade: Also, if the benchmarks are reproducible, it should be trivial for someone somewhere to re-run the experiment to confirm
  • Aris: There for example competitions that are allowed to use external data for training
  • Need to put in MLCommons Repository
  • MLPerf Benchmarks are generally speaking are on competition track. Aa alternative is to have open submissions and peer review by owners and at least one other
  • Do we have fixed date or rolling submission
  • Reference owner is responsible for making the standard and will be peer reviewed by WG.
  • More thoughts are needed around standardising process. Need to define what needs to be submitted and how to check
  • David suggested to replicate what HPC WG is doing. Aris noted they use 2 scripts. https://github.com/mlperf-hpc/logging/tree/hpc-0.5.0/mlperf_logging/compliance_checker and https://github.com/mlperf-hpc/logging/tree/hpc-0.5.0/mlperf_logging/system_desc_checker
  • The standarisation aspect will be discussed as one of the main points in the next meeting.