Skip to content
Science

January 13, 2021

January 13, 2021

See slides here.

Discussion Points

  1. New Members to the WG: Nathan Frey (https://www.linkedin.com/public-profile/in/ncfrey) signed up to the WG shortly before the move to MLCommons he was lost in the move. Also WG has two new members: Anuroop Sriram from Open Catalyst (https://opencatalystproject.org) and Siddharth Goyal from the Facebook side that are interested in participating in the WG. Open Catalyst itself could be a good future benchmark candidate. When Geoffrey spoke to them in December, he suggested that they hold off joining so as not to get lost in the migration.

So that we do a proper job of on-boarding new participants, Geoffrey would like to get some thoughts on when you think it might be best for them to sign-up – perhaps towards the end of Feb once we have completed some testing on the initial benchmarks?
When these folks join at the WG meeting they can provide a 10 – 15 minute overview of themselves and what they are working on, followed by a WG overview and how it works?

  1. Status of the benchmarks: All participants reported on the progress with the anticipated benchmarks. All are progressing towards releasing a code. However, the main discussion was around getting the benchmarks prepared by the 4th of June. Gregg Barrett has not been successful in finding any suitable hardware instances in South Africa for testing the benchmarks (for example when he last checked, there was not a single DGX unit of any kind on the African continent). Gregg also mentioned that someone will be joining his group who is very keen to contribute to the Science WG efforts. He also wanted access to this platform for the new person to be enabled.

  2. Systems for benchmarking, and access to those systems: Systems from TACC, RAL, ANL and ORNL were suggested. However, no mechanism for access to members was formulated. This is possibly to be explored in the up-coming meetings.

  3. Access to WG minutes and documents: There is a great source of confusion on accessing minutes, documents and presentations connected to the WG, which is causing frustration across members. Most of the members, despite being part of the WG, do not have access. In December, Geoffrey attempted to merge the MLPerf Science list and the MLCommons Science list. However, he was not able to confirm the MLPerf Science list as he lost access to it. The MLCommons list was also in a state of flux at the time.

  4. I believe that Ruth McGinnis (ruth@mlcommons.org) is the person to resolve issues with the lists, but if that is not the case and you need me to do anything please let me know.

  5. Poster for the WG: As was done with some of the other WG’s, Geoffrey was happy to start working on a poster as soon as we have the initial benchmarks confirmed – and perhaps an online overview/introduction as we did for MLCube on Towards Data Science may be a good idea.

Action Points

  1. Jeyan to formulate a plan for providing access to the DGX2 system at RAL
  2. The WG group to decide the class of ML algorithms on which the benchmarks are based
  3. The Benchmark owners are requested to come up with relevant metrics (some of them being domain-specific)

TEMPLATE: Meeting, \<Month, dd, yyyy>

Attendance is taken automatically by the meeting software.

Agenda:

  • \<agenda item 1>
  • ...
  • \<agenda item M>

Meeting slides: \<path relative to the minutes doc, if applicable>

Agenda Item: \<agenda item 1>

\<text summarizing the discussion, data, outcomes; links to additional documents/materials that were presented>

Agenda Item: \<agenda item 2>

\<text summarizing the discussion, data, outcomes; links to additional documents/materials that were presented>

Action items:

  • \<description of AI 1> (\<list of people taking the AI>).
  • \<description of AI 2> (\<list of people taking the AI>).

TEMPLATE: Vote, \<brief proposition summary> \<Month, dd, yyyy>

Proposition: \<text describing the proposition as it was defined for the vote>.

Number of members eligible to vote: \<number of working group members eligible to vote, as defined in the working group policy document>

Participants in the vote: \<number N of eligible members that voted>. \<voting member 1 (\<affiliation>)>, \<voting member 2 (\<affiliation>), … \<voting member N (\<affiliation>)>.

Vote result: {proposition passed by required majority and quorum, proposition did not pass due to quorum not reached, proposition did not pass by not reaching required majority}.

  • \<vote option 1>: \<number of votes for this option>
  • ...
  • \<vote option K>: \<number of votes for this option>