Abdulkareem Alsudais, Armstrong Foundjem, Ben Hawks, Geoffrey Fox, Gregg Barrett, Gregor von Laszewski, Howard Pritchard, Juri Papay, Marco Colombo, Nhan Tran, Piotr Luszczek, Satoshi Iwata, Victor Lu, Xiaoqin Huang
Tentative Agenda
Any New Members Introduction
Continuing discussion of New Benchmarks and the catalog of Science benchmarks
Abdulkareem Alsudais was introduced as a new member, and Geoffrey Fox encouraged him to share their LinkedIn and Google Scholar references. Ben Hawks provided updates on the benchmark collection paper, noting its readiness for ArXiv publication and a terminology change to "benchmark ontology" or "benchmark collection," along with the need for acknowledgements. Gregor von Laszewski requested the carpentry paper be cited and the website updated with the new terminology.
Marco Colombo demonstrated the updated website, featuring a new table view, detailed benchmark information, and a "cards view" with enhanced graphics. Gregor von Laszewski suggested re-enabling full-width table view, adding the radar chart and citation column to exports, and ensuring content consistency with Ben Hawks. Nhan Tran pointed out domain and data linkage issues between the website and paper, requesting direct dataset and code links on benchmark cards, and suggested an interactive heat map.
Geoffrey Fox suggested adding a comment about recognizing non-formal benchmarks and asked Nhan Tran to add instructions on contributing new benchmarks via YAML files and pull requests, which Marco Colombo and Nhan Tran supported. Gregor von Laszewski presented updates on the AI benchmarks carpentry paper, renaming it to "AI benchmarks carpentry and democratization," and requested contributors review the introduction, definitions, and ensure "carpentry and democratization" are addressed throughout. Armstrong Foundjem suggested an "implications" section for the paper, which Gregor von Laszewski agreed to add as a conclusion.
Suggested Next Steps
Ben Hawks will add a section to the paper's summary and cite the carpentry paper.
Marco Colombo will ensure that the CSV and JSON export includes a common column for citation.
Marco Colombo will rename the 'view local' function to 'make surf' and separate the publishing step from the make step.
Marco Colombo will add a link to the dataset and code for each card in the cards view.
Marco Colombo will implement a feature that displays all ratings in a 90-degree form when a button is clicked.
Marco Colombo will add the heat diagram to a future version of the website as an interactive feature.
The group will add a section to the website's main page explaining how to contribute a new benchmark via pull request.
Gregor von Laszewski will resolve the duplication created between the definition and carpentry sections in the paper and ensure the GPU benchmarking section is not overwhelming other sections.
Marco Colombo and Ben Hawks will check whether Ben's content changes are reflected in the current website version.
Marco Colombo and Ben Hawks will implement a visual tag for benchmarks with an average rating of 4.5 or higher.
New Members
Abdulkareem Alsudais, AI REsearcher and Associate Professor, PSAU Prince Sattam bin Abdulaziz University in Al-Kharj, Saudi Arabia. He obtained his PhD in Claremont Graduate University
