Navigating Dataset Documentation in AI: A Large-Scale Analysis of Dataset Cards on Hugging Face

Xinyu Yang*, Weixin Liang*, James Zou

ICLR 2024

[PDF], [Code]

• We present a comprehensive large-scale analysis of 7,433 ML dataset documentation on Hugging Face.
• Based on our research findings, we emphasize the importance of comprehensive dataset documentation and offer suggestions to practitioners on how to write documentation that promotes reproducibility, transparency, and accessibility of their datasets, which can help to improve the overall quality and usability of the dataset community.

What’s documented in AI? Systematic Analysis of 32K AI Model Cards

Weixin Liang*, Nazneen Rajani*, Xinyu Yang*, Ezinwanne Ozoani, Eric Wu, Yiqun Chen, Daniel Scott Smith, James Zou

Nature Machine Intelligence 2024

[PDF], [Code]

• We conduct a comprehensive analysis of 32,111 AI model documentations on Hugging Face, which provides a systematic assessment of community norms and practices around model documentation through large-scale data science and linguistic analysis.
• Our findings reveal that while most popular models have model cards, they often vary in detail. Sections on environmental impact, limitations, and evaluation are frequently incomplete, whereas training details are more consistently provided. We also analyze the content of each section to characterize practitioners’ priorities.

Can large language models provide useful feedback on research papers? A large-scale empirical analysis

Weixin Liang*, Yuhui Zhang*, Hancheng Cao*, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel A. McFarland, James Zou

NEJM AI 2024

[PDF], [Twitter], [Code]

• We created an automated pipeline using GPT-4 to provide comments on the full PDFs of scientific papers.
• Our results suggest that LLM and human feedback can complement each other. While human expert review is and should continue to be the foundation of rigorous scientific process, LLM feedback could benefit researchers, especially when timely expert feedback is not available and in earlier stages of manuscript preparation before peer-review.

Reconnecting the Broken Civilization: Patchwork Integration of Fragments from Ancient Manuscripts

Yuqing Zhang*, Zhou Fang*, Xinyu Yang*, Shengyu Zhang, Baoyi He, Huaiyong Dou, Junchi Yan, Yongquan Zhang, Fei Wu

ACM MM 2023 (Oral)

[PDF]

• We developed a multimodal pipeline for Dunhuang manuscript fragments reconstruction, leveraging text-based localization and a self-supervised contour matching framework, accompanied by a global reconstruction process. Our empirical evaluations reveal that this pipeline exhibits a remarkable success rate in fragment assembly.

Accuracy on the Curve: On the Nonlinear Correlation of ML Performance Between Data Subpopulation

Weixin Liang*, Yining Mao*, Yongchan Kwon*, Xinyu Yang, James Zou

ICML 2023

[PDF], [Website], [Video], [Code]

• We show that there is a “moon shape” correlation (parabolic uptrend curve) between the test performance on the majority subpopulation and the minority subpopulation. This nonlinear correlations hold across model architectures, training settings, datasets, and the imbalance between subpopulations.

MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts

Weixin Liang*, Xinyu Yang*, James Zou

Contributed Talk at ICML 2022 Workshop on Shift happens: Crowdsourcing metrics and test datasets beyond ImageNet

[PDF], [Website], [Video], [Code]

• MetaShift introduces a collection of >10K sets of images with annotated contexts. It enables evaluating how ML works in different contexts (e.g. indoor cat vs outdoor cat).
• We provided a distance score that measures the amount of distribution shift between any two of the data sets.
• We presented methods to match labels to ImageNet hierarchy via WordNet ID and construct classification tasks over MetaShift to enable evaluating off-the-shelf ImageNet models.