Model cards and datasheets, explained

A model card is a short document that accompanies a trained machine learning model and reports what it is for, how it was evaluated, and where it falls short. A datasheet does the same for a dataset. Mitchell and colleagues proposed model cards at the 2019 Conference on Fairness, Accountability, and Transparency (FAT*), as records of one to two pages. Gebru and colleagues proposed datasheets for datasets, published in Communications of the ACM in December 2021. Both papers say the documents are only as good as the people who write them, so a buyer reads one as a set of claims to check.

The nine sections of a model card

Mitchell's paper lists nine sections:

Section	What it covers
Model details	Developer, date, version, type, training approach, license, contact
Intended use	Primary uses, primary users, out-of-scope uses
Factors	Groups, instruments, and environments where performance may vary
Metrics	Performance measures, decision thresholds, how variation is estimated
Evaluation data	Datasets used, why they were chosen, preprocessing
Training data	Ideally mirrors evaluation data; the paper says this may not be possible
Quantitative analyses	Results broken down by each factor and by combinations of factors
Ethical considerations	Sensitive data, risks, mitigations, fraught use cases
Caveats and recommendations	Further testing needed, groups missing from the evaluation

Several prompts are pointed. The out-of-scope section is inspired by warning labels on food and toys. The metrics section asks which decision thresholds were used and why. Evaluation data should cover both typical use and challenging cases. The paper's sample card for a smile detector reports 95 percent confidence intervals from bootstrap resampling, with all metrics at a 0.5 threshold. The model details section is not, the authors write, a requirement to reveal proprietary training techniques.

The seven question groups of a datasheet

Gebru groups the questions by stage of a dataset's life: motivation; composition; collection process; preprocessing, cleaning, and labeling; uses; distribution; and maintenance. For creators, the aim is careful reflection on how the dataset was made. For consumers, the aim is enough information to decide whether to use it.

The questions came out of roughly two years of revision. The authors tested them by writing datasheets for two public datasets, gave them to product teams at two major U.S. technology companies, and had lawyers review them. They added a section on uses, deleted a separate section on legal and ethical considerations, and replaced questions about regulatory compliance with factual ones. Product teams answered legal and ethical questions more readily when the questions sat inside the relevant lifecycle stage.

Composition questions include whether the dataset recommends training, validation, and test splits, and whether it contains errors, noise, or redundancies. The paper says the questions are not prescriptive, that writing a datasheet is not meant to be automated, and that a dataset updated now and then should ship each version with an updated datasheet.

What the authors say these documents cannot do

Mitchell's paper states that the usefulness and accuracy of a model card rely on the integrity of its creators. The authors consider it unlikely, at least in the near term, that cards could be standardized enough to prevent misleading results, intended or not. They present cards as one transparency tool among several, including third-party auditing and adversarial testing.

Gebru's paper says datasheets do not fully solve the problem of bias or other risks, because creators cannot anticipate every use of a dataset. The authors' own example datasheet answers some questions "Unknown to the authors of the datasheet," because they did not create that dataset and its documentation did not say. Writing a datasheet also takes time, and the authors expect organizations to change workflows and incentives to make room for it.

What published model cards contain

Liang and colleagues analyzed model cards on Hugging Face, a platform for distributing AI models. As of October 1, 2022, 32,111 of 74,970 model repositories had a model card; the paper gives that share as 44.2 percent in one place and 42.8 percent in another. Those models accounted for 90.5 percent of download traffic.

Sections were filled in unevenly. Training appeared in 74.3 percent of cards, limitations in 17.4 percent, evaluation in 15.4 percent, citation in 14.4 percent, and environmental impact in 2.0 percent. The 100 most downloaded models did better on limitations, at 39.0 percent, and evaluation, at 47.0 percent.

The authors then added detailed cards to 42 popular models that had none or only sparse ones. In one batch, weekly downloads rose 29.0 percent while a control group showed no significant change. A second batch gave less clear results, and the authors call the sample relatively small.

What a buyer should expect

The National Institute of Standards and Technology (NIST) Generative AI Profile, NIST AI 600-1 of July 2024, tells organizations to review transparency artifacts such as system cards and model cards for third-party models (action MG-3.1-005). It also says to re-assess risks after fine-tuning or retrieval-augmented generation, and for third-party models used in ways not evaluated in initial testing (MG-3.1-003). A card written at release does not describe those later changes.

The Office of Management and Budget memo M-25-22, dated April 3, 2025, encourages agencies to obtain documentation that supports transparency and explainability. For high-impact uses, agencies must tell vendors what documentation will be required. The memo does not apply to AI acquired for use as a component of a national security system. For contracts, NIST AI 600-1 action GV-6.1-006 is to include clauses that allow an organization to evaluate third-party generative AI processes and standards.

What to ask

• Does the card give the exact model version and date, and does that match the model delivered?
• Does it list out-of-scope uses, and does the intended use cover the buyer's task?
• Which evaluation data were used, and do they resemble the buyer's documents?
• Are decision thresholds and confidence intervals stated with the results?
• Is the limitations section filled in, or missing as in most cards in Liang's study?
• Has the model been re-assessed since fine-tuning or connection to new documents, as NIST action MG-3.1-003 suggests?

Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.

Sources

1. 1Model Cards for Model Reporting · arXiv (Mitchell et al.; FAT* '19, January 2019)
2. 2Datasheets for Datasets · arXiv (Gebru et al., version 8, December 2021; published in Communications of the ACM, December 2021)
3. 3What's documented in AI? Systematic Analysis of 32K AI Model Cards · arXiv (Liang et al., February 2024)
4. 4Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 · National Institute of Standards and Technology (July 2024)
5. 5M-25-22, Driving Efficient Acquisition of Artificial Intelligence in Government · Office of Management and Budget (April 3, 2025)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.