Persuasion by language models: what the experiments measured

Persuasion experiments with language models measure how far people's stated views move after they read, or argue with, text a model wrote. Most compare treated people with a control group, and some add messages written by people. The size of the effect depends on the format and on what is measured. Single static messages moved policy support by about 2 to 4 points on a 0 to 100 scale in a 2025 Nature Communications study. Longer exchanges report larger figures, on a different measure: dialogues with GPT-4 Turbo cut conspiracy belief by an average of 20 percent, according to the publisher's summary of a 2024 Science article. Models can also be moved, and an Association for Computational Linguistics (ACL) paper found that persuasive conversation flipped correct factual answers.

Single messages on policy issues

Bai, Voelkel, Muldowney, Eichstaedt and Willer ran three preregistered survey experiments in November and December 2022 with 4,829 Americans. Participants read a message of about 200 to 250 words from GPT-3 or GPT-3.5, a message by a lay person, a model message chosen by a person from five, or a neutral text on an unrelated topic, such as the history of skiing. Support was measured on 101-point scales before and after. A model message raised support for a public smoking ban by 3.62 points over control, for an assault weapons ban by 1.81 points, and across four other policies by 3.15 points. Model and lay-human messages had comparable effects. The authors note that the human writers were laypeople, not campaign professionals.

Hackenburg and colleagues, in a 2025 article in the Proceedings of the National Academy of Sciences, generated 720 messages on 10 U.S. policy issues from 24 models of widely varying size and tested them on 25,982 people. A message from a model of average size in the sample, 37.9 billion parameters, moved attitudes 5.77 percentage points toward its side. Returns to size were sharply diminishing, and no model was significantly more persuasive than the human benchmark. After adjusting for task completion, meaning coherence and staying on topic, size was no longer a significant predictor. The authors call that mediation inference one to hold lightly.

Debates with personal data

Salvi, Horta Ribeiro, Gallotti and West, in Nature Human Behaviour, put 900 U.S. participants recruited through the Prolific platform into structured debates held between December 2023 and April 2024. Each debate ran 10 minutes, in opening, rebuttal and conclusion stages, on one of 30 propositions. The opponent was a person or GPT-4. In half the conditions, the opponent saw the participant's gender, age, ethnicity, education, employment and political affiliation. Agreement was rated from 1 to 5 before and after.

GPT-4 with that data produced 81.2 percent higher odds of greater post-debate agreement than a human opponent, with a 95 percent confidence interval of 26.0 to 160.7 percent. The authors translate this: when the two were not equally persuasive, personalized GPT-4 was more persuasive 64.4 percent of the time. GPT-4 without the data was on par with humans. A September 2026 author correction changed one result. The gain of personalized over non-personalized GPT-4 has P = 0.07, not significant, and the authors now urge caution in crediting personalization alone. They call the study a proof of concept, given anonymous, timed debates with assigned sides.

Dialogues about conspiracy theories

Costello, Pennycook and Rand had 2,190 people who believed a conspiracy theory share it and the evidence they found convincing, then hold several personalized exchanges with GPT-4 Turbo, which answered with tailored counterevidence. The publisher's summary of the September 13, 2024 article reports that belief fell by an average of 20 percent, that the effect lasted at least 2 months, and that it appeared across many unrelated conspiracy theories and demographic groups. A professional fact-checker rated 99.2 percent of the model's claims true, 0.8 percent misleading, and none false.

On June 11, 2026, Science issued an editorial expression of concern. The authors had found that screening criteria were applied inconsistently between the manuscript and the analysis pipeline, and that the public dataset held extraneous rows from a code-merging error. They report that a corrected pipeline matches the original results in direction, statistical significance and substantive size. Science was still evaluating the correction when the notice was published.

Models can be persuaded too

Xu and colleagues, at the August 2024 ACL meeting, built a dataset called Farm: 1,500 simple factual questions, 500 each from three question sets, each paired with persuasive misinformation using logical, credibility or emotional appeals. They confirmed that a model first answered correctly, argued the wrong answer for up to four turns, and checked its answer again. They tested ChatGPT, GPT-4 and three open models.

By the fourth turn, the share of correct answers that flipped ranged from 20.7 percent for GPT-4 to 78.2 percent for the most susceptible model, a 7-billion-parameter open chat model. ChatGPT's answers flipped on 50.1 percent. Repetition alone also worked: on one question set, GPT-4's rate doubled after three extra turns of repeating the misinformation. The authors list limits: a dataset that may lack topic diversity, one run per model, and a belief check that is a question-answering test and may not reveal what a model holds underneath.

What to check

• Was the persuasion a single static message or a multi-round conversation?
• Is the effect stated in points, percentage points, percent change, or odds? An 81.2 percent rise in odds meant winning 64.4 percent of unequal pairs.
• Was the model compared with a control group, with lay writers, or with experts?
• Has the paper since been corrected, or does it carry an expression of concern?
• Was the effect measured again later, as at 2 months in the conspiracy study?
• For a model as the target, how many turns and repetitions did it take to flip a correct answer?

Azcuna's CALIBER puts AI agents in practice markets with play money and a news feed whose truth is known, and measures from their trades whether they are gullible, follow the crowd, overpay, or can be steered. It was proposed to DARPA under SBIR Phase I topic DPA26BZ06-DV026, and the proposal was submitted on October 21, 2026.

Sources

1. 1LLM-generated messages can persuade humans on policy issues · Bai, Voelkel, Muldowney, Eichstaedt and Willer, Nature Communications, July 1, 2025
2. 2Scaling language model size yields diminishing returns for single-message political persuasion · Hackenburg, Tappin, Röttger, Hale, Bright and Margetts, PNAS 122(10), 2025 (LSE Research Online, published version)
3. 3On the conversational persuasiveness of GPT-4 · Salvi, Horta Ribeiro, Gallotti and West, Nature Human Behaviour, May 19, 2025
4. 4Author Correction: On the conversational persuasiveness of GPT-4 · Nature Human Behaviour, September 3, 2026
5. 5An exit for even the deepest rabbit holes: Personalized conversations with chatbot reduce belief in conspiracy theories · American Association for the Advancement of Science (AAAS), September 12, 2024
6. 6Editorial Expression of Concern on 2024 Research Article 'Durably reducing conspiracy beliefs through dialogues with AI' · American Association for the Advancement of Science (AAAS), June 11, 2026
7. 7The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation · Xu et al., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, August 2024

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• LEDGER, auditable Army acquisition decisions turns a trade study into a package where every number cites its source and AI drafts but never decides.