Generative Persuasion — Derek Curry

Large language models can easily be weaponized as instruments of psychological manipulation, capable of generating hyper-personalized disinformation at scale. Generative Persuasion is a hands-on simulation and evolving workshop where participants generate their own tailored propaganda: political ads, conspiracy theories, social media posts, using a large language model conditioned to produce disinformation micro-targeted to a specified personality profile and political orientation. The system runs on a locally hosted, fully offline model, so that nothing it generates ever reaches the internet, and no output can be weaponized beyond the simulation itself. The application is staged under the guise of a fictitious startup, Psybernetica — a thinly veiled reference to Cambridge Analytica and its use of five-factor personality profiling to microtarget political messaging in campaigns including the 2016 U.S. election and the Brexit referendum. Generative Persuasion doesn't speculate about a future threat — it demonstrates an existing reality.

Psybernetica and the Cambridge Analytica Legacy

Microtargeting using the five-factor model of personality (sometimes called psychographic profiling) has been a marketing tool for decades, but became far more effective once social media made behavioral data abundant. In 2018, it was revealed that the British consulting firm Cambridge Analytica had used a Facebook app to harvest personal data from more than 87 million users. The app was really a veiled five-factor personality quiz, used to find correlations between quiz responses and online behavior. The correlations were then applied to users who never took the quiz or agreed to share their data at all. Cambridge Analytica is credited with the success of both the 2016 Trump campaign and the Leave.EU Brexit campaign; in the wake of the scandal, its parent company filed for insolvency as former employees founded a new consulting firm, Emerdata Limited, accused of the same practices. Psybernetica is a thinly veiled reference to that legacy.

Targeting a Personality, Not a Person

In Generative Persuasion, participants can choose which five-factor model personality traits to target, along with a political orientation and content type. The system then generates social media posts, political advertisements, or conspiracy theories designed to appeal specifically to that profile. The generated disinformation is then displayed in real time as mock social media content, mirroring tools already documented in real-world disinformation campaigns.

An Existing Reality, Not a Prediction

Large language models are transformer-based systems capable of tasks ranging from summarization to highly believable disinformation. AI companies are fully aware of this risk: a 2021 workshop report involving Georgetown’s Center for Security and Emerging Technology, OpenAI, and the Stanford Internet Observatory concluded that LLMs will change influence operations, and found it “all but impossible to ensure that large language models are never used to generate disinformation” (p. 64). In October 2024, OpenAI disclosed that it had blocked 20 global campaigns using its models for cybercrime and disinformation — though these figures come from the company’s own detection systems, and reflect only the disinformation they were able to catch; a simple rewording that preserves the underlying message would likely go undetected.

We built Psybernetica’s targeting system independently, based on our own research into psychometric profiling and LLM capability. It was only afterward that we came across a RAND Corporation report — commissioned by the Pentagon’s Irregular Warfare Technical Support Directorate — recommending that the Department of Defense acquire exactly this kind of generative AI capability for its own influence operations. Generative Persuasion is a speculative present, not a forecast: a counterpoint to present technological exuberance, offered without waiting for a whistleblower or a scandal to prove the point.

References

Acquiring Generative Artificial Intelligence to Improve U.S. Department of Defense Influence Activities

Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations

Facebook Says Cambridge Analytica Harvested Data of Up to 87 Million Users