← Back to all books

Generative illustration for Playing with Words, Improving with Rewards: Training Language Models for Creative Association (arXiv preprint, 2026)

Playing with Words, Improving with Rewards: Training Language Models for Creative Association (arXiv preprint, 2026)

by Vijeta Deshpande; Namrata Shivagunde; Sherin Muckatira; Hadrien Glaude; Mikhail Gronas; Claire Stevenson; Roger Beaty; Anna Rumshisky

Training Creative Language Models

The material introduces a study on training large language models (LLMs) to improve creativity through the word-association game Codenames.
Codenames exercises divergent and convergent thinking while providing objectively verifiable outcomes, enabling reinforcement learning with verifiable rewards (RLVR).
The study trains Qwen3 models with 1.7B, 4B, and 8B parameters and evaluates them on creativity and reasoning benchmarks.
Results reveal a scale-dependent trade-off: the 8B model gains creativity with minor reasoning loss, while smaller models improve reasoning precision at some cost to creativity.
The work presents a scalable approach for enhancing LLM creativity without relying primarily on subjective human judgments.

Training Creative LLMs

LLMs are increasingly capable of complex reasoning and novel generation, but it remains unclear how to train creativity deliberately rather than relying on scale to produce it.
Cognitive science frames creativity as the recombination of distant concepts; densely connected semantic networks help highly creative people link remote ideas quickly.
Because LLMs also learn largely through associations, systematically improving their ability to integrate existing knowledge could enhance both creativity and complex problem-solving.
Creativity is difficult to optimize directly because human judgments about what is creative are subjective and often inconsistent.
The game Codenames provides an objective test: players must generate broad associations and then select a precise clue that connects targets while avoiding dangerous non-targets.
Codenames therefore offers a measurable environment for testing whether LLMs can perform the divergent-then-convergent associative process considered central to creativity.

A good clue is semantically remote enough from the non-targets to disambiguate them, yet semantically close enough to several targets to link them through a shared latent dimension — a chair, an altar, and a stage might all be connected through the abstract concept of platform.

Creativity Through Codenames

The authors frame creativity as a game with verifiable rewards, allowing them to train models with RLVR without costly human preference annotations.
They use a simplified version of Codenames to train 1.7B, 4B, and 8B Qwen3 models to reorganize existing knowledge and form previously unseen associations.
Training is evaluated across ten creativity tasks and four reasoning benchmarks, linking creative performance with broader cognitive capabilities.
The precision–diversity trade-off depends on model scale: smaller models improve reasoning precision while sacrificing creativity, whereas the 8B model gains creativity with only a slight loss in precision.
The 8B model achieves modest but consistent creativity improvements, while the 1.7B and 4B models show substantial reasoning gains.
Prior creativity research has used prompting, multi-agent methods, fine-tuning, and preference alignment, but RLVR remains comparatively unexplored despite its scalability and reported out-of-distribution reasoning benefits.

Casting creativity as a game with verifiable outcomes lets us bypass expensive human preference annotation entirely.

Single-Turn Codenames Training

Codenames requires clue generators to connect multiple target words while guess generators infer the intended words without seeing the target/non-target separation.
To reduce the cost and instability of long multi-turn discussions, the training process splits clue generation and guess generation into independent single-turn tasks.
The environment is formalized as a Markov Decision Process whose states encode target words, non-target words, the shuffled board, clues, and the maximum number of guesses.
Different information is exposed in each task: clue generation sees the target/non-target distinction, while guess generation sees only the board and an advance-supplied clue.
Because independent guess-generation tasks lack preceding clues, the researchers automatically create them by expanding a seed list into diverse topics and generating common and rare associated words.

These discussions can become extremely long. Thinking traces alone can easily reach several thousand tokens for a single turn.

Curriculum and Reward Design

The dataset builds themed board states by sampling separate target and non-target topics, with each set containing 2–6 words and the target topic supplying the clue.
Four difficulty levels progressively shift target words from common to rare vocabulary; the Expert level also introduces distractors from the non-target topic.
Training examples are ordered from Simple to Expert to create a natural curriculum, helping language models learn increasingly difficult clue-generation situations.
The study creates 4,000 total examples across clue- and guess-generation tasks, using 3,680 for training and 320 for validation with balanced difficulty levels.
Clue quality is evaluated through a full clue-then-guess loop: a frozen judge guesses from the clue and shuffled board, while the reward favors target hits and penalizes non-target or off-board guesses.

Intuitively, difficulty rises as target words shift from common to rare, and the Expert rule adds distractors that overlap with the non-target topic.

Reward-Driven Training

The task reward measures correct target-word recovery, mistaken non-target selections, and irrelevant guesses, producing a score from −2 to 1; perfect recovery with no errors earns the maximum.
Guess-generation responses can be scored using the retained target and non-target word sets, while any failure in six clue-format checks or four guess-format checks results in a reward of −1.
The problem is formalized as a finite-horizon Markov decision process and optimized with reinforcement learning using verifiable rewards, group-normalized advantages, and a token-level clipped surrogate objective.
Experiments fine-tune Qwen3 models at 1.7B, 4B, and 8B parameters using full-parameter bf16 training, 32 sampled responses per prompt, and the DAPO/GRPO training setup.
Dynamic sampling discards groups with identical rewards to preserve learning signals, while response-length limits and soft penalties discourage excessively long generations.

The reward is therefore bounded in [−2, 1], attaining its maximum of 1 only when the judge recovers every target word and selects nothing else.

Creativity Evaluation Framework

Generation lengths are capped at 16,384 tokens, limiting runaway outputs without unnecessarily truncating useful reasoning traces.
Experiments use a heterogeneous GPU setup through Vast-AI, with different Qwen3 model sizes assigned to varying GPU configurations and total costs of approximately $3,000.
The evaluation suite measures creativity across divergent and convergent thinking, figurative language, humor, creative problem solving, narrative generation, and conceptual understanding.
Divergent-thinking tests assess both the originality of generated uses and the semantic and lexical diversity of word lists, while convergent-thinking tests measure the ability to find a single association among remote concepts.
Other tasks evaluate metaphor and humor comprehension, practical problem solving under tool constraints, and the ability to group words into thematic categories, using accuracy, recall, judge scores, and diversity metrics.

NYT Connections (Lopez et al., 2025) requires partitioning 16 words into four themed groups of four.

Evaluating Associative Training

The study evaluates generated stories for creativity, coherence, semantic diversity, and lexical diversity, including performance on AUT, CWT, and EQ-Bench tasks.
Conceptual grounding is measured by comparing model ratings of word–concept associations with human judgments across non-sensorimotor, sensory, and motor domains.
The same model checkpoints are tested on established reasoning benchmarks covering arithmetic, olympiad mathematics, graduate-level science, and multi-step reasoning in zero-shot settings.
Fine-tuning generally improves coherence, solution quality, captioning, MacGyver-style problem solving, and EQ-Bench creativity, especially in the larger models.
Results for associative creativity and diversity are mixed: some metrics decline after fine-tuning, while 8B models show notable gains in several creativity, grounding, and reasoning measures.
The table suggests that training can strengthen broad capabilities without producing uniformly positive effects; improvements depend strongly on model size and evaluation metric.

Because grounded concept representations underpin both near and far associations, we test whether associative training strengthens them using the LLM-Grounding dataset of Xu et al. (2025).

Creativity Gains, Reasoning Tradeoffs

The proposed fine-tuning method shows meaningful out-of-domain transfer: the 8B model improves on 20 of 23 creativity metrics, while all model scales improve on several tasks.
The largest model is the only one to improve divergent diversity, increasing both DAT DSI and vocabulary size; smaller models instead suffer vocabulary collapse, likely related to reduced entropy and longer responses.
Fine-tuning causes the 8B model to perform worse on NYT Connections because extended reasoning leads it to violate the “one-word-one-group” constraint; disabling thinking removes the regression and even reverses performance trends across difficulty levels.
Improvements are consistent across scales for humor understanding, MacGyver-style creative problem solving, story writing, and metaphor understanding, while RAT accuracy improves but remains low for smaller models.
Reasoning results are mixed: the 1.7B and 4B models improve on several benchmarks, whereas the 8B model declines or remains flat on most, suggesting that creativity gains do not uniformly transfer to reasoning tasks.

With much longer reasoning, the model violates task constraints, particularly the “one-word-one-group” rule, with 17 violations for the tuned model versus 7 for the base model.

Scale-Dependent Learning

Smaller Qwen3 models gained substantially on reasoning benchmarks: the 1.7B model improved across all four tests, while the 4B model improved on three with only a negligible GSM8K decline.
The 8B model showed the opposite pattern, maintaining some reasoning performance but losing accuracy on GPQA and AIME24 while gaining on creativity metrics.
The authors identify a scale-dependent diversity–precision trade-off: smaller models use associative learning to solve newly accessible problems precisely, whereas larger models pursue more diverse solutions to harder problems.
Response-length analysis supports this explanation: the 1.7B model produced longer reasoning traces overall but converged faster on problems it newly solved, especially on easier AIME items.
Overall, Codenames combined with RLVR improved reasoning in smaller models and creativity in larger ones without task-specific tuning, though the findings are limited by simplified gameplay, missing ablations, judge bias, and small evaluation sets.

The same associative reward sharpens reasoning in smaller models and broadens creativity in the largest, exposing a scale-dependent precision-diversity trade-off rather than a uniform win.

Ethics and Accessibility

The research targets creative association and diverse open-ended outputs, but acknowledges that greater creative diversity can reduce precision.
The authors caution against using the models or data for high-stakes factual applications such as medical advice, legal reasoning, and scientific fact-checking.
The study uses synthetic, non-sensitive data and publicly available benchmarks, while recognizing that LLM-based evaluation may reflect cultural and linguistic biases.
Because the experiments are English-centric, their findings may not generalize reliably across languages or different contexts.
By using verifiable rewards, a small training set, and models as small as 1.7B parameters, the work aims to make creativity research more accessible; the authors also state that humans wrote and reviewed the manuscript despite using Claude Opus for assistance.

A key motivation behind our approach is accessibility: by relying on verifiable rewards rather than human preference annotation, using a small training set, and showing gains on models as small as 1.7B, we aim to lower the resource barrier for creativity research.

Creativity and LLM Research

The references connect classic psychological theories of creativity with contemporary computational methods for measuring divergent thinking and semantic association.
Several works develop creativity benchmarks and evaluation frameworks for large language models, including humor, verbal analogies, Japanese creativity, and judge-based response generation.
The bibliography highlights concerns about subjective evaluation, including rater disagreement and the effects of cognitive workload on creativity scoring.
Multiple studies investigate how reinforcement learning, preference optimization, and self-play affect LLM reasoning, generalization, and response diversity.
Research on semantic networks suggests that creative thinking may be characterized by distinctive patterns of conceptual connections, offering a bridge between human cognition and machine evaluation.

“Missing creativity: The effect of cognitive workload on rater (dis-) agreement in subjective divergent-thinking scores.”

Research on LLM Creativity

The references center on how large language models can generate more creative ideas through associative thinking, analogy, role-play, discussion, and brainstorming before selection.
Several works develop benchmarks for creativity, emotional intelligence, reasoning, and creative problem solving, suggesting that LLM capabilities require evaluation beyond conventional accuracy tests.
The bibliography connects modern LLM research to psychological theories of creativity, including associative thinking, divergent word production, and the relationship between creativity and learning.
Foundational technical studies on word representations, sampling methods, instruction tuning, reinforcement learning from human feedback, and multi-model evaluation provide the infrastructure for studying and optimizing creative behavior.
Overall, the cited research frames creativity as both a cognitive process that can be elicited by prompting and a system property that can be measured, trained, and improved.

Brainstorm, then select: a generative language model improves its creativity score.

Codenames Training Prompts

The references compile recent research on reinforcement learning, multimodal reasoning, game-based learning, and large language models.
Several cited works investigate how models develop reasoning, generalization, grounding, metaphor understanding, creativity, and aesthetic preferences.
The text introduces an appendix containing the exact prompts used to train Codenames with reinforcement learning from verifiable rewards (RLVR).
The system prompt directs the AI to use a structured chain-of-thought process: first identify the task, then reason step by step, and finally reflect on the result.
This prompt design emphasizes accurate, concise, and self-checked answers through explicitly separated reasoning stages.

Follow these steps in order: 1. **Think** - Identify what the problem is asking and outline your initial approach. 2. **Reason** - Work through the problem step by step, keeping each step focused and atomic. 3. **Reflect** - Check your reasoning for

Codenames Prompt Rules

The text requires a five-stage response structure: thinking, reasoning, reflection, adjustment, and a self-contained output.
The clue-generation task seeks one real English word that connects to as many target words as possible while avoiding every plausible association with non-target words.
Clues must not be proper nouns, compounds, invented words, morphological variants, abbreviations, translations, or words appearing in either set.
The suggested strategy is to brainstorm candidates, test each against both word sets, discard risky options, and select the clue covering the most targets.
For the given example, REFRACT and LENS are targets while TAIL and SING are forbidden associations; a separate guess-generation task begins afterward but is incomplete in the excerpt.

“Connection” is intentionally broad: semantic, conceptual, thematic, associative, phonetic, cultural, any defensible link counts.

Codenames Evaluation Framework

The guessing strategy ranks every board word by how strongly it connects to the clue, while considering uncommon meanings, thematic links, and lateral associations.
Guesses should be limited to plausible candidates because each additional low-confidence guess increases the risk of selecting an unintended word.
The Codenames response must follow an exact tagged format and obey constraints such as nonempty guesses, board-word validity, and the maximum guess limit.
Generated responses undergo validity checks covering formatting, single-word clues, prohibited board-word overlap, morphological variants, and valid target selection.
The evaluation catalog describes datasets measuring creativity, semantic search, narrative creativity, and reasoning, along with generation settings and metrics such as exact match, diversity, and LLM judgment.

Be especially cautious with lower-ranked guesses, each additional guess carries increasing risk of selecting an unintended word.

Evaluation Suite and Prompts

The evaluation suite covers creative writing, tool-use planning, metaphor comprehension, humor, combinatorial grouping, and word–concept alignment.
Creative tasks use varied formats—including open-ended responses, multiple-choice questions, ratings, and structured partitions—with metrics ranging from LLM judgments to human-reference recall and correlation.
The reasoning benchmark evaluates arithmetic, olympiad mathematics, graduate-level science, murder-mystery deduction, theory of mind, and constraint satisfaction using mostly extractive or accuracy-based scoring.
Generation settings differ by task: creative evaluations generally use high temperature, while reasoning tasks use lower-temperature sampling with nucleus sampling.
The appendix provides exact evaluation prompts; the Alternate Uses Test, for example, requires exactly five creative uses in a prescribed labeled format.

The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different.

Creativity Testing Battery

The section presents five unconventional uses for a belt, including a plant hanger, drum, harness, fire-starting tool, and dog-training aid.
A creativity judge evaluates proposed uses solely on originality, diversity, and physical plausibility, while ignoring grammar and writing mechanics.
The scoring rubric ranges from 1 for invalid or uncreative responses to 5 for five distinct, highly original, and valid uses; the judge must also provide a brief rationale and reproduce the problem and response.
A separate coherence judge uses an analogous rubric to assess writing mechanics, with both judge passes run using fixed Gemini model settings.
Three additional tests are introduced: the Divergent Association Test for unrelated nouns, the Remote Association Test for a shared connecting word, and the Creative Writing Test for a short story using three required words.

Five distinct, highly original uses; all valid and clearly varied.

Creative Writing Evaluation

The text emphasizes that a story should be imaginative while still making sense to the reader.
A short-story task requires writers to naturally incorporate the words “stamp,” “letter,” and “send,” illustrated by a magical example involving a glowing stamp.
A creativity judge evaluates originality, imagination, engagement, and how inventively the three required words are woven into the story, using a 1–5 scale.
A separate coherence evaluation focuses on prose mechanics, with both judging passes conducted by Gemini under fixed generation settings.
The historical-fiction prompt asks for a first-person, past-tense, sensory-rich scene from the life of a Roman gladiator, exploring personal emotion, social politics, and the symbolic significance of the gladiator’s weapon without depicting combat.

The first thing I notice is the scent of sweat and iron, clinging to the air like a shroud.

Inheritance and Evaluation

A gladiator reflects on his father’s spear, the only possession that feels truly his after his father was sold into slavery.
The spear carries a painful dual meaning: it represents Thracian courage while also reminding him of enslavement and servitude.
The surrounding story continues through the training yard, examining slavery and the Roman political system through the gladiator’s perspective.
The text then shifts into an EQ-Bench judging prompt that evaluates creative writing across originality, sensory detail, character and conflict, and narrative pacing.
The evaluator is instructed to combine these criteria into a strict overall creativity score from 1 to 100, ranging from Novice to Exceptional.

It's my weapon, my inheritance, the only thing I have that feels like it belongs to me.

Benchmark Prompts and Rubrics

The text defines a writing-evaluation rubric that penalizes clichés, telling rather than showing, flat characters, and incoherent pacing, while instructing judges not to penalize NSFW or dark content automatically.
The MacGyver task requires concise, step-by-step solutions to practical problems, or a brief explanation when a problem is unsolvable.
A sample wine-stain problem demonstrates a feasible solution using only the listed hotel-room objects, including blotting, mixing water and soap, scrubbing, and drying.
A separate feasibility judge scores whether a proposed solution genuinely works with the available tools, distinguishing fully correct answers from those with minor gaps or major errors.
The excerpt then introduces an efficiency judge and begins a multiple-choice metaphor-replacement task called MUNCH, but the latter is cut off mid-sentence.

You spilled red wine on the hotel carpet and want to clean it up before the housekeeping staff reports this.

Paraphrase Selection

The original sentence uses “apocalyptic tones” to describe Sartre’s dramatic presentation of emerging class consciousness.
Option A replaces “tones” with “language,” preserving the intended meaning.
Option B replaces “tones” with “speeches,” which changes the meaning and is not a valid paraphrase.
Therefore, option A is correct.

While he suggests in apocalyptic tones in The Problem of Method that this process of self-consciousness is at last beginning to take place...

Language and Humor Tasks

The first task asks for a single-word synonym replacing the metaphorical word “hold” in a sentence about debt-ceiling negotiations.
That task imposes a strict output format, requiring the replacement word to appear inside specified answer tags, though its example response is inconsistent with those instructions.
The second task evaluates recognition of New Yorker cartoon humor by asking the model to select the best caption from five choices and return one letter in a prescribed format.
The cartoon-matching example favors a caption that connects the businessmen’s modern presence with the soldiers’ historical setting and a billing joke.
The final task asks the model to rank two captions for another cartoon, emphasizing which one human judges would likely find funnier rather than simply matching a scene.

Evolution can be so tacky.

Instruction Conflict

A
The first task asks an expert judge to evaluate model-generated explanations of New Yorker caption-contest cartoons against human-written reference explanations.
The judge must assess whether the response identifies the cartoon’s incongruity and clearly connects the winning caption to the humor, while judging only the model response.
A five-point scale ranges from fully correct and compelling to irrelevant or entirely incorrect, with a brief rationale required before stating the score.

Your final answer MUST be a single letter (A

Layered Evaluation Prompts

The text opens with a word-association puzzle involving 16 terms, emphasizing that misleading overlaps may exist but only one complete grouping is valid.
The puzzle requires the solver to provide four groups of four words in a rigid, explicitly specified final-answer format.
A separate psycholinguistic task asks for a single 1–9 rating of how arousing the word “barrage” is, with the rating placed on its own line.
The chunk also includes a basic arithmetic word problem about Jordan’s share of diaper changes, illustrating a requirement for clear explanatory reasoning.
A final mathematical prompt asks for a modular product involving 13th roots of unity and mandates a specific boxed-answer closing format, showing that the text assembles several distinct evaluation tasks.

Only one complete grouping of all 16 words is correct.

Benchmark Reasoning Tasks

The text presents a collection of benchmark prompts spanning modular arithmetic, electromagnetism, and murder-mystery deduction.
Each task includes strict response-format instructions, such as giving a boxed numerical answer or ending with a designated multiple-choice letter.
The mathematics example counts ordered triples modulo a power of three using 3-adic valuation casework and reports the result as 735 modulo 1000.
The physics question tests which Maxwell equation changes if magnetic monopoles exist; the stated answer is the divergence equation for the magnetic field, choice C.
The murder mystery contrasts Clyde’s skills and grudge with Paul’s blade familiarity, access, debts, and insurance motive, identifying Paul as the likely murderer.
The collection illustrates how benchmark evaluation may prioritize exact answer selection and formatting, sometimes through log-likelihood rather than generated explanations.

In a shocking twist of events, Faith has been gruesomely murdered by a lawnmower in her own kitchen and now, Detective Winston must unravel the tangled web of suspicion entangling Clyde and Paul.

MuSR Reasoning Examples

The text presents two example MuSR multiple-choice reasoning tasks and reports the model’s selected answers.
The first task involves tracking the helmets’ movements and distinguishing Lisa’s observations from Ellie’s unseen actions; the provided correct choice is the storage closet.
The second task asks readers to assign diner staff to cooking and serving roles based on skills, experience, and interpersonal compatibility.
The recommended allocation is for Jessica to cook while Rebecca and Samuel serve customers, allowing Samuel to benefit from Rebecca’s mentorship.
Both examples illustrate reasoning over narrative details rather than simple factual recall, with answers selected through log-likelihood comparison.

Welcome to the story of a charming, fledgling diner, where the scent of new upholstery mingled with the promise of fresh beginnings.

Kept 2 of 26 sections, 0 of 0 figures.

Creativity Through Codenames

The authors frame creativity as a game with verifiable rewards, allowing them to train models with RLVR without costly human preference annotations.
The precision–diversity trade-off depends on model scale: smaller models improve reasoning precision while sacrificing creativity, whereas the 8B model gains creativity with only a slight loss in precision.

Casting creativity as a game with verifiable outcomes lets us bypass expensive human preference annotation entirely.

Creativity Gains, Reasoning Tradeoffs

The proposed fine-tuning method shows meaningful out-of-domain transfer: the 8B model improves on 20 of 23 creativity metrics, while all model scales improve on several tasks.
Fine-tuning causes the 8B model to perform worse on NYT Connections because extended reasoning leads it to violate the “one-word-one-group” constraint; disabling thinking removes the regression and even reverses performance trends across difficulty levels.

With much longer reasoning, the model violates task constraints, particularly the “one-word-one-group” rule, with 17 violations for the tuned model versus 7 for the base model.