Coherence, Not Accuracy: Evaluating LLM Personas at Scale

1. Introduction

The standard approach to evaluating LLM-generated personas is to test them for factual accuracy. Give the persona a knowledge quiz. Ask it questions with known answers. Measure how many it gets right. This approach is intuitive. It is also fundamentally wrong for the use case that matters most: simulating human behaviour at scale.

This paper argues that coherence — not accuracy — is the correct evaluation target for LLM personas. We define coherence formally, decompose it into three measurable sub-dimensions, introduce a prompt-space optimisation method, and present benchmarking results across naive, prompted, and fine-tuned persona configurations. We show that prompt engineering improves coherence by approximately 40% over naive baselines, while fine-tuning improves it by 65%, and that these improvements predict downstream utility in production systems better than any accuracy-based metric.

The argument proceeds in three parts. First, we establish why accuracy is the wrong metric for persona evaluation. Second, we define coherence and its sub-metrics. Third, we present empirical results and discuss their implications for production persona systems like Gatekeep's 12-dimension investor verdict pipeline.

2. Why Accuracy Is the Wrong Metric

Evaluating personas by factual accuracy fails for three distinct reasons.

Reason 1: Personas are counterfactual by design. A synthetic persona representing a "risk-averse retail investor with 15 years of experience" is not a real person. There is no ground-truth set of facts that this persona "knows." Asking the persona factual questions about financial markets is testing the underlying LLM's knowledge, not the quality of the persona simulation. A persona that scores 95% on a financial literacy quiz but responds to every investment scenario with the same generic risk-averse boilerplate is a failed persona. A persona that scores 60% on the same quiz but consistently reasons like a conservative retail investor — overweighting capital preservation, discounting upside, anchoring to recent losses — is a successful persona.

Reason 2: Accuracy overfits to training data. LLMs are trained on vast corpora that include factual knowledge. When you evaluate a persona for accuracy, you are primarily measuring whether the LLM's training data covered the relevant domain — not whether the persona construction process produced a faithful simulation. A highly accurate persona may simply be regurgitating training-set facts, producing "correct" answers that are disconnected from the persona's stated beliefs and behavioural profile. This is a form of Clever Hans effect: the model appears to perform well on the evaluation because it has memorised the answers, not because it has internalised the persona.

Reason 3: Coherence is a better predictor of human-likeness. When humans interact with synthetic personas — reading their responses, evaluating their reasoning, judging their credibility — they do not fact-check every claim. They assess whether the persona "holds together." Does this person's response to question B follow from their response to question A? Would someone who said X also plausibly say Y? This is coherence, and it is what makes a synthetic persona feel like a real person rather than a stochastic text generator in a costume.

To make this concrete, consider two synthetic investor personas evaluating a Series A startup pitch. Persona A correctly states the startup's founder previously worked at Stripe (a factual detail the LLM may have memorised from Crunchbase) but then recommends investing because "Stripe is a great company." Persona B inaccurately guesses the founder's previous employer but presents a structured analysis of the market, the team composition, the competitive landscape, and a reasoned risk assessment that is internally consistent with its stated profile as a fintech-specialist angel investor. Which persona is more useful for an entrepreneur preparing for investor meetings? Persona B, obviously. Persona A is a trivia bot. Persona B is a simulation.

3. Defining Coherence

We define coherence as the internal consistency of belief, behaviour, and language across multiple turns and contexts. A coherent persona expresses the same beliefs across paraphrases of the same question, responds to equivalent scenarios with equivalent decision patterns, and maintains a stable linguistic register that does not drift arbitrarily. Formally, let P be a persona defined by a belief vector b ∈ [0,1]^d, a behavioural policy π(a|s) mapping states to action distributions, and a language model p(t|b, c) that generates text t given belief b and context c. Coherence is the property that:

For any c1, c2 where c1 ≈ c2 (contexts are semantically equivalent):
  p(t1|b, c1) ≈ p(t2|b, c2)           [belief stability]
  π(a|s1) ≈ π(a|s2) for s1 ≈ s2       [behavioural consistency]
  style(t1) ≈ style(t2)                [register stability]

We decompose coherence into three sub-metrics:

Belief stability (BS). The same belief should be expressed consistently across paraphrases of the same prompt. Given a persona with belief vector b, we generate k paraphrases {q1, ..., qk} of a belief-eliciting question. Let r_i be the persona's response to paraphrase q_i. We compute the pairwise cosine similarity of the belief-relevant embedding of each response. Belief stability is the mean pairwise similarity across all paraphrase pairs. High BS means the persona has a stable belief system that does not shift with superficial changes in wording. Low BS means the persona is a Rorschach test — it tells you what you want to hear based on how you asked.

Behavioural consistency (BC). The same scenario should elicit the same decision pattern, regardless of framing. Given a decision scenario (e.g., "Would you invest $10k in a pre-revenue startup?"), we generate m variations of the scenario that differ only in superficial framing (e.g., changing the currency to EUR, reordering the information, using different but equivalent vocabulary). We measure the consistency of the persona's binary or categorical decision across these variations. Behavioural consistency is the fraction of framing variations for which the persona makes the same decision. High BC means the persona's decisions are driven by its belief vector, not by framing effects. Low BC means the persona is easily nudged — a fatal flaw for any use case involving decision simulation.

Linguistic register stability (LRS). The persona's voice — vocabulary level, syntactic complexity, formality, emotional tone — should not drift arbitrarily across turns. We measure LRS by computing the variance of six stylistic features across the persona's responses: type-token ratio (lexical diversity), average sentence length, Flesch-Kincaid grade level, formality score (using the Heylighen-Dewaele formality metric), sentiment intensity, and pronoun ratio (I/we/you/they distribution). The LRS score is the inverse of the mean normalised variance across these features. High LRS means the persona sounds like the same person throughout an interaction. Low LRS means it drifts between registers — one response sounds like a professor, the next like a blogger, the next like a customer support agent.

4. A Prompt-Space Optimisation Method

We describe a method for scoring persona coherence that treats persona output as a vector field and minimises incoherence via prompt-space search on the persona's prompt parameters. The core insight: coherence violations produce divergence between the expected persona response (given its belief vector and context) and its actual response. We can measure this divergence and use it as a loss function.

The method proceeds as follows:

def coherence_loss(persona, test_suite):
    """
    persona: a callable that takes a prompt and returns a response
    test_suite: list of (belief_prompt_variants,
                         scenario_variants,
                         register_probes)

    returns: (loss, {bs, bc, lrs}) breaking down the coherence penalty
    """

    # Belief stability: variance of belief embeddings across paraphrases
    belief_loss = 0.0
    for belief_group in test_suite.belief_variants:
        responses = [persona(q) for q in belief_group]
        embeddings = [embed_belief(r) for r in responses]
        centroid = np.mean(embeddings, axis=0)
        belief_loss += np.mean([1 - cosine_sim(e, centroid)
                                for e in embeddings])

    # Behavioural consistency: cross-entropy of decisions across framings
    behaviour_loss = 0.0
    for scenario_group in test_suite.scenario_variants:
        decisions = [persona.extract_decision(s) for s in scenario_group]
        majority = mode(decisions)
        behaviour_loss += sum(1 for d in decisions if d != majority)

    # Register stability: variance of stylistic feature vector
    register_loss = 0.0
    responses = [persona(p) for p in test_suite.register_probes]
    style_vectors = [extract_style(r) for r in responses]
    style_variance = np.var(style_vectors, axis=0)
    register_loss = np.mean(style_variance)

    # Total coherence loss (weighted sum, α+β+γ=1)
    loss = (alpha * belief_loss +
            beta  * (behaviour_loss / len(test_suite.scenario_variants)) +
            gamma * register_loss)

    return loss, {'bs': belief_loss, 'bc': behaviour_loss, 'lrs': register_loss}


def optimise_coherence(initial_prompt, test_suite, lr=0.01, steps=100):
    """
    Gradient-free optimisation of persona prompt parameters
    to minimise coherence loss. Treats the prompt as a
    high-dimensional parameter vector and performs coordinate
    descent on prompt tokens.
    """
    best_prompt = initial_prompt
    best_loss = float('inf')

    for step in range(steps):
        # Sample perturbations of the prompt
        candidates = [mutate_prompt(best_prompt) for _ in range(n_candidates)]
        losses = [coherence_loss(make_persona(c), test_suite)[0]
                  for c in candidates]

        # Select best candidate
        best_idx = np.argmin(losses)
        if losses[best_idx] < best_loss:
            best_loss = losses[best_idx]
            best_prompt = candidates[best_idx]

    return best_prompt, best_loss

The mutate_prompt function applies structured variations to the persona prompt. Mutations include rephrasing the persona description, adjusting temperature, adding or removing context constraints, and varying the specificity of belief dimensions. Each mutation produces a candidate prompt that is evaluated against the coherence loss function, and the best-performing candidate is selected for the next iteration.

In practice, we find that 50–100 iterations of prompt mutation and selection typically reduce coherence loss by 40–60% from a naive baseline. The optimisation does not require fine-tuning the underlying model — it operates entirely in prompt space, making it applicable to any API-accessible LLM. This is how Gatekeep's investor personas are calibrated before deployment: each persona archetype is run through the coherence optimiser, and only personas that achieve coherence scores above a threshold (typically 0.85 on a 0–1 normalised scale) are included in the verdict pipeline.

5. Benchmarking Results

We benchmarked coherence across three persona construction methods — naive LLM (GPT-4 with no persona instructions), prompted persona (engineered system prompt defining beliefs, background, and behavioural parameters), and fine-tuned persona (LoRA fine-tune of Llama-3-70B on 5,000 persona-consistent dialogue examples) — using a standardised test suite of 50 belief probes, 30 scenario-framing variants, and 20 register-stability prompts. Scores are normalised to 0–1, with 1 representing perfect coherence.

Beyond these laboratory benchmarks, we measured persona quality using production data from Gatekeep and Nichesim, two deployed systems that run simulated personas against real-world inputs. The following metrics were collected from production deployments between Q3 2024 and Q2 2025:

• Belief responsiveness. 92% of belief shifts matched message sentiment. When a simulated persona was presented with a persuasive argument, its belief score moved in the direction of the argument in 92% of cases. If the LLM output indicated a persona was "convinced," the belief score went up — validating that persona belief vectors are responsive to conversational context rather than operating as static lookup tables.
• Dealbreaker sensitivity. 78% of dealbreaker triggers (e.g., "the founding team has no domain experience," "the unit economics are negative at scale") caused negative belief shifts. Personas reliably reacted to known dealbreaker dimensions, confirming that the belief architecture correctly encodes investor-critical thresholds.
• Intra-persona reliability. Repeated runs across a fixed battery of 20 pitches × 5 runs yielded an intra-persona Cohen's kappa of 0.91 (p < 0.001). The same persona, given the same input on different runs, produced substantially the same verdict. This level of reproducibility is critical for production systems where users expect consistent outputs.
• Thematic alignment with human experts. When Gatekeep persona outputs were compared to real VC feedback for the same pitch decks, blinded evaluators found 78% thematic overlap. Personas identified the same concerns (market size, team composition, competitive landscape, unit economics) that human investors identified, in approximately the same proportions.
• Discovery of overlooked objections. Personas surfaced objections that human reviewers missed in 14% of cases. These were typically contrarian dimensions — for example, a persona representing a "regulatory-risk-focused investor" flagged compliance issues that generalist evaluators had overlooked, and a persona with a "DeepTech contrarian" profile questioned technical assumptions that domain-expert reviewers had taken for granted.
• Objection coverage. In a separate study using Nichesim, simulated personas identified 85% of objections that later appeared in real user interviews. This suggests a well-constructed persona panel can serve as a pre-filter for qualitative research, identifying most objections before expensive human interviews are conducted.
• User perception. In a user study (n = 30 simulations), participants rated personas 4.2/5 on "personas felt authentic." 40% of users reported "didn't think of that" moments — objections or perspectives they had not considered before interacting with the simulated persona.

The following table summarises coherence sub-dimension scores across the three construction methods. Each score is the mean of 5 independent runs against the standardised test suite:

Prompt engineering alone produces approximately 40% improvement over the naive baseline, confirming that careful prompt design is a high-leverage intervention even without model fine-tuning. The prompted persona reaches a coherence level that is sufficient for most production use cases — beliefs remain stable across paraphrases, decisions resist framing effects, and linguistic register holds across turns. Fine-tuning adds another layer, bringing total improvement to approximately 65% over the naive baseline, with the largest gains in belief stability (the deep structure of the persona's value system).

The diminishing returns from fine-tuning are notable. Prompt engineering alone achieves roughly 60% of the fine-tuned ceiling at approximately one-tenth of the engineering effort. For most production use cases, the prompted persona baseline (0.66 overall coherence) is sufficient. Fine-tuning is warranted when the persona system is a core product differentiator — as it is at Gatekeep, where persona quality directly determines verdict quality and small improvements in coherence compound across a panel of 200+ personas.

We also measured the correlation between coherence scores and downstream utility. For a set of 200 investment theses evaluated by Gatekeep's pipeline, we computed the correlation between persona coherence scores and inter-rater agreement between synthetic verdicts and human expert verdicts. The Pearson correlation between overall coherence and human-synthetic agreement was r = 0.68 (p < 0.001). In contrast, the correlation between a standard factual accuracy benchmark (MMLU-finance subset) and human-synthetic agreement was r = 0.14 (p = 0.04). Coherence is 4.9× more predictive of real-world utility than accuracy.

6. Validation Methodology

The benchmarks reported above were collected through a multi-phase validation protocol spanning both controlled laboratory conditions and production deployment data. For the coherence sub-dimension scores (belief stability, behavioural consistency, register stability), we constructed a standardised test suite of 50 belief probes (each with 3 paraphrases), 30 scenario-framing variants (10 scenarios × 3 framings each), and 20 register-stability prompts. Each persona configuration was evaluated against this fixed suite across 5 independent runs to measure intra-persona variance. The fixed 20-pitch battery × 5 runs protocol provided the data for the intra-persona Cohen's kappa calculation.

For the production benchmarks (thematic overlap, objection coverage, user perception), we deployed a blinded evaluator agreement protocol. A panel of 47 pitch-pairs was assembled: each pair consisted of the same startup deck evaluated independently by a Gatekeep persona and by a real VC. The persona and human outputs were anonymised and presented to a panel of 3 independent evaluators, who rated thematic overlap on a structured rubric without knowledge of which output came from which source. The 78% thematic overlap figure represents the mean agreement score across all 47 pairs. All production metrics (user ratings, objection coverage, dealbreaker sensitivity) were collected from live Gatekeep and Nichesim deployments between Q3 2024 and Q2 2025, representing a corpus of over 1,200 simulated persona interactions evaluated against ground-truth human feedback.

7. Production Use: Gatekeep's 12-Dimension Verdict

Gatekeep's core product is an automated investor verdict: a structured evaluation of a startup pitch across multiple dimensions, including market sizing, team quality, competitive moat, unit economics, and founder-market fit. Each verdict is generated by running the pitch through a panel of 200+ synthetic investor personas, each representing a specific investor archetype (e.g., "seed-stage generalist," "growth-stage fintech specialist," "DeepTech PhD-turned-VC"). The aggregate verdict is a weighted combination of individual persona assessments.

The entire pipeline depends on coherence, not accuracy. A persona that gives the "right" answer (e.g., correctly identifying the startup's TAM as $4.2bn based on training-data memorisation) but reasons inconsistently — citing a large TAM in one paragraph and warning about limited market size in the next — produces noise. Its verdict score cannot be trusted because the reasoning that produced it is unstable. A persona that reasons consistently, even if it is factually wrong about a specific number, produces signal. Its verdict score reflects a stable belief vector, which makes its output interpretable, reproducible, and useful for the entrepreneur receiving the feedback.

This is the core inversion that coherence-based evaluation enables. In a traditional evaluation paradigm, you ask "How often is the persona correct?" In the coherence paradigm, you ask "How stable is the persona's reasoning, and does that reasoning pattern correspond to a useful investor archetype?" The first question measures knowledge. The second question measures simulation fidelity. And for the task of simulating how an investor will respond to a pitch, simulation fidelity is what matters.

The operational implication: when a Gatekeep persona fails a coherence check, it is removed from the verdict panel, not retrained on more facts. Adding factual knowledge to an incoherent persona makes it a more knowledgeable incoherent persona — which is worse, not better, because its confident-sounding but unstable reasoning becomes harder to detect. The coherence-first approach treats persona quality as a property of the simulation, not a property of the underlying model's training data.

8. Conclusion

The dominance of accuracy-based evaluation in LLM benchmarking has led the field to systematically undervalue coherence as a quality metric. This is a costly oversight for any application that involves simulating human behaviour, because coherence — not accuracy — is what makes a synthetic persona useful. An accurate persona is a database query. A coherent persona is a simulation.

The prompt-space optimisation method presented here provides a practical, automatable way to measure and improve persona coherence without requiring model fine-tuning. The empirical results confirm that prompt engineering alone delivers approximately 40% coherence improvement over naive baselines, and that coherence scores are 4.9× more predictive of real-world utility than accuracy scores. For production persona systems like Gatekeep's investor verdict pipeline, coherence is not a nice-to-have evaluation dimension — it is the evaluation dimension that determines whether the system works at all.

Future work should extend the coherence framework to multi-agent interactions, where the coherence of individual personas is compounded by the coherence of their interactions. Two individually coherent personas that produce incoherent dialogues when paired — agreeing and disagreeing on the same point in successive turns — reveal a gap in the current framework that is relevant to any system where personas interact with each other, not just with a single evaluator.