NicheSim: A Multi-Agent Belief Simulation Platform

Introduction

NicheSim is an experimental three-tier, streaming-first simulation system for modeling authentic community discourse around product ideas. The system generates 80–150 message conversations between 5–8 psychologically coherent personas, producing in ~90 seconds what traditional user research delivers in weeks at 100× the cost. Unlike synthetic survey response generators that treat each user as an independent data point, NicheSim models interaction — personas read each other's messages, form alliances, dispute claims, change their minds, and produce the emergent social dynamics that define how real communities evaluate new ideas.

The platform was built to solve a structural problem in product research: traditional methods are slow (2–6 weeks per round), expensive ($15K+ per round for moderated studies), and statistically underpowered (n=8–12 focus group participants cannot capture the combinatorial space of reactions across belief profiles). NicheSim collapses the cost-to-insight ratio by generating synthetic communities with calibrated belief models, running them through controlled conversation scenarios, and surfacing not just "what people think" but the trajectories of belief — who converts, who hardens, and why. This paper presents the full technical architecture: the three-tier system design, the burst generation engine that enables organic multi-agent dialogue, the belief-as-state tracking infrastructure, the conversation phase system, the grounded verdict pipeline, and the cross-simulation learning corpus that compounds quality with every run.

System Architecture

NicheSim is designed as a three-tier system with clean separation between presentation, application logic, and persistent state. The presentation layer uses React 18 with Next.js 14, leveraging Server-Sent Events (SSE) for real-time streaming of message generation. Users see conversations unfold message-by-message as the backend generates them, with typed indicators, typing pauses, and social dynamics cues (“Alex agreed with Jordan’s point”) rendered inline. The streaming architecture is critical to the user experience: a synchronous 90-second black box would feel slow and opaque; the same 90 seconds rendered as a live conversation feels immediate and transparent.

The application layer is a Node.js API server that orchestrates six core engines. The generation engine manages LLM calls using burst generation (batched multi-message generation with scene directives) to produce organic conversation dynamics. The interaction engine tracks reply chains, mention networks, and social graph formation within each simulation. The analysis engine performs the batched analytical pass that tracks belief trajectories without burdening the generation pipeline with self-assessment. The belief tracker maintains discrete belief state vectors for each persona across conversation turns. The learning engine writes structured learnings from each completed simulation into the cross-simulation corpus. The research integrator fetches real-world context — competitor data, market data, news articles — and injects it as structured context during persona and scenario generation.

The data layer runs on PostgreSQL 17 via Neon serverless, with 20+ tables covering users, simulations, personas, conversation turns, belief snapshots, objectional ledgers, conversation networks, learning entries, persona templates, message exemplars, research context, and simulation metadata. External dependencies are minimal by design: Anthropic Claude Sonnet 4 serves as the sole LLM provider, and web search APIs (SerpAPI, Tavily) provide the research integration layer. The architecture diagram captures the flow:

├── Presentation Layer (React 18 + Next.js 14 + SSE streaming)
┞   ├── Simulation Dashboard
┞   ├── Real-time Message Stream
┞   └── Verdict & Analytics View
┞
├── Application Layer (Node.js API)
┞   ├── Generation Engine
┞   ┞   ├── Burst Generator (batched LLM calls)
┞   ┞   └── Scene Directive System
┞   ├── Interaction Engine
┞   ┞   ├── Reply Chain Tracker
┞   ┞   └── Mention Network Builder
┞   ├── Analysis Engine
┞   ┞   ├── Belief Trajectory Analyzer
┞   ┞   └── Batched Analytical Pass (every 10 msgs)
┞   ├── Belief Tracker
┞   ├── Learning Engine
┞   └── Research Integrator
┞
└── Data Layer (PostgreSQL 17 / Neon serverless)
    ├── simulations, personas, messages
    ├── belief_snapshots, objection_ledgers
    ├── conversation_networks, persona_templates
    ├── message_exemplars, learning_corpus
    └── research_context, simulation_metadata

Persona Generation

NicheSim generates personas that are psychologically coherent agents with distinct linguistic fingerprints and internally consistent cognitive models — not just demographic shells with LLM-style personas tacked on. Each persona is defined by a rich attribute set: an identity block (username, bio, avatar, account age), a communication style fingerprint (syntax patterns, vocabulary register, emoji usage frequency and palette, average message length, capitalization habits), a cognitive model (prior_belief strength on a 0–1 scale, a convincers[] array of argument types that shift their position, a dealbreakers[] array of red lines that trigger rejection, an evidence_preference enum of {data, story, authority}, and an update_style enum of {jump, gradual, stubborn}), an expertise model (domain knowledge areas with confidence scores), and engagement triggers (knowledge blindspots that provoke curiosity-driven questions, proof triggers that demand citations, emotional hot buttons).

Personas are not generated from scratch for each simulation. The learning engine queries the persona_templates table for high-quality archetypes that match the simulation’s niche context, then applies an exploration policy to inject 10–30% novelty into the attribute set. This means a simulation about a developer tool will draw from proven templates of developers, engineering managers, indie hackers, and skeptics, while introducing controlled mutations — a developer who is unusually skeptical of open-source licensing models, or an indie hacker who overweights design aesthetics in product evaluation. The exploration-exploitation balance ensures that each simulation benefits from accumulated pattern quality without becoming repetitive. The persona generation pipeline also incorporates real-world research context: if the simulation targets a specific market (e.g., productivity SaaS for remote teams), the research integrator fetches recent news, competitor information, and community sentiment to ground each persona in authentic, time-relevant context.

While the cognitive model (prior_belief, convincers, dealbreakers, evidence_preference, update_style) governs how a persona reacts to new information, the 12-dimension belief vector defines what the persona believes about the idea being simulated. Each belief dimension — risk tolerance, time preference, technical depth, domain expertise, decision speed, social proof sensitivity, contrarian tendency, loss aversion, narrative susceptibility, information processing style, trust modality, and value alignment — is a scalar value in [0,1] that shapes the persona's evaluation output. The cognitive model and the belief vector serve distinct purposes: the former controls the dynamics of opinion change during conversation, while the latter establishes the baseline worldview that conversation then perturbs.

Burst Generation Engine

The burst generation engine is NicheSim’s key architectural innovation and the subject of a companion paper (Chua, “Burst Generation: Scene-Directed Multi-Agent Dialogue,” 2025). Traditional multi-agent systems run a round-robin loop: select an agent, generate one message, repeat. This produces 80+ sequential LLM calls for an 80-message conversation, each call blocked on the previous one, and the resulting discourse is stilted — agents respond to each other mechanically, each waiting their turn, with none of the overlapping energy, pile-ons, or rapid back-and-forth that characterizes real online conversation.

Burst generation solves this by generating 2–4 messages in a single LLM call with a scene directive that sets the interaction pattern for that burst. Scene directives include: pile-on (multiple agents reinforce a shared sentiment), alliance (two agents find common ground and build on each other’s points), debate (structured disagreement with point-counterpoint), derail (a tangent that pulls conversation off-topic before returning), callback (reference to an earlier message, creating narrative continuity), and normal (standard sequential replies). The LLM receives: the conversation state so far, the selected scene directive, the participant personas in the burst (2–4 selected from the pool), their current belief states, and the target interaction dynamic. It returns 2–4 fully formatted messages with speaker attribution, reply-to references, and optional metadata (emoji reactions, quote embeds).

Participant selection for each burst uses a weighted scoring function: score = base_weight + opinion_strength_factor + trigger_topic_match + expert_boost - balance_penalty + random_jitter. The base_weight ensures every persona gets a baseline chance to speak. opinion_strength_factor boosts personas with strong, unexpressed views on the current topic. trigger_topic_match activates personas whose engagement triggers align with the current conversation thread. expert_boost gives a 1.5× multiplier to personas with domain expertise on the active topic. balance_penalty suppresses personas who have dominated recent bursts, preventing a single voice from drowning out the conversation. random_jitter (±15%) introduces controlled stochasticity so that repeated simulations with the same product generate different interaction patterns. The distribution of scene directives also varies by conversation phase: pile-ons cluster in the engagement phase, debates peak during deep dive, derails are most common in wind-down, and alliances appear uniformly.

Belief Tracking Engine

NicheSim models belief as a persistent state that evolves across a conversation, but it decouples belief tracking from message generation. This is a deliberate architectural choice born from a failed initial approach. The first prototype attempted self-grading — each persona evaluated its own belief shift after every message, producing a numerical delta (e.g., “belief 0.72 → 0.68”). This failed on three dimensions. First, LLMs are demonstrably poor at numerical self-assessment: they produce optimistic, low-variance estimates that under-report genuine belief change. Second, self-grading adds cognitive load to the generation prompt, degrading message quality by forcing the LLM to simultaneously produce natural dialogue and introspect on its internal state. Third, self-grading creates a form of meta-awareness that is unnatural in conversation — real people do not consciously recalculate their beliefs sentence by sentence, and prompting an LLM to do so introduces a reflective tone that leaks into the dialogue itself, producing messages that feel like debate-club analysis rather than casual conversation.

The solution is a batched analytical pass that runs asynchronously every 10 messages. A separate LLM call — using a dedicated analysis prompt, not the generation prompt — receives the full conversation segment, the persona’s prior belief state, and their cognitive model (convincers, dealbreakers, update style). The analysis prompt asks: “Given this persona’s cognitive model and the messages they’ve read, how should their beliefs shift?” The output is a structured belief delta with confidence scores and a natural-language reasoning trace. This separated architecture produces more accurate belief tracking (the analyzer can focus entirely on belief inference without the distraction of generating dialogue) and more realistic messages (the generator can focus entirely on authentic conversation without self-monitoring overhead). The belief snapshots are stored in PostgreSQL with timestamps, delta vectors, and reasoning traces, enabling post-hoc trajectory visualization and causal analysis of which messages moved which beliefs.

Conversation Phase System

Not all messages in a conversation carry equal weight, and treating a 80-message thread as a flat sequence misses the structural rhythm of authentic discourse. NicheSim models conversations in four phases. The opening phase (messages 1–15) establishes context: personas introduce themselves (or are already known to each other, depending on the community simulation), the product idea is presented, and initial hot-take reactions surface. Messages are short (1–3 sentences), emotionally charged, and heavy on surface-level evaluation. The engagement phase (messages 16–45) is where the conversation thickens: positions develop nuance, alliances form, disagreements emerge, and personas begin citing evidence and experience. Message length increases to 3–6 sentences. The deep dive phase (messages 46–65) is the analytical core: personas engage in structured debate, dissect specific aspects of the product (pricing, UX, market positioning), and belief trajectories show the largest deltas. Messages are long (4–8 sentences) and dense with reasoning. The wind-down phase (messages 66–80) brings closure: positions are summarized, recommendations are offered, and the conversation energy dissipates.

A key innovation in the phase system is the filler message mechanism. Real communities do not operate at maximum information density; they include low-content, high-social-signal messages (“this ^”, “lmao”, “facts”, “honestly same”) that regulate conversational rhythm and signal alignment without advancing the analytical thread. NicheSim models this with a phase-dependent filler rate: 5% in opening, 8% in engagement, 3% in deep dive (where every message should carry analytical weight), and 25% in wind-down (where social cohesion and conversational closure dominate). Filler messages are generated via a lightweight template system with persona-specific stylistic variations — the Gen-Z-coded persona uses different filler vocabulary than the academic-coded persona — ensuring that even low-content messages reinforce persona coherence.

Grounded Verdict System

The verdict system follows a measure-then-interpret pipeline. Rather than asking the LLM to produce a verdict from raw conversation text — which would invite hallucination, recency bias, and lossy summarization — the system first computes quantitative metrics directly from the database: belief trajectory deltas (which personas shifted, by how much, and in which direction), engagement signals (reply-chain depth, mention frequency, quote rate), consensus measures (agreement clustering across personas, calculated via pairwise sentiment alignment on key topics), and expert splits (whether domain experts and laypeople diverged on specific claims). These metrics are computed deterministically; the LLM never touches raw numbers.

The metrics are then provided to the LLM as grounded context for interpretation. The prompt includes: the computed metrics, representative quotes extracted from the conversation, the research context used to initialize the simulation, and the product details. The LLM synthesizes this into a structured verdict: an overall confidence score, a narrative summary of community sentiment, a breakdown by persona archetype, and key themes. The confidence formula is deliberately conservative: confidence = min(95, 40 + message_volume_bonus + persona_count_bonus + consensus_bonus), where message_volume_bonus contributes up to 25 points (scaled from 0 for ≤20 messages to 25 for ≥80), persona_count_bonus contributes up to 15 points (scaled from 0 for ≤3 personas to 15 for ≥8), and consensus_bonus contributes up to 25 points (inverse of pairwise agreement variance). The cap at 95% is intentional: it signals that even ideal simulation conditions cannot produce certainty, and users should treat synthetic community feedback as a directional signal, not ground truth. The verdict system also produces an objection ledger — a structured table of every distinct pushback raised during the conversation, with severity score, frequency count, representative quotes, and suggested responses — giving product teams an actionable, prioritized list of concerns to address.

Cross-Simulation Learning

Every completed simulation feeds into a growing learning corpus structured across five knowledge categories. Niche categorization maps products to taxonomy nodes (industry, target user, price tier, stage) so that future simulations in the same niche can bootstrap from accumulated context. Knowledge retrieval extracts reusable factual claims, market data points, and competitive insights that persist across simulations. Persona templates capture which archetypes produced the most internally coherent, conversationally realistic behavior and promote them for reuse. Message exemplars store high-quality messages tagged by persona type, conversation phase, and scene directive, creating a reference library that improves few-shot prompting. Objection patterns aggregate common pushbacks across simulations in the same niche, enabling the system to proactively seed conversations with known counter-arguments.

The corpus operates with an explicit exploration-exploitation policy. For each new simulation, 70–90% of persona attributes, scene directives, and conversation parameters draw from proven high-quality patterns in the corpus. The remaining 10–30% are novel mutations: untested persona combinations, rare scene directives, or experimental parameter variations. This creates a flywheel effect: more simulations → richer pattern library → higher baseline quality for new simulations → lower cost-per-insight → more simulations. The corpus grows nonlinearly in value because the most useful learnings are combinatorial: a persona archetype that works well for SaaS products may also work well for marketplace products in a different configuration, and the system discovers these transfer patterns through controlled variation.

Performance

An 80-message simulation with 6 personas completes in 60–90 seconds of wall-clock time. This is achieved through approximately 25–30 LLM calls (versus the 80+ calls that a naive round-robin approach would require), representing a 60–75% reduction in API calls via burst generation. Each burst call generates 2–4 messages, with the number of messages per burst varying inversely with the information density of the conversation phase (more messages per burst in wind-down, fewer in deep dive). The batched analytical pass adds 3–5 additional LLM calls per simulation (one every 10 messages), and the verdict synthesis adds 1–2 calls. The streaming SSE architecture means users begin seeing messages within 3–5 seconds of simulation start, even as later messages continue generating, providing a responsive experience despite the total generation time.

Evaluation

NicheSim’s quality is measured against five benchmarks drawn from a controlled evaluation study (full results in Chua, “Coherence, Not Accuracy: Evaluating LLM Personas at Scale,” 2026). Belief-message alignment: 92% of belief vector shifts correspond to the sentiment direction of the message that triggered them, measured by comparing the belief delta direction (+/-) against independent message sentiment classification. Intra-persona coherence: 0.91 Cohen’s kappa across repeated simulation runs, confirming that personas produce consistent behavior patterns within their cognitive models. Thematic alignment with human feedback: 78% overlap between themes surfaced by NicheSim simulations and themes surfaced by traditional user interviews for the same product, as measured by blinded evaluator agreement on theme categorization. User-perceived authenticity: 4.2/5 average rating from product teams asked to evaluate the realism of synthetic conversations against real community threads (n=47 evaluators). Objection coverage: 85% of objections raised in real user interviews were also surfaced in at least one NicheSim conversation about the same product, confirming that synthetic discourse captures the critical dimensions of real community pushback.