96% of surveyed conversational AI papers can't be faithfully replicated

Reproducibility in conversational AI research is broken in a specific, measurable way: of 29 CUI 2026 papers examined, 28 were missing information needed to faithfully replicate them — questionnaire wording, agent prompts, stimulus materials. These papers had passed peer review. The problem isn't carelessness; it's that the experimental condition in a conversational AI study is distributed across a dozen moving parts — interface layout, system prompts, retrieval context, condition assignment logic, procedural branching — and no existing publication format captures all of them.

Gricea attacks this structurally. Rather than asking researchers to separately document what they built, it makes the documentation identical to the thing that runs. Studies are authored as two connected directed graphs: a Study Flow governing how participants move through consent, condition assignment, tasks, and surveys, and a Task Flow governing what happens turn-by-turn inside each interactive condition. Publication locks those graphs, their prompts, and their configurations into an immutable version. The runtime executes that version directly. There is no translation step where fidelity can be lost.

The dual-graph abstraction does real work. A researcher can change agent behavior without touching the procedural assignment logic, or modify the between-subject structure without rebuilding the conversational task. When someone forks a published study to test a related hypothesis, the diff between source and fork is explicit — which parameters changed, which were held constant. That's the shared frame of reference cumulative science actually needs.

The replication study is the most concrete evidence here. Of 29 eligible papers, 27 were replicated as executable artifacts — 10 completely, 17 partially. The partial replications were blocked by missing source materials, not by Gricea's expressive limits. Two studies couldn't be replicated at all because interview and survey materials were simply unavailable.

The usability evaluation involved ten participants across PhD students, faculty, industry practitioners, and one undergraduate. All successfully implemented runnable studies during open-ended authoring sessions. The range was genuinely wide: a debate between two voice agents and a human moderator, a study on how AI-disclosure affects text evaluation, a comparison of standard search against RAG architectures for serendipity. One participant estimated Gricea would save three to six months of engineering effort on their protocol.

Frictions were real and worth noting. The transition from Study Flow to Task Flow confused several participants — the nested canvas felt like entering a different system. Variable passing across flow levels tripped up others. Participants with social-science backgrounds found the Task Flow granularity excessive for their needs. The platform works best when a research question is already concrete; early-stage ideation still requires something Gricea doesn't yet provide.

The planned AI copilot — which would propose initial study configurations from high-level research questions — is the most interesting future direction, and also the one that most depends on community adoption to validate. Shared artifacts only enable cumulative knowledge if researchers actually share them.