Ambi: a neurosymbolic loophole finder for policies
Published:
Ambi is the prototype behind a startup I co-founded (I’m the CTO). It reads a policy or rulebook and finds two things: where readers will disagree about what it means, and where someone can follow the letter of the rules while defeating their purpose.
How it works. An LLM translates the policy into fuzzy-logic primitives and a violation formula, with a validate-and-repair loop and a back-translation check. A panel of deliberately weak LLM readers then judges real situations (288 cases with real outcomes, collected from Academia Stack Exchange). Ambi measures how often the readers disagree and uses Shapley values to attribute that disagreement to specific phrases, shown as Grammarly-style underlines. Writers can compare two versions of a policy on the same cases.
Loophole solver. Given an outcome the author doesn’t want, an exact search looks for fact patterns the formula still allows, returns a verified Boolean witness, and explains which clauses block it when none exists. The solver itself makes no LLM calls.
Evaluation.
- 27/27 correct decisions on hand-built policies with known answers, 0 false positives.
- 54/54 satisfiable cases recovered against an exhaustive six-variable Boolean oracle, 0 false positives.
- On a public loophole benchmark (Choi, Bansal & Stengel-Eskin, EMNLP 2025), with the rules encoded by hand: 360/360 known loopholes found and 0 false positives on 120 closed controls.
Stack: Python, FastAPI, Anthropic and OpenAI APIs, fuzzy logic, SAT-style search, vanilla JS.