Sullivan & Cromwell teaches AI to review contracts by its own precedents

Sullivan & Cromwell built an internal Agreement Analyzer with OpenAI for reviewing draft contracts. It combines ChatGPT Enterprise with selected firm precedents, negotiating playbooks and information about market practice.

A Codex prototype became a multi-user review process

Partner John Savva first built a single-user proof of concept in Codex. OpenAI engineers then helped improve output quality, run parts of the analysis in parallel and prepare the application for several lawyers to use at once.

A lawyer supplies a draft agreement, selected precedents and a playbook covering firm and client positions. The system runs several review cycles and returns a memorandum plus proposed tracked changes. The lawyer can accept, modify or reject them.

Codified judgment is the firm's real advantage

A general chatbot can summarize a clause. Agreement Analyzer is meant to do something more specific: compare it with how one firm allocates risk, which deviations it considers material and what it tends to pursue in negotiations. The precedents and playbooks are more valuable here than the text interface itself.

For law firms, that changes the economics of internal knowledge. A partner's experience no longer has to remain in one person's head or a folder of old documents. It can become a process used by a wider team while final responsibility stays with a lawyer.

Inspectable evidence helps, but accuracy metrics are missing

The tool has specialized agents develop conclusions and other agents challenge them. Its output separates findings that survived challenge, dismissed findings and unverified issues retained with qualifications. That is a sounder design than one confident answer.

The public account gives no accuracy rate, missed-issue rate or independent evaluation results. Savva also frames the short-term benefit as more thorough analysis in the available time, not necessarily fewer working hours. The productivity claim therefore rests on broader coverage rather than demonstrated time savings.

Rejected edits and audit trails will reveal the value

The decisive test is whether lawyers can trace each recommendation to a precedent, an explanation and a reason it survived challenge. Firms should also measure how many suggestions lawyers reject and which material issues the system misses.

If Sullivan & Cromwell collects that evidence, Agreement Analyzer can improve the firm's institutional memory. Without the feedback loop, it will merely produce polished revisions faster than a lawyer can verify their legal significance.

Lilith's verdict

Sullivan & Cromwell has placed three piles beside the lawyer: supported findings, dismissed findings and unresolved questions. The value appears only if the partner reaches for the right pile before signing.