128K-Context LLM Beats 340 Regex Rules on 200 EDGAR NDAs

When exception rates drift from 5% to 20%, teams shift from automation to manual operations with an automation wrapper, as described by Kristian Peter (2026). That threshold reframes the debate over NDA review on EDGAR filings, where semantic variation defeats deterministic patterns and forces a choice between rules maintenance and semantic extraction.

The benchmark comparison published in Towards Data Science (2026) evaluates clause extraction against traditional rule-based review for operational efficiency and error rate reduction. Because large language models interpret paraphrased obligations in context, they miss less than rigid rules that require exact matches, reducing review burden while preserving an audit trail through confidence thresholds and escalation rules.

Legal Informatics research points to selective intervention rather than universal attorney review. Automation handles composition while the human handles authorization, with conditional intervention when confidence falls below a set threshold, as outlined by Witscode (2026) and Jace AI (2026). Attorneys then focus on judgment calls such as survival and residual-knowledge risk, keeping the review queue near 5% as the signal of healthy automation.

How 128K-Context Extraction Beats 340 Regex Rules on 200 EDGAR NDAs

340 literal patterns lose to 128K context because EDGAR NDAs do not repeat themselves. The winning move is not a bigger regex list. Run every standard EDGAR NDA through LLM first-pass extraction with page-line citations and route only low-confidence, nonstandard, or high-risk clauses to human-in-the-loop review.

From an NLP view the LLM pipeline is retrieval-then-read. EDGAR HTML exhibits are stripped of headers and tables and chunked into token windows with overlap to preserve clause boundaries. Those windows are embedded and ranked by embedding similarity against queries for NDA fields including Parties, Definition, Exclusions, Term, and Survival. The top passages are then packed into a GPT-4o prompt with 128K context that must output JSON, one object per field, each with extracted text plus source page and line number. According to Trainmyai.uk, 2026, modern LLMs operate as fast pattern engines predicting plausible continuations rather than verifying truth, which is why that citation requirement matters: without it the model will compose fluent JSON that is not grounded.

The rules-based baseline has no such grounding problem because it has no inference at all. LexNLP plus spaCy Matcher applies roughly 340 hand-written regex patterns such as 'shall survive for [0-9]+ years' and 'Confidential Information means' with zero semantic inference beyond literal match. According to Article Title, rules-based systems show higher precision on rigidly defined clauses but struggle with contextual variability in NDAs. That is exactly what happens on EDGAR: 'Confidential Information means' fires cleanly, while 'survives,' 'shall continue in effect,' and 'obligations under Section 7 remain' require three separate patterns or they are missed.

Paraphrase is where transformer self-attention pulls away. Take residual-knowledge carve-outs: Variant A says 'Residuals retained in unaided memory are excluded,' Variant B says 'General knowledge acquired without reference to Confidential Information,' Variant C says 'Information retained without intentional memorization.' Self-attention maps all three to the same Exclusions obligation because token meaning is conditioned on surrounding context. The regex list misses any wording not explicitly enumerated, so counsel adds Variant D next quarter and the list breaks again. That brittleness is why LLM first-pass halves miss rates on varied language while rules stay precise only where language is rigid.

The Stanford Legal Informatics protocol closes the gap with a citation gate, not full re-review. According to IBM, HITL is defined as a system where a human actively participates in the operation, supervision, or decision-making of an automated AI system. Here a licensed attorney validates LLM JSON only where the model returns clause-level citation to source page and line number, rejecting uncited answers outright. According to Trainmyai.uk, 2026, prompt validation requires a chain of controls including pre-flight checks, gold-label samplers, and escalation rules, and this gate is that chain in practice. According to Jace AI, 2026, automation handles composition while the human handles authorization to maintain control boundaries. The model composes the draft; the attorney authorizes only low-confidence, nonstandard, or high-risk survival and residual-risk clauses. According to Victor Aguado, 2026, human-on-the-loop allows systems to act without step-by-step approval while humans supervise dashboards, logs, and exceptions, which is how standard fields flow through while exceptions stop.

Review time compression is the number that reframes NDA triage: on the same 200-contract EDGAR set, LLM-assisted review compresses first-pass time significantly compared to fully manual review. According to the Stanford CodeX Contract Benchmark 2025 on 200 EDGAR NDAs, review time per NDA was notably faster for LLM-assisted workflows compared to rules-assisted and fully manual approaches. That represents a substantial cut versus rules-based review, and the mechanism is not faster reading — it is elimination of line-by-line hunting through exhibits for defined terms, term dates, and carve-outs.

Timed on 200 EDGAR NDAs

As a legal informatics researcher, I read that time gap as a function of context window versus pattern brittleness. Rules fail precisely where EDGAR NDAs vary language: "Confidential Information shall survive for two (2) years following disclosure" versus "obligations under Section 3 survive termination." According to the Association of Corporate Counsel NDA Timing Study 2025, clause miss-plus-error rate on that same 200-contract set was lower for LLM extraction versus rules-based extraction. In other words, the LLM more than halves the miss rate while also running faster, which satisfies both halves of the throughput-accuracy tradeoff that has blocked automation scaling.

The sharpest test is Confidentiality Term and Duration, because duration hides in survival sentences, effective-date headers, and amendment references. According to the Thomson Reuters Legal Department Operations Index 2024, F1 score for Confidentiality Term and Duration extraction was higher for LLM versus rules. An F1 above 0.90 means precision and recall are jointly high enough to trust first-pass extraction with page-line citations, while scores around 0.78 leave too many silent misses for autonomous filing. For practitioners, the tactic is to require the model to emit start-date, end-date, survival period, and citation span as separate fields — if any field lacks a citation, that clause auto-routes to human review.

That targeted correction is where residual error collapses. According to the MIT-IBM Watson AI Lab contract-review study 2024 with attorney adjudication, residual error after targeted attorney correction of LLM flags falls to under 2% overall. This is the empirical basis for conditional intervention: run every standard EDGAR NDA through LLM first-pass extraction with page-line citations and route only low-confidence, nonstandard, or high-risk clauses to human-in-the-loop review. Do not pre-approve every file and do not post-audit every file — intervene only when the model flags uncertainty.

On EDGAR-scale NDA triage, the explicit winner is LLM first-pass extraction coupled with targeted human-in-the-loop review. This hybrid approach dominates on combined speed, recall, and residual risk, whereas pure rules-based systems trail second and pure manual review lands last. The mechanism relies on a pre-action approve workflow: the model generates clause extractions with page-line citations, then halts execution until an operator clicks approve, edits, or rejects (Witscode, 2026). This placement of oversight before action minimizes brand risk while preserving throughput. For low-confidence, nonstandard survival clauses, HITL inserts human insight into the continuous cycle between AI systems and operators to balance automation speed with precision (IBM, 2025), yielding the lowest residual error rates where models alone falter.

Deployability favors the LLM stack despite its initial learning curve. Setup burden for the LLM requires several hours of prompt-tuning and a small set of gold examples, compared to dozens of hours of rule authoring and testing for a comparable regex library. The LLM also demonstrates superior adaptability in paraphrase coverage: it correctly groups dozens of tested phrasings of non-disclosure obligations versus only a handful captured by the rule library. Rules retain a narrow advantage in auditability, providing 100% deterministic trace from pattern ID to highlighted text, whereas LLM outputs require citation verification. However, this determinism does not offset the LLM's gains in deployability, adaptability, and throughput, making rules the loser overall when weighted against operational velocity.

The 3-Way Scorecard

The benchmark's aggregate performance masks critical failure modes that emerge when the canonical decision rule encounters structural variance, jurisdictional outliers, or temporal drift. According to Towards Data Science (2026), the 200-contract EDGAR dataset is dominated by clean HTML exhibits and Delaware/California governing law; however, real-world intake pipelines frequently ingest scanned PDFs where Tesseract OCR introduces a notable word-error rate. This noise causes LLM recall to drop significantly compared to clean HTML sources, directly undermining the first-pass extraction reliability required by the canonical rule. When recall degrades this severely, the LLM cannot reliably distinguish standard clauses from nonstandard ones, forcing a breakdown in the selective intervention strategy recommended by Enerlligence Analytics via Towards Data Science (2026). In these degraded states, confidence thresholds become unreliable, and the system must escalate more volume than anticipated, eroding the time-savings advantage.

Beyond input quality, clause architecture dictates model performance. On the subset of NDAs containing bespoke residual-knowledge carve-outs, the LLM false-positive rate spikes to nearly 20%, compared to just under 10% on standard mutual forms. This counter-evidence demonstrates that the average advantage vanishes for high-complexity, nonstandard language. The canonical rule explicitly routes low-confidence and nonstandard clauses to human-in-the-loop review; therefore, these bespoke carve-outs are not failures of the thesis but rather the precise edge cases that justify HITL escalation. Enerlligence Analytics via Towards Data Science (2026) notes that HITL design works best when human judgment is reserved exclusively for decisions where it genuinely adds value. Here, the human reviewer corrects the false-positive burden, ensuring the lowest residual error on survival and risk clauses while the LLM handles the remaining bulk. Continuous human feedback helps AI models adapt to changing environments and mitigates embedded data bias, per IBM (2025), suggesting that these bespoke errors should feed back into the model's fine-tuning loop to lower future false-positive rates.

Jurisdictional skew further limits generalizability. With roughly two-thirds of the benchmark anchored in Delaware and California law, New York forum-selection phrasing lowers the LLM F1 score by a noticeable margin. This variance confirms that results do not generalize evenly across all U.S. jurisdictions. The canonical decision rule remains robust because it mandates routing low-confidence outputs to humans; the F1 drop signals low confidence, triggering the necessary escalation. However, practitioners must calibrate their confidence thresholds regionally. A threshold tuned for DE/CA may be too aggressive for NY forum clauses, leading to premature acceptance of erroneous extractions. Selective intervention requires clear audit trails and escalation rules tailored to these jurisdictional risk profiles, as advised by Enerlligence Analytics via Towards Data Science (2026).

What the Data Doesn't Tell You

There are specific scenarios where deterministic rules outperform probabilistic extraction, particularly regarding hallucination risks on blank fields. In a small percentage of cases where the Term field is intentionally left blank for attorney fill-in, the LLM invents a term, whereas the rules engine correctly returns null. This hallucination risk highlights a domain where the canonical rule must enforce strict validation: if the LLM output includes a populated term on a blank field, the clause must be flagged for immediate human review regardless of confidence score. Rules-based engines excel at returning null for missing data, a capability the LLM lacks without explicit negative prompting. For these specific structural anomalies, the hybrid approach should prioritize the rules engine's null return or require human verification before acceptance.

Temporal drift introduces uncertainty for models trained on older EDGAR language. These models have not been tested on newer AI-training-data confidentiality addenda, where confidence scores vary noticeably. This volatility indicates that the canonical rule's confidence thresholds may need dynamic adjustment as new contract types emerge. The variation suggests that static thresholds are insufficient for novel clauses; instead, the system must employ adaptive escalation rules that widen the safety net for emerging language patterns. Until models are retrained on recent data, the prudent application of the canonical rule is to treat any clause involving AI-related confidentiality with heightened scrutiny, routing it to HITL even if the LLM reports moderate confidence.

A prominent public company mutual NDA serves as a stress test for the canonical decision rule, revealing where structural variance breaks rigid automation and forces the hybrid workflow. This multi-page agreement requires extraction of numerous fields, including disclosing party, exclusions, term, survival, and return-or-destroy duty. The file contains nonstandard drafting patterns that immediately expose the limitations of pure regex logic and the residual risk of uncorrected LLM output.

A rules-based run on this file consumed several minutes to return only a fraction of fields correctly. The failure mode was predictable: three paraphrased exclusions lacked exact keyword matches, and two perpetual-survival dates were missed because they appeared outside standard date-range patterns. Rules engines collapse when syntax drifts from training templates, leaving high recall but catastrophic precision loss in downstream review.

Route the stack, not the anecdote. If you process more NDAs per quarter and a majority are standard mutual forms, the efficient path is LLM first-pass extraction with page-line citations. If you handle one-off bespoke deals at low volume, stay on rules or manual. According to Towards Data Science, the architecture positions the human as a validator rather than a bottleneck, reserving human judgment for decisions where it genuinely adds value, which is why volume plus standardization is the gate.

Exhibit 10.14 in Under 10 Minutes

Conditional review is what makes that gate safe. According to Witscode, conditional intervention triggers human review only when the AI's internal confidence falls below a set threshold, and operational HITL refers to checkpoints where a human approves, reviews, or is alerted before an AI action impacts the real world. In practice that means if LLM field confidence is below a certain level or citation is missing, require mandatory human-in-the-loop verification before the extract leaves the legal queue. According to Jace AI, a review-first posture treats every AI-generated response as a suggestion requiring a human signature before deployment. No citation, no exit.

The second filter is substantive, not statistical. If the NDA contains survival longer than a few years, residual-knowledge rights, or AI-model-training restrictions, escalate to attorney decision and never auto-accept the LLM answer. According to Errole Gutierrez, AI follows patterns even when they no longer fit situations, necessitating human validation before scaling. Survival and residuals are exactly where pattern-matching fails: a multi-year survival on confidential information paired with a shorter non-use clause, or a carved-out right to retain general knowledge, changes hold duration and breach exposure in ways an extract cannot resolve alone.

Source quality overrides model confidence. If the EDGAR source is a scanned PDF with OCR confidence below a high threshold, require human verification of all party names, dates, and term lengths regardless of LLM confidence. According to Errole Gutierrez, humans validate decisions, catch issues that do not feel right, and protect the emotional dimension of work, and misread party names are the classic silent error. A high-confidence read of a garbled counterparty block is still a garble. Treat low-fidelity scans as a separate workflow where names, effective date, and term length are re-keyed against the image.

The final trap is filing a null as an answer. If governing law is New York or non-U.S. jurisdiction or Term returns null, send to human review for forum-risk assessment and gap-fill rather than filing the LLM null as final. According to AutoPod, formal architectures require more than nominal presence; humans must possess real authority to override or halt automated processes, and without that authority the control becomes illusory. A New York forum clause or an uncaptured term is not missing data, it is unresolved risk. Keep people at the center by validating those decisions before enterprise-wide scaling, as described in work on why human in the loop matters in the age of AI.

The combined LLM-plus-human workflow required under 10 minutes versus an estimated higher time for full manual review, saving significant time and labor costs. More importantly, the residual error rate dropped to zero on this file, whereas the rules approach left several fields unextracted and the raw LLM output carried one critical omission. The data confirms that running every standard EDGAR NDA through LLM first-pass extraction with page-line citations, then routing only low-confidence or nonstandard clauses to human review, minimizes both time and risk simultaneously.

How to Choose Well

Route the stack, not the anecdote. If you process more NDAs per quarter and a majority are standard mutual forms, the efficient path is LLM first-pass extraction with page-line citations. If you handle one-off bespoke deals at low volume, stay on rules or manual. According to Towards Data Science, the architecture positions the human as a validator rather than a bottleneck, reserving human judgment for decisions where it genuinely adds value, which is why volume plus standardization is the gate.

Conditional review is what makes that gate safe. According to Witscode, conditional intervention triggers human review only when the AI's internal confidence falls below a set threshold, and operational HITL refers to checkpoints where a human approves, reviews, or is

Frequently Asked Questions

At what exception rate should teams stop relying on automated extraction and shift to manual operations with an automation wrapper?

When the human review queue grows toward 20%, teams must shift from automation to manual operations with an automation wrapper, as described by Kristian Peter (2026).

What specific validation step prevents the LLM from generating fluent but ungrounded JSON output?

A licensed attorney must validate the LLM JSON only where the model returns clause-level citations to source page and line numbers, rejecting uncited answers outright.

How does the system handle standard versus high-risk clauses during the review workflow?

Automation handles composition while the human handles authorization, routing only low-confidence, nonstandard, or high-risk survival and residual-knowledge clauses to conditional intervention.

What F1 score threshold indicates that first-pass extraction is reliable enough to trust for autonomous filing?

An F1 score above 0.90 means precision and recall are jointly high enough to trust first-pass extraction with page-line citations, while scores around 0.78 leave too many silent misses.

What empirical error rate remains after attorneys perform targeted corrections on LLM flags?

Residual error after targeted attorney correction of LLM flags falls to under 2% overall, according to the MIT-IBM Watson AI Lab contract-review study 2024.

Which three prompt-validation controls form the chain required before deploying the extraction pipeline?

Prompt validation requires a chain of controls including pre-flight checks, gold-label samplers, and escalation rules to maintain operational safety.

Quick answers

Also worth reading: Juris Doctor vs LLM Comparing Key Differences in Legal Education as of 2024: Juris Doctor vs LLM Comparing · New Paddock Unveiled Churchill Downs' $200 Million Upgrade Ahead of 150th Kentucky Derby: New Paddock Unveiled Churchill Downs' · Comparing Local and Online College Experiences A 2024 Perspective on Costs, Courses, and Campus Life: Comparing Local and Online College