Models for judgement. Code for everything that must be exact.
This page explains where language models sit in the system, how they are prompted and constrained, how we evaluate them, and the ways they fail. It is deliberately specific, so that a technical reader can decide whether the design is sound before trusting the output.
What proposal work actually consists of, and which parts need a model
Reading obligation language in context
"Offeror shall" is easy to find lexically. Whether a sentence is a format rule, a staffing requirement, or an evaluation criterion, and what it obliges the offeror to do, requires reading it in context. That is a model task, constrained to a fixed category set and a verbatim span.
Judging fit from prose
Capability statements and past performance are unstructured. Whether a firm's prior work resembles a scope of work is a similarity judgement over text. That is a model task, scored on fixed dimensions with cited reasons, after deterministic rules have already excluded ineligible opportunities.
Writing from evidence
A first draft that addresses a specific requirement using a firm's own documents is language generation. That is a model task, restricted to supplied passages, with a per-sentence citation obligation.
Checking a claim against its evidence
Whether a sentence is entailed by the passages it cites is a reading-comprehension judgement. That is a model task, run in isolation from the draft so it cannot be swayed by context, after exact-match checks on every number and name.
Everything else, from polling to hashing to page counting, is ordinary code. The full split is on the home page.
Two tiers, chosen per task, recorded per call
| Tier | Used for | Why | Settings |
|---|---|---|---|
| Small | Soft fit scoring over hundreds of opportunities per firm per month; category pre-classification | High volume, low stakes: a wrong score costs a minute of the owner's attention, and every score carries reasons | Schema-constrained output, fixed dimensions, low temperature |
| Large | Requirement structuring, drafting, verification, amendment diffs | Low volume, high stakes: these outputs reach a submitted document | Schema-constrained output, temperature zero for extraction and verification, budgets per step |
| Embeddings | Passage retrieval over firm documents and solicitation text | Semantic similarity for retrieval, combined with lexical search so exact terms are never missed | Fixed chunk size with overlap; per-firm index |
Model identifier, prompt version, token counts, and latency are written to the Ledger on every call. Changing a model or a prompt creates a new version; old records keep pointing at what produced them. We do not name specific models on this site because the choice will change, and the Ledger is the source of truth for what ran.
Every call is an envelope with a schema and a budget.
Schema-constrained responses
Each task declares a response schema. The model's output is parsed and validated; anything that fails is discarded and retried with the validation error included, up to a fixed count. Downstream code only ever sees valid records.
Versioned prompts
Prompts are files under version control with a name and date. A prompt change is a release. Evaluation runs against the golden set before a prompt version is allowed into production.
Minimal context by design
Extraction sees one candidate with a window of surrounding text, not the whole document. Verification sees one sentence and its spans, nothing else. Smaller context means fewer places for the model to invent from.
Budgets
Token and time budgets are declared per task and enforced. A per-firm daily budget caps total spend so that a malformed 900-page attachment cannot consume the account.
Extraction call envelope, abbreviated
{
"task": "structure_requirement",
"prompt_version": "extract@2026-09-08",
"model_tier": "large",
"temperature": 0,
"response_schema": "requirement.v1",
"budget": { "max_input_tokens": 6000, "max_output_tokens": 600, "max_ms": 20000 },
"input": {
"candidate": "Offeror shall submit a technical volume not exceeding 15 pages, 12-point font, single-spaced.",
"context_before": "...", "context_after": "...",
"section": "L", "page": 42, "span": [1180, 1274]
}
}Verification call envelope, abbreviated
{
"task": "verify_sentence",
"prompt_version": "verify@2026-09-08",
"model_tier": "large",
"temperature": 0,
"response_schema": "verdict.v1",
"input": {
"sentence": "For the State Department of Labor we designed and ran a twelve-module supervisor curriculum for 340 staff across six sites.",
"spans": [
{ "doc_sha256": "a71e...9d", "page": 1, "text": "Designed and delivered a 12-module supervisor curriculum for 340 staff at six DOL sites (2024)." }
],
"exact_match_precheck": { "numbers": ["340", "6", "12"], "all_found": true }
}
}Grounding the draft in the firm's own words
Chunking
Normalized text is split into passages of a fixed token length with overlap, aligned to paragraph and table boundaries where possible. Each passage keeps its (doc hash, page, char range) so a citation can point into it.
Hybrid search
Queries run against both a lexical index and a vector index, and results are merged and re-ranked. Lexical search guarantees that exact identifiers, contract numbers, and agency names are found; vector search finds paraphrased scope.
Partitioning
Every index is per firm. A retrieval call is bound to one firm's partition when the workflow starts. There is no global index.
How we measure, before we claim anything
We publish no accuracy figures yet, because we have not earned them. This is the method by which we will, and the numbers will appear here with their test set described.
Golden set
Public solicitations with hand-built compliance matrices by people who write proposals for a living. Each requirement in the golden set has a verbatim span, category, and due kind. The set is versioned and grows with every pilot.
Extraction metrics
Requirement recall (did we find it), precision (is it really a requirement), category accuracy, and span accuracy (does the verbatim text match). Recall is the number we care most about, because a missed "shall" is a disqualification.
Citation metrics
Citation support rate: the fraction of drafted sentences whose verification verdict is supported. Flag precision: when we flag a sentence, how often a human agrees it needed a source. Both are measured on pilot bids with human review.
Regression gate
No prompt or model change reaches production unless it matches or improves the golden-set metrics. Results per prompt version are kept alongside the prompt.
How the models fail, and what catches it
Hallucinated facts in a draft
Caught by the citation obligation, the exact-match pre-check on numbers and names, and the independent verification pass. Flagged sentences cannot be exported without acknowledgement.
Missed requirement
Reduced by exhaustive lexical candidate selection over the whole package rather than model-chosen sampling. Not eliminated: requirements phrased without obligation language can be missed, which is why the matrix is editable and additions are versioned.
Wrong category or due date
Reduced by a fixed category set and schema validation; measured on the golden set. Category errors do not change the verbatim text, so the requirement is still in the matrix and visible.
Over-confident fit score
Hard rules run first without a model, so ineligible opportunities never reach scoring. Scores carry cited reasons, and go/no-go remains a human decision recorded as a label.
Verification agreeing with itself
Verification is a separate call with a separate prompt that receives only the sentence and the cited spans. It cannot see the draft, the requirement, or the drafting model's reasoning.
OCR errors on scanned pages
OCR'd pages are marked, and citations into them carry the marker so a reviewer knows to check the original image.
Lines we do not cross
Deciding go or no-go
The system scores and explains. A named person decides, and the decision is recorded under their name.
Predicting win probability
There is no historical outcome data to train on yet. We will not display a number we cannot back.
Pricing
Cost volumes and pricing strategy are yours. The matrix says what the pricing volume must contain; the model does not fill in numbers.
Submitting
No tool can write to an external system. Submission happens outside EveryShall, by you.
Training on customer data
We do not train or fine-tune any model on customer documents, and our providers are contractually barred from doing so.
Silent correction
Unsupported sentences are flagged, never rewritten or removed automatically. The human may hold the missing source.
Bring a technical reviewer to the first call.
We will walk through a live solicitation's matrix, citations, verdicts, and Ledger, and answer any question about the models.