documents: extraction-protocol.md

The exact instructions every agent role received (briefs), the verification prompt, the extraction protocol, screening criteria and the release datasheet.

Data license: Not yet licensed — internal review only · Data source: Ahmed ElBatanony, Innopolis University

This data as json

name	text
extraction-protocol.md	# pub-10 feature-extraction protocol — v1 (17.10.2026, after the pilot) Extraction turns an application's own documentation into feature statements (codebook §2). There is no human in the loop (owner decision, 17.10.2026). Quality comes from an independent second model, a script that checks every quote, and a recorded adjudication of every disagreement. ## 1. Sources per application In priority order, first-party only: 1. Official documentation site: user and administrator docs. Read the table of contents to the second level, and the pages it links for each top-level section. 2. Advertised feature page (product or features page), used when the documentation does not describe a capability or does not exist. Rules from the pilot: - Thin page: if the feature page yields fewer than 20 concrete statements, or only names modules ("Payroll, Benefits…"), follow each module's own page, one level down, at most 10 pages. - Large documentation: read to the second level of the table of contents; do not crawl deeper. - Never used: testimonials, case studies, customer stories, roadmaps, changelogs, competitor comparisons, third-party reviews, forums. - Archiving: each page fetched is saved as text (`tools/business-apps/fetch-source.ts`, to be written) with URL, retrieval date, version (or "unversioned") and a sha256. Archived texts are kept locally and not published (third-party copyright); only hashes and verbatim excerpts are released. ## 2. Roles \| Step \| Who \| Instructions \| \|---\|---\|---\| \| Extraction \| Two independent Sonnet subagents (Agent model alias `sonnet`), blind to each other, per application; their statements are concatenated \| §3 below \| \| Verbatim check \| Script \| Every excerpt must occur in the archived text (whitespace-normalised), or the statement is dropped and counted \| \| Verification \| Sonnet subagent (alias `sonnet`), a separate instance that saw neither extraction; removes duplicates between the two extractions \| [verification-prompt.md](verification-prompt.md) \| \| Adjudication \| The main Opus session \| Rules on every `corrected`, `rejected` and missed item, recording its reasoning in `adjudication.json` \| \| Escalation \| The owner \| Only items the adjudicator marks `undecidable` \| Model choice follows the owner's rule (17.10.2026): clearly defined steps run on Sonnet; Opus is kept for judgement (adjudication, taxonomy building). §5 shows the Sonnet setup reaches 98.2% of the capabilities an Opus extraction finds. ## 3. The extraction prompt (fixed) ``` You extract FEATURE STATEMENTS from software documentation for a research dataset. A feature statement describes ONE capability of the application, in one of two forms: "The system lets <actor> <action> <object> [<condition>]" — something a user does "The system automatically <action> <object> when <event>" — an automation Rules: 1. One capability per statement. Split "create, send and track quotes" into three. 2. Keep conditions — "approve invoices above a threshold" keeps "above a threshold". 3. The actor is a role ("a manager", "a customer"); use "a user" if none is stated. 4. Drop marketing words (fast, beautiful, powerful, seamless). 5. Name the KIND of integration, not the vendor: "exchange accounting records with an external accounting system", not "sync with QuickBooks". 6. Record each capability ONCE per application, even if the source repeats it. 7. Never infer beyond the excerpt. Every statement must be directly supported by the excerpt you quote, copied EXACTLY from the source. 8. Do NOT write statements for: qualities, compliance claims, deployment or hosting, pricing or plans, social proof, TESTIMONIALS OR CUSTOMER STORIES, claims about a platform the app is built on, or IT-infrastructure administration no business role uses. List those excerpts under "skipped" with a reason. User-facing sign-in features (two-factor, single sign-on) ARE capabilities. Return JSON only: {"features":[{"statement":"...","excerpt":"<verbatim source text>","sourceUrl":"..."}], "skipped":[{"excerpt":"...","reason":"quality\|compliance\|deployment\|pricing\|social-proof\|testimonial\|platform\|other"}]} ``` ## 4. Quality measures reported in the paper Computed by script; see `pilot/verification/metrics.json` for the pilot's values. \| Measure \| Definition \| Pilot (F1+F2, 224 statements) \| \|---\|---\|---\| \| Supported share \| statements not rejected ÷ extracted \| 97.8% \| \| Strict precision \| unchanged or form-only corrections ÷ extracted \| 96.0% \| \| Missed share \| statements added at verification ÷ final set \| 2.2% \| \| Verifier–adjudicator agreement \| Cohen's κ on confirmed / corrected / rejected \| 0.985 \| \| Verbatim failures \| excerpts not found in the source ÷ extracted \| 0 of 317 \| These are inter-model measures: they show consistency, not ground truth. That limitation is stated in the paper. ## 5. Calibration before the main run A blind Sonnet extraction of two pilot applications is compared with the adjudicated pilot set. Report: - the share of adjudicated capabilities Sonnet also found (recall against the pilot set); - the share of Sonnet statements matching an adjudicated capability (precision against it); - verbatim failures. If Sonnet's recall is clearly below the pilot extractor's, the extractor role reverts to Opus. Result (17.10.2026): Sonnet recalled 92.7% of the adjudicated capabilities (101 of 109), against about 99% for Opus on the same apps, with no verbatim failures and almost all extra statements being finer splits. That ruling was superseded by round 2 below. Details are in `pilot/calibration/result.json`. Round 2 (17.10.2026, after the owner's model rule): \| Setup \| Recall against the adjudicated set \| \|---\|---:\| \| Opus extraction \| 99.1% \| \| Sonnet extraction alone \| 92.7% \| \| Sonnet extraction + Sonnet verifier's missed pass \| 95.4% \| \| Two blind Sonnet extractions + Sonnet verifier \| 98.2% \| The last setup is adopted. Two independent runs miss different capabilities, and the union closes most of the gap at a fraction of Opus's cost. The residual gap (about 1 capability per 100) is reported as a limitation.

name

text

extraction-protocol.md

# pub-10 feature-extraction protocol — v1 (17.10.2026, after the pilot) Extraction turns an application's own documentation into feature statements (codebook §2). There is **no human in the loop** (owner decision, 17.10.2026). Quality comes from an independent second model, a script that checks every quote, and a recorded adjudication of every disagreement. ## 1. Sources per application In priority order, first-party only: 1. **Official documentation site:** user and administrator docs. Read the table of contents to the second level, and the pages it links for each top-level section. 2. **Advertised feature page** (product or features page), used when the documentation does not describe a capability or does not exist. Rules from the pilot: - **Thin page:** if the feature page yields fewer than 20 concrete statements, or only names modules ("Payroll, Benefits…"), follow each module's own page, one level down, at most 10 pages. - **Large documentation:** read to the second level of the table of contents; do not crawl deeper. - **Never used:** testimonials, case studies, customer stories, roadmaps, changelogs, competitor comparisons, third-party reviews, forums. - **Archiving:** each page fetched is saved as text (`tools/business-apps/fetch-source.ts`, to be written) with URL, retrieval date, version (or "unversioned") and a sha256. Archived texts are kept locally and not published (third-party copyright); only hashes and verbatim excerpts are released. ## 2. Roles | Step | Who | Instructions | |---|---|---| | Extraction | **Two independent Sonnet** subagents (Agent model alias `sonnet`), blind to each other, per application; their statements are concatenated | §3 below | | Verbatim check | Script | Every excerpt must occur in the archived text (whitespace-normalised), or the statement is dropped and counted | | Verification | **Sonnet** subagent (alias `sonnet`), a separate instance that saw neither extraction; removes duplicates between the two extractions | [verification-prompt.md](verification-prompt.md) | | Adjudication | The main Opus session | Rules on every `corrected`, `rejected` and missed item, recording its reasoning in `adjudication.json` | | Escalation | The owner | Only items the adjudicator marks `undecidable` | Model choice follows the owner's rule (17.10.2026): clearly defined steps run on Sonnet; Opus is kept for judgement (adjudication, taxonomy building). §5 shows the Sonnet setup reaches 98.2% of the capabilities an Opus extraction finds. ## 3. The extraction prompt (fixed) ``` You extract FEATURE STATEMENTS from software documentation for a research dataset. A feature statement describes ONE capability of the application, in one of two forms: "The system lets <actor> <action> <object> [<condition>]" — something a user does "The system automatically <action> <object> when <event>" — an automation Rules: 1. One capability per statement. Split "create, send and track quotes" into three. 2. Keep conditions — "approve invoices above a threshold" keeps "above a threshold". 3. The actor is a role ("a manager", "a customer"); use "a user" if none is stated. 4. Drop marketing words (fast, beautiful, powerful, seamless). 5. Name the KIND of integration, not the vendor: "exchange accounting records with an external accounting system", not "sync with QuickBooks". 6. Record each capability ONCE per application, even if the source repeats it. 7. Never infer beyond the excerpt. Every statement must be directly supported by the excerpt you quote, copied EXACTLY from the source. 8. Do NOT write statements for: qualities, compliance claims, deployment or hosting, pricing or plans, social proof, TESTIMONIALS OR CUSTOMER STORIES, claims about a platform the app is built on, or IT-infrastructure administration no business role uses. List those excerpts under "skipped" with a reason. User-facing sign-in features (two-factor, single sign-on) ARE capabilities. Return JSON only: {"features":[{"statement":"...","excerpt":"<verbatim source text>","sourceUrl":"..."}], "skipped":[{"excerpt":"...","reason":"quality|compliance|deployment|pricing|social-proof|testimonial|platform|other"}]} ``` ## 4. Quality measures reported in the paper Computed by script; see `pilot/verification/metrics.json` for the pilot's values. | Measure | Definition | Pilot (F1+F2, 224 statements) | |---|---|---| | Supported share | statements not rejected ÷ extracted | 97.8% | | Strict precision | unchanged or form-only corrections ÷ extracted | 96.0% | | Missed share | statements added at verification ÷ final set | 2.2% | | Verifier–adjudicator agreement | Cohen's κ on confirmed / corrected / rejected | 0.985 | | Verbatim failures | excerpts not found in the source ÷ extracted | 0 of 317 | These are **inter-model** measures: they show consistency, not ground truth. That limitation is stated in the paper. ## 5. Calibration before the main run A blind Sonnet extraction of two pilot applications is compared with the adjudicated pilot set. Report: - the share of adjudicated capabilities Sonnet also found (recall against the pilot set); - the share of Sonnet statements matching an adjudicated capability (precision against it); - verbatim failures. If Sonnet's recall is clearly below the pilot extractor's, the extractor role reverts to Opus. **Result (17.10.2026):** Sonnet recalled 92.7% of the adjudicated capabilities (101 of 109), against about 99% for Opus on the same apps, with no verbatim failures and almost all extra statements being finer splits. That ruling was superseded by round 2 below. Details are in `pilot/calibration/result.json`. **Round 2 (17.10.2026, after the owner's model rule):** | Setup | Recall against the adjudicated set | |---|---:| | Opus extraction | 99.1% | | Sonnet extraction alone | 92.7% | | Sonnet extraction + Sonnet verifier's missed pass | 95.4% | | **Two blind Sonnet extractions + Sonnet verifier** | **98.2%** | The last setup is adopted. Two independent runs miss different capabilities, and the union closes most of the gap at a fraction of Opus's cost. The residual gap (about 1 capability per 100) is reported as a limitation.