{"database": "pub10", "table": "documents", "rows": [["extraction-protocol.md", "# pub-10 feature-extraction protocol \u2014 v1 (17.10.2026, after the pilot)\n\nExtraction turns an application's own documentation into feature statements (codebook \u00a72). There is **no human in the loop** (owner decision, 17.10.2026). Quality comes from an independent second model, a script that checks every quote, and a recorded adjudication of every disagreement.\n\n## 1. Sources per application\n\nIn priority order, first-party only:\n\n1. **Official documentation site:** user and administrator docs. Read the table of contents to the second level, and the pages it links for each top-level section.\n2. **Advertised feature page** (product or features page), used when the documentation does not describe a capability or does not exist.\n\nRules from the pilot:\n\n- **Thin page:** if the feature page yields fewer than 20 concrete statements, or only names modules (\"Payroll, Benefits\u2026\"), follow each module's own page, one level down, at most 10 pages.\n- **Large documentation:** read to the second level of the table of contents; do not crawl deeper.\n- **Never used:** testimonials, case studies, customer stories, roadmaps, changelogs, competitor comparisons, third-party reviews, forums.\n- **Archiving:** each page fetched is saved as text (`tools/business-apps/fetch-source.ts`, to be written) with URL, retrieval date, version (or \"unversioned\") and a sha256. Archived texts are kept locally and not published (third-party copyright); only hashes and verbatim excerpts are released.\n\n## 2. Roles\n\n| Step | Who | Instructions |\n|---|---|---|\n| Extraction | **Two independent Sonnet** subagents (Agent model alias `sonnet`), blind to each other, per application; their statements are concatenated | \u00a73 below |\n| Verbatim check | Script | Every excerpt must occur in the archived text (whitespace-normalised), or the statement is dropped and counted |\n| Verification | **Sonnet** subagent (alias `sonnet`), a separate instance that saw neither extraction; removes duplicates between the two extractions | [verification-prompt.md](verification-prompt.md) |\n| Adjudication | The main Opus session | Rules on every `corrected`, `rejected` and missed item, recording its reasoning in `adjudication.json` |\n| Escalation | The owner | Only items the adjudicator marks `undecidable` |\n\nModel choice follows the owner's rule (17.10.2026): clearly defined steps run on Sonnet; Opus is kept for judgement (adjudication, taxonomy building). \u00a75 shows the Sonnet setup reaches 98.2% of the capabilities an Opus extraction finds.\n\n## 3. The extraction prompt (fixed)\n\n```\nYou extract FEATURE STATEMENTS from software documentation for a research dataset.\n\nA feature statement describes ONE capability of the application, in one of two forms:\n  \"The system lets <actor> <action> <object> [<condition>]\"   \u2014 something a user does\n  \"The system automatically <action> <object> when <event>\"   \u2014 an automation\n\nRules:\n1. One capability per statement. Split \"create, send and track quotes\" into three.\n2. Keep conditions \u2014 \"approve invoices above a threshold\" keeps \"above a threshold\".\n3. The actor is a role (\"a manager\", \"a customer\"); use \"a user\" if none is stated.\n4. Drop marketing words (fast, beautiful, powerful, seamless).\n5. Name the KIND of integration, not the vendor: \"exchange accounting records with an\n   external accounting system\", not \"sync with QuickBooks\".\n6. Record each capability ONCE per application, even if the source repeats it.\n7. Never infer beyond the excerpt. Every statement must be directly supported by the\n   excerpt you quote, copied EXACTLY from the source.\n8. Do NOT write statements for: qualities, compliance claims, deployment or hosting,\n   pricing or plans, social proof, TESTIMONIALS OR CUSTOMER STORIES, claims about a\n   platform the app is built on, or IT-infrastructure administration no business role\n   uses. List those excerpts under \"skipped\" with a reason.\n   User-facing sign-in features (two-factor, single sign-on) ARE capabilities.\n\nReturn JSON only:\n{\"features\":[{\"statement\":\"...\",\"excerpt\":\"<verbatim source text>\",\"sourceUrl\":\"...\"}],\n \"skipped\":[{\"excerpt\":\"...\",\"reason\":\"quality|compliance|deployment|pricing|social-proof|testimonial|platform|other\"}]}\n```\n\n## 4. Quality measures reported in the paper\n\nComputed by script; see `pilot/verification/metrics.json` for the pilot's values.\n\n| Measure | Definition | Pilot (F1+F2, 224 statements) |\n|---|---|---|\n| Supported share | statements not rejected \u00f7 extracted | 97.8% |\n| Strict precision | unchanged or form-only corrections \u00f7 extracted | 96.0% |\n| Missed share | statements added at verification \u00f7 final set | 2.2% |\n| Verifier\u2013adjudicator agreement | Cohen's \u03ba on confirmed / corrected / rejected | 0.985 |\n| Verbatim failures | excerpts not found in the source \u00f7 extracted | 0 of 317 |\n\nThese are **inter-model** measures: they show consistency, not ground truth. That limitation is stated in the paper.\n\n## 5. Calibration before the main run\n\nA blind Sonnet extraction of two pilot applications is compared with the adjudicated pilot set. Report:\n- the share of adjudicated capabilities Sonnet also found (recall against the pilot set);\n- the share of Sonnet statements matching an adjudicated capability (precision against it);\n- verbatim failures.\n\nIf Sonnet's recall is clearly below the pilot extractor's, the extractor role reverts to Opus.\n\n**Result (17.10.2026):** Sonnet recalled 92.7% of the adjudicated capabilities (101 of 109), against about 99% for Opus on the same apps, with no verbatim failures and almost all extra statements being finer splits. That ruling was superseded by round 2 below. Details are in `pilot/calibration/result.json`.\n\n**Round 2 (17.10.2026, after the owner's model rule):**\n\n| Setup | Recall against the adjudicated set |\n|---|---:|\n| Opus extraction | 99.1% |\n| Sonnet extraction alone | 92.7% |\n| Sonnet extraction + Sonnet verifier's missed pass | 95.4% |\n| **Two blind Sonnet extractions + Sonnet verifier** | **98.2%** |\n\nThe last setup is adopted. Two independent runs miss different capabilities, and the union closes most of the gap at a fraction of Opus's cost. The residual gap (about 1 capability per 100) is reported as a limitation.\n"]], "columns": ["name", "text"], "primary_keys": ["name"], "primary_key_values": ["extraction-protocol.md"], "units": {}, "query_ms": 1.9153431057929993, "source": "Ahmed ElBatanony, Innopolis University", "license": "Not yet licensed \u2014 internal review only"}