documents: release/README.md (datasheet)

The exact instructions every agent role received (briefs), the verification prompt, the extraction protocol, screening criteria and the release datasheet.

Data license: Not yet licensed — internal review only · Data source: Ahmed ElBatanony, Innopolis University

This data as json

name	text
release/README.md (datasheet)	# Business-application capability statements — release 0.1 (subset) Datasheet in the format of Gebru et al., Datasheets for Datasets (CACM 64(12), 2021). Data files are generated by `bun tools/business-apps/export.ts` from the per-application stage files in `data/business-apps/run/`; do not edit them by hand. This README is written by hand. \| File \| Rows \| What it is \| \|---\|---\|---\| \| `apps.csv` \| 25 \| One row per application: frame, PCF category, name, archive status, source and statement counts \| \| `features.csv` \| 2,021 \| One row per capability statement: statement, verbatim excerpt, source URL and sha256, which run found it, verification outcome \| \| `sources.csv` \| 241 \| One row per fetched documentation page: URL, retrieval date, HTTP status, word count, sha256 of the archived text (failed fetches have no file) \| \| `dataset.json` \| — \| The three tables as JSON \| \| `metrics.json` \| — \| Pipeline counts and agreement measures, recomputed by `run.ts metrics` \| \| `provenance.json` \| — \| Every agent invocation: role, apps, model id, effort, max output tokens, Claude Code version, token usage \| \| `briefs/` \| — \| The exact instructions each agent role received, and the verification prompt \| ## Motivation - Purpose. A systematic, evidence-bound record of what business applications let their users do, to study which capabilities recur across kinds of business. Each statement is tied to a verbatim excerpt of public documentation, so every row can be audited. - Creators. Ahmed ElBatanony and Manuel Mazzara (PhD research). No external funding for this dataset. - Protocol. Pre-registered in `drafts/publications/pub-10-business-app-dataset-plan.md` (v1.0 frozen 17.10.2026, amendments to v1.5 dated 27.10.2026 in §9). ## Composition - Instances. Applications (`apps.csv`) and their capability statements (`features.csv`). - This release is a subset. 25 applications: per sampling frame and APQC PCF Level-1 category, the first build application in draw order with archived sources (`run/subset.json`). F1 contributes 12 apps (its final sample has no application in PCF 12.0 Manage External Relationships), F2 13. The remaining build applications will be added in a later release. - Frames. F1: open-source business applications on GitHub (topic search, ≥ 500 stars, recent commit). F2: commercial products from G2 category listings. - Statement forms. Every statement begins with `The system lets …` (a user action, 1,703 rows) or `The system automatically …` (an automation or enforced rule, 318 rows). - Labels. `found_by`: `extract-a`, `extract-b` (the blind extraction run whose wording was kept), `verifier` or `adjudicator` (a capability both runs missed). `found_by_runs`: `both`, `a`, `b` or `neither`, from the verifier's duplicate judgements. `verification`: `confirmed` (1,895), `corrected` (77, statement reworded by the verifier/adjudicator), `added` (49). - Not included. The archived source texts. They are vendor documentation under the vendors' copyright; the release keeps only short excerpts, the URL, the retrieval date and the sha256 of the archived text, so anyone can re-fetch a page and check whether it changed. - Known gaps. Three apps are archived thin (F1-029, F1-034, F2-037; few or short pages), so their statement counts understate the product. Statements describe what the documentation advertises, not what the software was observed to do. No application was run or tested. - Personal data. None. Statements and excerpts are about software products; testimonials and customer stories were excluded by rule. ## Collection process - Sources. Vendor websites, help centres and READMEs, chosen by the archiver agent and fetched once on 21.10.2026 by `tools/business-apps/fetch-source.ts` (HTML converted to plain text; sha256 over that text). - Pipeline per application. 1. Two blind extraction runs (A and B) apply the fixed prompt in `briefs/extractor.md` to all archived pages; neither sees the other's output. 2. A script keeps only statements whose excerpt appears verbatim in the archived text and whose form matches `^The system (lets\|automatically) \S` (0 of 3,819 dropped in this subset). 3. A verifier agent judges every statement (confirm, correct, reject, incl. duplicates across runs) and proposes missed capabilities (`briefs/verification-prompt.md`). 4. An adjudicator agent rules on every non-confirmed verdict and every proposed addition (`briefs/adjudicator.md`). 5. `run.ts finalize` merges the rulings into the statement set. - Models and settings. Archiver, extractors and verifier: `claude-sonnet-5-5`; adjudicator: `claude-opus-5-5`; effort `medium`; max output 128,000 tokens; run as Claude Code 2.1.304/2.1.308 subagents on 21.–28.10.2026. Temperature and sampling seed are not exposed by the runtime, so a re-run will not reproduce the same statements; the A-vs-B agreement in `metrics.json` measures that run-to-run variation (the runs agreed on 79% of extracted capabilities; each run alone covered 89% and 86% of the final set). `provenance.json` also lists agent calls that failed on rate limits (model `<synthetic>`); their partial output was discarded and the app re-run. - Humans. No human annotation in this release. The owner set the protocol and standing rulings; no statement was hand-edited. - Blind audit. 150 stratified random rows were judged by a separate Opus instance that saw only statement, excerpt and source page (`data/business-apps/audit/`): 97.3% acceptable as is (95% CI 94.4–100%), but only 62.2% supported by the excerpt alone (99.7% in context). Treat excerpts as pointers into the page, not as complete evidence. Same vendor as the pipeline; not a human audit. ## Preprocessing HTML → text conversion only. Statements are as written by the extraction agents, or as corrected by the verifier/adjudicator. `pipeline_id` links each row back to `run/<app>/candidates.json`. ## Uses - Intended. Studying the distribution of capabilities across kinds of business applications; building and testing a capability codebook; benchmarking information extraction from software documentation. - Not intended. Comparing or ranking vendors; claims about what a product actually does (only what its documentation says, on 21.10.2026). ## Distribution and licence - To be released on Zenodo with a DOI alongside the paper. - Licence: to be confirmed by the owner (proposed: CC BY 4.0 for the statements and metadata). Excerpts are short quotations of vendor documentation, included for verification. PCF category names: APQC Process Classification Framework® Cross-Industry v7.4, an open standard by APQC (www.apqc.org/pcf). ## Maintenance Maintained by Ahmed ElBatanony. Later releases add the remaining build applications and keep row ids stable (`feature_id` = `<app>-S<nnn>` in final-set order; `pipeline_id` is the permanent link to the pipeline record). ## Reproducing the numbers ```bash bun tools/business-apps/run.ts metrics bun tools/business-apps/export.ts bun tools/business-apps/tables.ts ```

name

text

release/README.md (datasheet)

# Business-application capability statements — release 0.1 (subset) Datasheet in the format of Gebru et al., *Datasheets for Datasets* (CACM 64(12), 2021). Data files are generated by `bun tools/business-apps/export.ts` from the per-application stage files in `data/business-apps/run/`; do not edit them by hand. This README is written by hand. | File | Rows | What it is | |---|---|---| | `apps.csv` | 25 | One row per application: frame, PCF category, name, archive status, source and statement counts | | `features.csv` | 2,021 | One row per capability statement: statement, verbatim excerpt, source URL and sha256, which run found it, verification outcome | | `sources.csv` | 241 | One row per fetched documentation page: URL, retrieval date, HTTP status, word count, sha256 of the archived text (failed fetches have no file) | | `dataset.json` | — | The three tables as JSON | | `metrics.json` | — | Pipeline counts and agreement measures, recomputed by `run.ts metrics` | | `provenance.json` | — | Every agent invocation: role, apps, model id, effort, max output tokens, Claude Code version, token usage | | `briefs/` | — | The exact instructions each agent role received, and the verification prompt | ## Motivation - **Purpose.** A systematic, evidence-bound record of what business applications let their users do, to study which capabilities recur across kinds of business. Each statement is tied to a verbatim excerpt of public documentation, so every row can be audited. - **Creators.** Ahmed ElBatanony and Manuel Mazzara (PhD research). No external funding for this dataset. - **Protocol.** Pre-registered in `drafts/publications/pub-10-business-app-dataset-plan.md` (v1.0 frozen 17.10.2026, amendments to v1.5 dated 27.10.2026 in §9). ## Composition - **Instances.** Applications (`apps.csv`) and their capability statements (`features.csv`). - **This release is a subset.** 25 applications: per sampling frame and APQC PCF Level-1 category, the first build application in draw order with archived sources (`run/subset.json`). F1 contributes 12 apps (its final sample has no application in PCF 12.0 *Manage External Relationships*), F2 13. The remaining build applications will be added in a later release. - **Frames.** F1: open-source business applications on GitHub (topic search, ≥ 500 stars, recent commit). F2: commercial products from G2 category listings. - **Statement forms.** Every statement begins with `The system lets …` (a user action, 1,703 rows) or `The system automatically …` (an automation or enforced rule, 318 rows). - **Labels.** `found_by`: `extract-a`, `extract-b` (the blind extraction run whose wording was kept), `verifier` or `adjudicator` (a capability both runs missed). `found_by_runs`: `both`, `a`, `b` or `neither`, from the verifier's duplicate judgements. `verification`: `confirmed` (1,895), `corrected` (77, statement reworded by the verifier/adjudicator), `added` (49). - **Not included.** The archived source texts. They are vendor documentation under the vendors' copyright; the release keeps only short excerpts, the URL, the retrieval date and the sha256 of the archived text, so anyone can re-fetch a page and check whether it changed. - **Known gaps.** Three apps are archived thin (F1-029, F1-034, F2-037; few or short pages), so their statement counts understate the product. Statements describe what the documentation *advertises*, not what the software was observed to do. No application was run or tested. - **Personal data.** None. Statements and excerpts are about software products; testimonials and customer stories were excluded by rule. ## Collection process - **Sources.** Vendor websites, help centres and READMEs, chosen by the archiver agent and fetched once on 21.10.2026 by `tools/business-apps/fetch-source.ts` (HTML converted to plain text; sha256 over that text). - **Pipeline per application.** 1. Two **blind** extraction runs (A and B) apply the fixed prompt in `briefs/extractor.md` to all archived pages; neither sees the other's output. 2. A script keeps only statements whose excerpt appears verbatim in the archived text and whose form matches `^The system (lets|automatically) \S` (0 of 3,819 dropped in this subset). 3. A verifier agent judges every statement (confirm, correct, reject, incl. duplicates across runs) and proposes missed capabilities (`briefs/verification-prompt.md`). 4. An adjudicator agent rules on every non-confirmed verdict and every proposed addition (`briefs/adjudicator.md`). 5. `run.ts finalize` merges the rulings into the statement set. - **Models and settings.** Archiver, extractors and verifier: `claude-sonnet-5-5`; adjudicator: `claude-opus-5-5`; effort `medium`; max output 128,000 tokens; run as Claude Code 2.1.304/2.1.308 subagents on 21.–28.10.2026. Temperature and sampling seed are not exposed by the runtime, so a re-run will not reproduce the same statements; the A-vs-B agreement in `metrics.json` measures that run-to-run variation (the runs agreed on 79% of extracted capabilities; each run alone covered 89% and 86% of the final set). `provenance.json` also lists agent calls that failed on rate limits (model `<synthetic>`); their partial output was discarded and the app re-run. - **Humans.** No human annotation in this release. The owner set the protocol and standing rulings; no statement was hand-edited. - **Blind audit.** 150 stratified random rows were judged by a separate Opus instance that saw only statement, excerpt and source page (`data/business-apps/audit/`): 97.3% acceptable as is (95% CI 94.4–100%), but only 62.2% supported by the excerpt alone (99.7% in context). Treat excerpts as pointers into the page, not as complete evidence. Same vendor as the pipeline; not a human audit. ## Preprocessing HTML → text conversion only. Statements are as written by the extraction agents, or as corrected by the verifier/adjudicator. `pipeline_id` links each row back to `run/<app>/candidates.json`. ## Uses - **Intended.** Studying the distribution of capabilities across kinds of business applications; building and testing a capability codebook; benchmarking information extraction from software documentation. - **Not intended.** Comparing or ranking vendors; claims about what a product actually does (only what its documentation says, on 21.10.2026). ## Distribution and licence - To be released on Zenodo with a DOI alongside the paper. - **Licence: to be confirmed by the owner** (proposed: CC BY 4.0 for the statements and metadata). Excerpts are short quotations of vendor documentation, included for verification. PCF category names: APQC Process Classification Framework® Cross-Industry v7.4, an open standard by APQC (www.apqc.org/pcf). ## Maintenance Maintained by Ahmed ElBatanony. Later releases add the remaining build applications and keep row ids stable (`feature_id` = `<app>-S<nnn>` in final-set order; `pipeline_id` is the permanent link to the pipeline record). ## Reproducing the numbers ```bash bun tools/business-apps/run.ts metrics bun tools/business-apps/export.ts bun tools/business-apps/tables.ts ```