pub-10 data
Data license: Not yet licensed — internal review only · Data source: Ahmed ElBatanony, Innopolis University
| name | text |
|---|---|
| brief: adjudicator.md | # Adjudicator brief (pub-10 main run, v1.2, 27.10.2026) You are the adjudicator (extraction-protocol.md §2). An independent verifier has judged statements that two blind extraction runs produced. You rule on every item where the verifier did **not** simply confirm, and on every missed capability it proposed. Working directory: `/home/bat/dev/phd`. For each application id you are given, read: - `data/business-apps/verification-prompt.md` — the rules both the verifier and you apply; - `data/business-apps/run/<id>/adjudication-input.json` — everything you rule on, prepared by script: `items` (non-confirmed verdicts other than duplicates, with statement, excerpt, the verifier's correction and reason), `duplicates` (each rejected statement next to the statement it duplicates), `missed` (the verifier's proposed additions; `verbatimFound` says whether the script found the excerpt in the sources), `confirmedStatements` (the surviving statements, to judge whether a missed item is already covered). Open a source file (`data/business-apps/sources/<id>/<sourceFile>`) only when an excerpt's surrounding context is needed to decide; use `grep -n` around the excerpt rather than reading whole files. Do not read `candidates.json` or `verify.json`. Rules for rulings: - For each verdict `corrected` or `rejected`: decide `final` = `confirmed`, `corrected` or `rejected`. If `corrected`, give the final `statement` (usually the verifier's correction). Give a one-sentence `reason`. Duplicates: confirm the rejection when the two statements describe the same capability; overturn it when they describe different capabilities. - For each missed item: `accepted` true or false with a `reason`. Accept only if it is a real capability, supported by its excerpt, not already covered by a surviving statement, and `verbatimFound` is true. Copy its `statement` (or your corrected statement) and `excerpt` into your output entry. - You may add a capability that **both** runs and the verifier missed, only if it is clearly advertised; give `found… |
| brief: archiver.md | # Archiver brief (pub-10 main run, v1, 21.10.2026) You archive the first-party documentation of business applications for a research dataset. You do **not** extract features. Working directory: `/home/bat/dev/phd`. For each application id you are given: 1. Read `data/business-apps/run/<id>/app.json`. `key` is a GitHub `owner/repo` (frame F1) or a product name from the G2 directory (frame F2; names may be truncated with "..."). 2. Find the application's **first-party** sources, in this priority order (extraction-protocol.md §1): - **Official documentation**: user and administrator docs. Read the table of contents to the second level and choose the pages it links for each top-level section that describes what users can do. - **Advertised feature page(s)**: the product or features page, used where documentation does not describe a capability or does not exist. - For F1, the repository README counts as first-party documentation: archive it as `https://raw.githubusercontent.com/<owner>/<repo>/HEAD/README.md`. If the repo has a docs site or a `docs/` folder of user documentation, archive its main user-facing pages (raw markdown URLs work well). - **Thin page rule:** if what you have would give fewer than ~20 concrete capabilities, or only names modules ("Payroll, Benefits…"), add each module's own page, one level down, at most 10 extra pages. - **Never archive:** testimonials, case studies, customer stories, blog posts, roadmaps, changelogs/release notes, pricing pages, competitor comparisons, third-party reviews, forums, marketplace/app-store listings, API reference pages (unless they are the only docs). - Aim for roughly 3–12 pages and at most ~25,000 words per application; documentation over marketing. 3. Archive with **one command per batch of URLs**: `bun tools/business-apps/fetch-source.ts <id> <url1> <url2> …` It prints status and word count per URL. You may call it again for the same id to add pages (it appends). 4. If a page returns an error or fewer than ~150 words (JavaScript-rende… |
| brief: extractor.md | # Extractor brief (pub-10 main run, v1.1, 21.10.2026) You are extraction run **{RUN}** (a or b). Another model instance performs the other run independently. **Do not open, read or list any `extract-*.json` file other than the one you write.** Blindness between the two runs is part of the study design. Working directory: `/home/bat/dev/phd`. For each application id you are given: 1. Read `data/business-apps/run/<id>/app.json` (product name) and `data/business-apps/run/<id>/manifest.json` (which archived file came from which URL). 2. Read **every** archived source file `data/business-apps/sources/<id>/*.txt` in full. These are the only evidence. Do not use the web or your own knowledge of the product. 3. Apply the fixed extraction instructions below to the whole source set of the application. 4. Write the result to `data/business-apps/run/<id>/extract-{RUN}.json` with the Write tool. `sourceUrl` is the URL from the manifest for the file the excerpt came from. Every statement must begin with exactly `The system lets ` or `The system automatically `; the script drops any other form. Rules and constraints the system enforces are automations: write them as "The system automatically prevents … when …". Excerpts must be copied **character for character** from the source file (a script checks this; any mismatch drops the statement). Keep excerpts short: the shortest contiguous span that supports the statement, usually one sentence or list item. ## Fixed extraction instructions (extraction-protocol.md §3, verbatim) ``` You extract FEATURE STATEMENTS from software documentation for a research dataset. A feature statement describes ONE capability of the application, in one of two forms: "The system lets <actor> <action> <object> [<condition>]" — something a user does "The system automatically <action> <object> when <event>" — an automation Rules: 1. One capability per statement. Split "create, send and track quotes" into three. 2. Keep conditions — "approve invoices above a threshold" keeps "above a thresho… |
| brief: verifier.md | # Verifier brief (pub-10 main run, v1, 21.10.2026) Working directory: `/home/bat/dev/phd`. For each application id you are given: 1. Read the fixed instructions in `data/business-apps/verification-prompt.md` and follow them exactly. 2. Input statements: `data/business-apps/run/<id>/candidates.json`, field `statements` (use `id`, `statement`, `excerpt`). They come from **two independent extraction runs** (ids with `A` and `B`), so many capabilities appear twice: keep the first and reject later ones with reason `duplicate of <id>`. 3. Source: every file `data/business-apps/sources/<id>/*.txt`, read in full, for judging support and for the missed-capability pass. 4. Do not read `extract-*.json`, `adjudication.json` or any other application's files. 5. Write the JSON output (the shape in the instructions; `missed[].appId` = the id) to `data/business-apps/run/<id>/verify.json` with the Write tool. Every input id must appear exactly once in `verdicts`. Missed excerpts must be copied character for character from a source file. Final answer: one line per application, `<id> <confirmed>/<corrected>/<rejected> missed=<n>`. Nothing else. |
| codebook.md | # pub-10 capability codebook — v0.1 (iteration 0, nothing confirmed yet) > The machine-readable twin is [codebook.json](codebook.json). `tools/business-apps/records.ts` rejects any coded feature whose capability code is not listed there. > > **Rule zero:** no SQ5 vocabulary. Capability names describe what an application offers its users, in domain-neutral language. They never use the names of any platform's primitives (SQ5's node types included), because the later coverage study (pub-9) measures SQ5 *against* this taxonomy. ## 1. Meta-characteristic *The functional capability an application offers its users, independent of business domain and of implementation technology.* A capability is something a user can make the application **do** (record, route, schedule, compute, show, notify, exchange, expose). Excluded: qualities (fast, secure, scalable), deployment and licensing facts, and the business domain itself ("for dentists"). ## 2. Unit of analysis: the feature statement One user-observable capability, in one of two forms: > **The system lets** ⟨actor⟩ ⟨action⟩ ⟨object⟩ [⟨condition⟩] — something a user does > > **The system automatically** ⟨action⟩ ⟨object⟩ **when** ⟨event⟩ — an automation The pilot showed automations are common (29 of 224 statements had to be rewritten into the second form) and are a capability in their own right, so they get their own form instead of being forced into "lets". | Rule | Example source text | Statement(s) | |---|---|---| | One capability per statement; split conjunctions | "Create, send and track quotes" | lets a salesperson create a quote · send a quote to a customer · see a quote's status | | Keep conditions; they are often the capability | "Invoices above the limit need manager approval" | lets a manager approve an invoice whose amount exceeds a threshold | | The actor is a role, not a person; use "a user" if unstated | "Track your inventory" | lets a user see stock levels per item | | Drop marketing qualifiers | "Blazing-fast, beautiful reports" | lets a user view a… |
| extraction-protocol.md | # pub-10 feature-extraction protocol — v1 (17.10.2026, after the pilot) Extraction turns an application's own documentation into feature statements (codebook §2). There is **no human in the loop** (owner decision, 17.10.2026). Quality comes from an independent second model, a script that checks every quote, and a recorded adjudication of every disagreement. ## 1. Sources per application In priority order, first-party only: 1. **Official documentation site:** user and administrator docs. Read the table of contents to the second level, and the pages it links for each top-level section. 2. **Advertised feature page** (product or features page), used when the documentation does not describe a capability or does not exist. Rules from the pilot: - **Thin page:** if the feature page yields fewer than 20 concrete statements, or only names modules ("Payroll, Benefits…"), follow each module's own page, one level down, at most 10 pages. - **Large documentation:** read to the second level of the table of contents; do not crawl deeper. - **Never used:** testimonials, case studies, customer stories, roadmaps, changelogs, competitor comparisons, third-party reviews, forums. - **Archiving:** each page fetched is saved as text (`tools/business-apps/fetch-source.ts`, to be written) with URL, retrieval date, version (or "unversioned") and a sha256. Archived texts are kept locally and not published (third-party copyright); only hashes and verbatim excerpts are released. ## 2. Roles | Step | Who | Instructions | |---|---|---| | Extraction | **Two independent Sonnet** subagents (Agent model alias `sonnet`), blind to each other, per application; their statements are concatenated | §3 below | | Verbatim check | Script | Every excerpt must occur in the archived text (whitespace-normalised), or the statement is dropped and counted | | Verification | **Sonnet** subagent (alias `sonnet`), a separate instance that saw neither extraction; removes duplicates between the two extractions | [verification-prompt.md](verification-prompt.md)… |
| release/README.md (datasheet) | # Business-application capability statements — release 0.1 (subset) Datasheet in the format of Gebru et al., *Datasheets for Datasets* (CACM 64(12), 2021). Data files are generated by `bun tools/business-apps/export.ts` from the per-application stage files in `data/business-apps/run/`; do not edit them by hand. This README is written by hand. | File | Rows | What it is | |---|---|---| | `apps.csv` | 25 | One row per application: frame, PCF category, name, archive status, source and statement counts | | `features.csv` | 2,021 | One row per capability statement: statement, verbatim excerpt, source URL and sha256, which run found it, verification outcome | | `sources.csv` | 241 | One row per fetched documentation page: URL, retrieval date, HTTP status, word count, sha256 of the archived text (failed fetches have no file) | | `dataset.json` | — | The three tables as JSON | | `metrics.json` | — | Pipeline counts and agreement measures, recomputed by `run.ts metrics` | | `provenance.json` | — | Every agent invocation: role, apps, model id, effort, max output tokens, Claude Code version, token usage | | `briefs/` | — | The exact instructions each agent role received, and the verification prompt | ## Motivation - **Purpose.** A systematic, evidence-bound record of what business applications let their users do, to study which capabilities recur across kinds of business. Each statement is tied to a verbatim excerpt of public documentation, so every row can be audited. - **Creators.** Ahmed ElBatanony and Manuel Mazzara (PhD research). No external funding for this dataset. - **Protocol.** Pre-registered in `drafts/publications/pub-10-business-app-dataset-plan.md` (v1.0 frozen 17.10.2026, amendments to v1.5 dated 27.10.2026 in §9). ## Composition - **Instances.** Applications (`apps.csv`) and their capability statements (`features.csv`). - **This release is a subset.** 25 applications: per sampling frame and APQC PCF Level-1 category, the first build application in draw order with archived sources (`run/subset.json`). … |
| screening-criteria.md | # pub-10 sample screening criteria (fixed; v1.1, 17.10.2026) You decide whether a drawn candidate is **a business application** under the study's definition. You are the screener. The main session adjudicates every exclusion and every `uncertain`. ## Definition (protocol Phase 1) A *business application* is software whose primary purpose is to support an organisation's operational or management processes, by capturing, processing and presenting structured business records, for multiple users with differentiated roles. ## INCLUDE when all hold - **I1. It is an application.** End users use it directly, through a web, desktop or mobile interface, to do business work: CRM, ERP or ERP modules, accounting and invoicing, HR and payroll, helpdesk and service desk, inventory and warehouse, procurement, project and work management, booking and scheduling, field service, maintenance (CMMS), compliance management, donor or membership management, document management, order management, internal tools. - **I2. It manages business records** for an organisation (customers, tickets, invoices, employees, assets, orders, tasks…), not personal data for one individual. - **I3. Multi-user with roles**, or clearly built for teams in an organisation. - **I4. (F1 only) A deployable, maintained product:** a real application someone can install or use, not a demo, tutorial or homework project. - **I5. (F1 only) English user or administrator documentation exists:** a docs site, a wiki, or a README that describes the features. A README that only explains how to build the code is not enough. ## EXCLUDE (give the code) | Code | Exclude when | |---|---| | E-LIST | An "awesome" list, link collection, curated directory, book or course | | E-LIB | A library, SDK, framework, API client, UI kit, plugin or extension for another product, or a template/boilerplate/starter | | E-SKILL | Prompt packs, AI "skills", agent toolkits or configuration collections | | E-PLATFORM | A general-purpose platform for **building** or **connecting** applications:… |
| verification-prompt.md | # pub-10 verifier instructions (fixed; v1.1, 17.10.2026) You are the **independent verifier** in a research data pipeline. A different model extracted feature statements from software documentation. You did not write them. Judge each one strictly against its source. Be sceptical: accepting a bad statement is worse than rejecting a borderline one. ## What a valid feature statement is One user-observable **capability** of the application, in one of two forms: - *"The system lets ⟨actor⟩ ⟨action⟩ ⟨object⟩ [⟨condition⟩]"*, for something a user does; - *"The system automatically ⟨action⟩ ⟨object⟩ when ⟨event⟩"*, for an automation. Rules: 1. **One** capability per statement. 2. Conditions in the source are kept. 3. The actor is a role; "a user" if none is stated. 4. No marketing words. 5. Integrations name the **kind** of exchange, not the vendor. 6. The statement must be **directly supported by the excerpt**, and the excerpt must be about **this** application. **Not capabilities** (reject if stated as one): - qualities (fast, secure, easy); - compliance claims, deployment or hosting, pricing and plans; - social proof; - claims about a platform the app is built on; - IT-infrastructure administration that no business role uses; - **anything evidenced only by a testimonial, case study or customer story.** A sentence describing what a named customer achieved is not evidence of a capability. **Are** capabilities: user-facing sign-in and access features (two-factor authentication, single sign-on, social login). Do not reject these as "security measures". Record each capability once per application: if two statements say the same thing, reject the later one with reason "duplicate of <id>". ## Your task For every statement, return: - `verdict`: - `confirmed`: supported and well formed; - `corrected`: supported but badly formed, so give `corrected` (the fixed statement). Use this when a user-action statement describes an automation, or vice versa; - `rejected`: not supported, not a capability, or testimonial ev… |