Business applications and what they let users do
The data behind the draft “Trust, but Audit: Measuring an LLM Agent Pipeline that Extracts Software Capabilities from Documentation” (RELIABLE-SE 2027) and the pub-10 dataset release. Every number in the paper can be traced to rows here.
Read the paper draft (PDF, 4 pages) · Download the release files (CSV/JSON + datasheet) · Prompts and briefs
How a statement gets into the dataset
All steps except the script check are Claude models under fixed briefs (Sonnet 5.5 for archiving, extraction and verification; Opus 5.5 for adjudication; medium effort). Every agent call is listed in provenance.
Start here
The quickest way to judge the data. Reload for another sample.
All final statements →Filter by application, verification outcome, form, or which run found it.
Blind audit →An independent Opus pass over 150 random rows, without pipeline verdicts.
Rows the audit rejected →Each with the auditor's note: which check failed and why.
One app, every step →Each candidate from both runs, with the verifier's and adjudicator's reasons.
Corrections →Original vs final wording for the 77 reworded statements.
Rejections →The 60 statements rejected for a reason other than duplication.
Missed capabilities →56 proposed by the verifier, 49 accepted.
Deliberately skipped →Marketing, testimonials, pricing, platform claims: what the extractors set aside, and why.
Run A vs run B →Per application: found by both, by one run only, or added.
Sample screening →576 candidate applications, each with the screener's verdict and reason.
All metrics →Every pipeline count, recomputed from the stage files.
Pilot →The 317 pilot statements behind Tables 1–2 of the paper.
Three examples
The system lets a user create a purchase order specifying the type and quantity of parts to order from a vendor.
The system automatically calculates the total man hours of a project when the project team is set.
The system lets a user benchmark business performance against the industry.
The 25 applications
| App | Name | PCF category | Pages | Statements | Both runs | A only | B only | Added |
|---|---|---|---|---|---|---|---|---|
| F1-001 | Operately | 1.0 Develop Vision and Strategy | 14 | 102 | 82 | 4 | 13 | 3 |
| F1-002 | It's a Plan | 2.0 Develop and Manage Products and Services | 8 | 86 | 57 | 12 | 14 | 3 |
| F1-005 | DeskcommCRM | 3.0 Market and Sell Products and Services | 5 | 68 | 47 | 6 | 11 | 4 |
| F1-011 | Asp.Net Core Inventory Order Management System (Indotalent WMS) | 4.0 Manage Supply Chain for Physical Products | 3 | 37 | 36 | 0 | 1 | 0 |
| F1-017 | ZenTao (now branded Sanplex) | 5.0 Deliver Services | 13 | 80 | 58 | 16 | 5 | 1 |
| F1-023 | Helper | 6.0 Manage Customer Service | 4 | 19 | 17 | 0 | 2 | 0 |
| F1-029 | Wukong HRM | 7.0 Develop and Manage Human Capital | 4 | 16 | 15 | 1 | 0 | 0 |
| F1-034 | WeCMDB | 8.0 Manage Information Technology (IT) | 4 | 36 | 28 | 4 | 2 | 2 |
| F1-037 | FOSSBilling | 9.0 Manage Financial Resources | 11 | 46 | 33 | 7 | 6 | 0 |
| F1-043 | QloApps | 10.0 Acquire, Construct, and Manage Assets | 5 | 47 | 44 | 3 | 0 | 0 |
| F1-046 | Probo | 11.0 Manage Enterprise Risk, Compliance, Remediation, and Resiliency | 12 | 86 | 52 | 27 | 3 | 4 |
| F1-047 | Mayan EDMS | 13.0 Develop and Manage Business Capabilities | 6 | 124 | 83 | 36 | 4 | 1 |
| F2-001 | Asana | 1.0 Develop Vision and Strategy | 7 | 73 | 63 | 6 | 2 | 2 |
| F2-007 | Userback | 2.0 Develop and Manage Products and Services | 14 | 101 | 72 | 18 | 5 | 6 |
| F2-015 | Pipedrive | 3.0 Market and Sell Products and Services | 5 | 79 | 52 | 4 | 19 | 4 |
| F2-019 | Xero | 4.0 Manage Supply Chain for Physical Products | 7 | 100 | 75 | 5 | 17 | 3 |
| F2-025 | Salesforce Field Service | 5.0 Deliver Services | 8 | 75 | 57 | 4 | 11 | 3 |
| F2-031 | Salesforce Service Cloud | 6.0 Manage Customer Service | 10 | 67 | 57 | 3 | 6 | 1 |
| F2-037 | RUN Powered by ADP | 7.0 Develop and Manage Human Capital | 2 | 7 | 6 | 1 | 0 | 0 |
| F2-043 | ServiceNow IT Service Management | 8.0 Manage Information Technology (IT) | 24 | 153 | 117 | 17 | 18 | 1 |
| F2-049 | Sage Intacct | 9.0 Manage Financial Resources | 12 | 108 | 91 | 13 | 4 | 0 |
| F2-055 | MaintainX | 10.0 Acquire, Construct, and Manage Assets | 10 | 135 | 108 | 16 | 4 | 7 |
| F2-061 | Vanta | 11.0 Manage Enterprise Risk, Compliance, Remediation, and Resiliency | 8 | 101 | 78 | 19 | 3 | 1 |
| F2-067 | Givebutter | 12.0 Manage External Relationships | 12 | 211 | 174 | 10 | 24 | 3 |
| F2-073 | Success.co | 13.0 Develop and Manage Business Capabilities | 9 | 64 | 59 | 4 | 1 | 0 |
Raw access
Every table: browse the database, run your own SQL, or append .csv / .json to any table or query URL. Built by tools/business-apps/site-db.ts from data/business-apps/run/.