Sticker research · Supply Bench
Frontier models on multi-vendor procurement workflows
We evaluate computer-use agents from approved request through supplier ordering, line allocation, exception handling, and reconciliation. Every task is deterministic and scored from final system state against programmatic rubrics.
← Back to sample coverProcurement judgment
The data behind this pilot
Visible product specifications, pack quantities, inventory, delivery dates, contract pricing, shipping thresholds, and clinic policies drive every accepted outcome. Tasks permit plausible wrong purchases to submit so the verifier captures concrete procurement mistakes.
| Product | Brand | Supplier SKU | Supplier | Pack | Price |
|---|---|---|---|---|---|
| McKesson Monofilament Sensory Test 10 Gram | McKesson | 1074987 | health supply | 1 test / EA | $12.68 |
| Gauze Sponge McKesson 4 X 4 Inch 12-Ply NonSterile 200 per Pack | McKesson | 440343 | health supply | 200 sponge / PK | $5.81 |
| Gauze Sponge McKesson 3 X 3 Inch 12-Ply NonSterile 200 per Pack | McKesson | 373776 | health supply | 200 sponge / PK | $3.80 |
| Bandage Scissors McKesson Argent Lister 7-1/4 Inch Surgical Grade Stainless Steel | McKesson | 487325 | health supply | 1 scissor / EA | $23.23 |
| Bandage Scissors McKesson Lister 5-1/2 Inch Office Grade Stainless Steel | McKesson | 487480 | health supply | 1 scissor / EA | $5.35 |
| Ammex Professional Series Powder Free Nitrile Exam Gloves, Latex Free, Blue, 100/Box | Ammex | 24474540 | health supply | 100 glove / BX | $53.55 |
| McKesson Confiderm 3.5C Nitrile Exam Gloves Medium Powder-Free NonSterile 100/Box | McKesson | AUTH-NTRL-EQ | health supply | 100 glove / BX | $16.99 |
| Vinyl Exam Gloves Medium Powder-Free NonSterile 100/Box | Generic | AUTH-VINYL-DECOY | health supply | 100 glove / BX | $6.99 |
| Nitrile Exam Gloves Medium Powder-Free NonSterile 50/Box | Generic | AUTH-NTRL-SHORT | health supply | 50 glove / BX | $9.99 |
| Laceration Tray McKesson Sterile | McKesson | 1013334 | health supply | 1 tray / CS | $4.40 |
| Fellowes Thermal Laminating Pouches | Fellowes | 2728675 | business supply | 100 pouch / PK | $41.99 |
| Fellowes Thermal Laminating Pouches | Fellowes | 24354534 | business supply | 200 pouch / PK | $47.99 |
| OdoBan Concentrated Odor Eliminator | OdoBan | ODO911062G4EA | business supply | 1 gallon / EA | $13.99 |
| OdoBan Concentrated Odor Eliminator case | OdoBan | ODO911062G4 | business supply | 4 gallon / CS | $51.99 |
| Fine Tip Assorted Markers 24/Pack | School Smart | 752356 | business supply | 24 marker / PK | $11.99 |
| Crayola Pip-Squeaks Water Based Markers | Crayola | 683860 | business supply | 16 marker / PK | $8.49 |
| Crayola Pip-Squeaks Washable Markers | Crayola | 649977 | business supply | 16 marker / PK | $8.99 |
20 or fewer representative offer snapshots shown. Evaluation does not require Supabase or supplier network access.
- 17% pass@1 across 18 graded attempts.
- Claude Fable 5 passed 1/9 graded attempts with 65% mean reward.
- GPT-5.5 passed 2/9 graded attempts with 61% mean reward.
- 5/18 attempts ended without a terminal disposition; 12 completed purchases across both suppliers, and 4 submitted a reconciled outcome.
Methodology: cohort v3-final-100step; target 3 attempts per task-model. Strict pass = a submitted, fully defensible procurement outcome. Mean reward = the average weighted verifier score from 0 to 1. Near-perfect identifies reward ≥ 0.95 and does not count as a strict pass. V2 and V3 results are never combined.
Task suite
Open a task for attempts, grading evidence, supplier mappings, and full trajectories.
| Task | Scope | Ceiling | Claude Fable 5 | GPT-5.5 |
|---|---|---|---|---|
| Task 1 · V3-1Approved PO - Northside Clinical Supplies | 3 lines | $100.00 | strict pass 0/3mean reward 0.40 0.400.400.40 Best trajectory → | strict pass 2/3mean reward 0.76 1.000.271.00 Best trajectory → |
| Task 2 · V3-2Approved PO - Downtown Clinic Restock | 5 lines | $220.00 | strict pass 0/3mean reward 0.95 0.950.950.95 Best trajectory → | strict pass 0/3mean reward 0.62 0.270.950.65 Best trajectory → |
| Task 3 · V3-3Approved PO - West End Clinic Operations | 4 lines | $260.00 | strict pass 1/3mean reward 0.60 1.000.400.40 Best trajectory → | strict pass 0/3mean reward 0.45 0.550.240.55 Best trajectory → |
Observed failure modes
Patterns across the selected v3 cohort.
- Incomplete execution. 5 attempts stopped before committing a terminal procurement disposition.
- Cross-system execution. 12 attempts successfully created orders with both supplier portals.
- Post-order recovery. 4 attempts completed discrepancy resolution and submitted reconciliation.