Evaluation

How well does it work?

Every number here comes from the project's own eval suite (npm run evals). Cases and pass criteria were written before the first run. These are synthetic tests, not user research: they show the system does what it claims, not that people love it.

Last run Tue, 27 Oct 2026 08:07:27 GMT · engine rec-1.3

Offer matching precision

100.0%

"Exact" matches that really were the same bottle, over 61 adversarial listings. Recall 100.0%.

Recommendation personas passing

29/32

Relevance checks 94.6%; hard exclusions 100.0%.

Fake ids or prices reaching users

0 / 7

Planted in adversarial model outputs; the grounding check must remove all of them.

Common bottles found by name

100.0%

38,365 fragrances; median search 12 ms (local).

What the evals changed

The first run found real problems. Each fix was a product decision, logged in the decision register, and the cases themselves were not edited to pass.

Run	Engine	Offer precision	Personas passing	Relevance checks
v1-baseline	rec-1.0	83.3%	27/30	94.2%
v2-allowlist-flankers	rec-1.1	100.0%	28/30	96.2%
v3-budget-alternatives	rec-1.2	100.0%	29/32	94.6%
v4-collection-intelligence	rec-1.2	100.0%	29/32	94.6%
v5-provider-ready	rec-1.2	100.0%	29/32	94.6%
v6-price-bands-accords	rec-1.3	100.0%	29/32	94.6%
v6-live-offers	rec-1.3	100.0%	29/32	94.6%
v7-ux	rec-1.3	100.0%	29/32	94.6%

Blocklists don't catch flankers
v1 matched “Khamrah Qahwa” and “1 Million Lucky” as exact. v2 only trusts words it recognises; anything else goes to “Needs verification”.
Similar meant “buy a flanker”
A La Vie Est Belle fan got two La Vie Est Belle flankers. v2 flags other versions of owned lines and down-weights them, the PRD's “avoid a redundant purchase”.
Different stayed in the same family
An all-amber shelf got another amber. v2 rewards a new family and gives Gap mode its own weights, so filling the gap outweighs matching current taste.

Recommendations: 32 synthetic personas

Every case runs hard checks (nothing owned, no disliked notes, budget and marketing respected) plus persona-specific relevance checks: closeness to the loved bottle for Similar, low overlap and a new family for Different, a named gap for Fill gaps, and an honest “no clear gap” for balanced or tiny shelves. Median 16.2 ms over 3,000 candidates.

Case	Persona	Checks	Top 3
S01	Fresh-blue fan who loves Bleu de Chanel EDP	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.7 (0.90) ✓ an annotated good answer in top 5	Armaf Club de Nuit Blue Iconic · French Avenue Zenith Deep · Armaf Club De Nuit Urban Elixir
S02	Loves Tobacco Vanille, likes Jazz Club	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.7 (0.93) ✗ an annotated good answer in top 5	Maison Alhambra Tobacco Touch · Franck Boclet Tobacco · Argos Triumph Of Bacchus
S03	Loves La Vie Est Belle, prefers women's and unisex	✓ returns results ✓ nothing owned ✓ marketing preference ✓ top-3 similarity to anchor ≥ 0.7 (0.93) ✗ an annotated good answer in top 5	La Rive Queen of Life · Chloé Nomade Naturelle Eau de Parfum · PARIS CORNER Khair Confection
S04	Loves Aventus	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.65 (0.88) ✓ an annotated good answer in top 5	Armaf Club de Nuit Intense Man Parfum · Afnan Supremacy Silver · Tiziana Terenzi Orion
S05	Loves Light Blue and Acqua di Gioia	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.65 (0.84)	Xerjoff XJ 1861 Renaissance · Roja Dove Isola Blu · Profumum Roma Acqua Viva
S06	Niche woody fan: Santal 33 and Gypsy Water	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.6 (0.74) ✓ top pick family in woody/aromatic/amber (woody)	Chanel Bleu de Chanel Parfum · Zara Energetically New York Elixir · Creed Bois du Portugal
S07	Loves Encre Noire (dark vetiver)	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.65 (0.93) ✓ top pick family in woody/aromatic/green (woody) ✓ reports LIMITED_PROFILE	Chanel Sycomore Eau de Parfum · Marc-Antoine Barrois Encelade · Maison Alhambra Anchor Black
S08	Five designer bottles, none rated yet	✓ returns results ✓ nothing owned ✓ reports LIMITED_PROFILE	Armaf Club De Nuit Urban Elixir · Afnan Historic Olmeda · French Avenue Zenith Blue
S09	Leather lover who avoids oud	✓ returns results ✓ nothing owned ✓ no disliked notes ✓ top-3 similarity to anchor ≥ 0.55 (0.59)	Maison Alhambra Amber & Leather · Tom Ford Tuscan Leather · French Avenue Scorpio
S10	Loves Baccarat Rouge 540 but has a $100 budget	✓ returns results ✓ nothing owned ✓ within budget ✓ top-3 similarity to anchor ≥ 0.6 (0.90)	Armaf Club de Nuit Untold · Natura Essencial Único · Ajmal Amber Wood
S11	Added in v3 for Q-01: loves Tobacco Vanille, wants originals only (no budget alternatives)	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.65 (0.89) ✗ an annotated good answer in top 5 ✓ no budget alternatives when switched off	Franck Boclet Tobacco · Argos Triumph Of Bacchus · Viktor&Rolf Spicebomb Extreme
S12	Added in v3 for Q-01: loves Aventus, budget alternatives on (default)	✓ returns results ✓ nothing owned ✓ top-3 similarity to anchor ≥ 0.65 (0.88) ✓ a top-5 pick has BUDGET_ALTERNATIVE	Afnan Supremacy Silver · Armaf Club de Nuit Intense Man Parfum · Tiziana Terenzi Orion
D01	All-fresh blue collection wants something new	✓ returns results ✓ nothing owned ✓ every pick < 0.75 overlap with owned (0.64) ✓ top pick family not aquatic (amber)	Montale Oud Tobacco · Roja Dove Enigma Pour Homme · Xerjoff Pikovaya Dama 2018
D02	Sweet-gourmand collection wants a change	✓ returns results ✓ nothing owned ✓ marketing preference ✓ every pick < 0.75 overlap with owned (0.60) ✓ top pick family not gourmand (floral)	Amouage Guidance · Mancera Red Tobacco Intense · BDK Parfums Gris Charnel Extrait
D03	UX doc Flow C: woody winter scent under $150, unlike what they own	✓ returns results ✓ nothing owned ✓ within budget ✓ every pick < 0.75 overlap with owned (0.62) ✓ top pick family in woody/amber/leather (woody) ✓ every pick suits winter	Aramis Aramis · Montblanc Explorer Extreme · Lalique Encre Noire A L'Extreme
D04	Men's-marketed only, avoids oud	✓ returns results ✓ nothing owned ✓ no disliked notes ✓ marketing preference ✓ every pick < 0.75 overlap with owned (0.57)	Gisada Ambassador Intense · Orto Parisi Terroni · Roja Dove Enigma Pour Homme
D05	Wants something new that features iris	✓ returns results ✓ nothing owned ✓ every pick < 0.75 overlap with owned (0.49) ✓ a top-3 pick has MATCHES_LIKED_NOTES	Chanel Chance Eau de Toilette · Ex Nihilo Blue Talisman Extrait de Parfum · Dior Dior Homme Intense 2007
D06	Owns just one bottle	✓ returns results ✓ nothing owned ✓ every pick < 0.75 overlap with owned (0.61) ✓ reports LIMITED_PROFILE	Roja Dove Enigma Pour Homme · Amouage Interlude Man · Amouage Interlude Black Iris
D07	Heavy woody-amber shelf wants a change	✓ returns results ✓ nothing owned ✓ every pick < 0.75 overlap with owned (0.67) ✓ top pick family in citrus/aquatic/aromatic/green/floral/chypre (floral)	Dior Dior Homme Intense 2007 · Nishane Nefs · Dior Sauvage Elixir
D08	Something new for daytime under $60	✓ returns results ✓ nothing owned ✓ within budget ✓ every pick < 0.75 overlap with owned (0.58) ✓ every pick works in the daytime	Lattafa Perfumes Khamrah Waha · Rasasi Hawas Tropical · Lattafa Perfumes Angham
D09	Floral collection, women's and unisex only	✓ returns results ✓ nothing owned ✓ marketing preference ✓ every pick < 0.75 overlap with owned (0.63) ✓ top pick family not floral (amber)	Frederic Malle Editions de Parfums The Moon · Al Haramain Perfumes Amber Oud Gold Edition · Dior Dior Addict Eau de Parfum
D10	Wants a long-lasting change	✓ returns results ✓ nothing owned ✓ every pick < 0.75 overlap with owned (0.52) ✓ top-3 longevity ≥ 3.5	Estée Lauder Beautiful · Creed Royal Oud · Prada Infusion d'Homme
G01	Summer-only fresh collection	✓ returns results ✓ nothing owned ✓ every top-3 pick names a gap it fills ✓ a top-3 pick has FILLS_SEASON_GAP ✓ top pick's strongest season in fall/winter (winter)	Nishane Nefs · Amouage Interlude Black Iris · Orto Parisi Terroni
G02	Winter-only cozy collection	✓ returns results ✓ nothing owned ✓ every top-3 pick names a gap it fills ✓ a top-3 pick has FILLS_SEASON_GAP ✓ top pick's strongest season in spring/summer (summer)	Rasasi Hawas Tropical · Jean Paul Gaultier Le Beau Paradise Garden · Kenzo Kenzo Homme Marine
G03	Balanced shelf within the families they like: no real gap	✓ returns results ✓ nothing owned ✓ reports NO_CLEAR_GAP	Montale Black Aoud · Amouage Guidance 46 · Louis Vuitton Attrape-Rêves
G04	Two bottles: too few to call anything missing	✓ returns results ✓ nothing owned ✓ reports NO_CLEAR_GAP + LIMITED_PROFILE ✓ invents no gap	Maison Francis Kurkdjian Oud Satin Mood Extrait de Parfum · Maison Crivelli Oud Stallion · Dior Dior Homme Parfum
G05	Fresh shelf, avoids oud	✓ returns results ✓ nothing owned ✓ no disliked notes ✓ every top-3 pick names a gap it fills	Orto Parisi Terroni · Amouage Guidance 46 · Amouage Purpose 50
G06	Gap filling on a $50 budget	✓ returns results ✓ nothing owned ✓ within budget ✓ every top-3 pick names a gap it fills	Rasasi Shuhrah Pour Homme · Ted Lapidus Lapidus Pour Homme · Rasasi La Yuqawam Homme
G07	Evening-only shelf needs a daytime scent	✓ returns results ✓ nothing owned ✓ a top-3 pick has FILLS_DAYPART_GAP/FILLS_SEASON_GAP ✓ top pick is daytime-leaning	Rasasi Hawas Ice · French Avenue Vulcan Feu · Jean Paul Gaultier Le Beau Paradise Garden
G08	Feminine floral-gourmand shelf, women's and unisex only	✓ returns results ✓ nothing owned ✓ marketing preference ✓ every top-3 pick names a gap it fills	Maison Crivelli Oud Maracujá · Roja Dove Diaghilev · Nishane Hacivat
G09	Winter shelf asks explicitly for summer	✓ returns results ✓ nothing owned ✓ a top-3 pick has FILLS_SEASON_GAP ✓ every pick suits summer	Rasasi Hawas Tropical · Jean Paul Gaultier Le Beau Paradise Garden · Kenzo Kenzo Homme Marine
G10	Aquatic-only shelf	✓ returns results ✓ nothing owned ✓ every pick < 0.85 overlap with owned (0.43) ✓ top pick family not aquatic (amber)	Orto Parisi Terroni · Afnan 9 PM Night Out · Roja Dove Enigma Pour Homme

Open question from the failures: for two personas the engine prefers close alternatives from smaller houses (a Tobacco Vanille-style blend from Maison Alhambra) over the famous names I annotated. That may be exactly what a value-minded user wants, or it may hurt trust. It's a product call to test with users, not something to tune away.

Offer matching: 61 adversarial listings

Hand-written listing titles for 12 bottles, each annotated before running: the right bottle phrased different ways (ounces vs ml, brand abbreviations), and wrong ones that look right (EDT for EDP, 50 ml for 100, flankers, testers, gift sets, refills, decants, pre-owned). Precision matters most: a wrong “exact” match is worse than a missing one.

Listings where the matcher disagreed with the annotation
Listing	Annotated	Matcher	Why
Lattafa Khamrah Qahwa Eau de Parfum 100 ml	rejected	needs verification	extra words (possible flanker)
Paco Rabanne 1 Million Lucky Eau de Toilette 100 ml	rejected	needs verification	extra words (possible flanker)
Lancome La Vie Est Belle Intensement EDP 100 ml	rejected	needs verification	extra words (possible flanker)

These remaining disagreements fail safe: the listing is shown under “Needs verification” with the reason, never ranked as a match.

Grounding: can the advisor slip in something false?

The agent loop is driven with scripted model outputs that cite invented fragrance ids, real fragrances the tools never returned, and made-up prices. The output contract requires every cited fragrance to come from a tool result in that turn and every dollar amount to appear in a tool result.

invented fragrance id: 1 planted, 0 reached the user; 1 valid citation kept.
id of a real fragrance the tools never returned: 1 planted, 0 reached the user; 1 valid citation kept.
invented price: 1 planted, 0 reached the user; 1 valid citation kept.
invented price range: 2 planted, 0 reached the user; 1 valid citation kept.
uppercase citation of a valid id: 0 planted, 0 reached the user; 1 valid citation kept.

Live agent eval (real model, 8 prompts on the sample shelf, expectations written before the run): not run in this report: no GEMINI_API_KEY or ANTHROPIC_API_KEY set; live agent eval skipped.

Collection Intelligence: 9/9 checks

The fixture plan from the Collection Intelligence PRD, run on real catalog data: an empty shelf, one bottle, the mixed demo shelf, twenty near-identical fresh bottles, and a shelf with all season and occasion data stripped. Plus the Today selector across 15 occasion and weather combinations, and every Scent Lab pairing of the demo shelf.

CS-01 empty shelf has no score
CS-small one bottle is provisional and low confidence
CS-02 same shelf, any order, same score
CS-size 20 fresh/aquatic bottles score lower diversity than 6 mixed (24 vs 85)
CS-03 missing metadata excluded and reweighted, low confidence
CS-04 previewing an owned bottle is a duplicate
DS-01/04 owned-only, unique, avoid list honoured, 2+ factors across 15 contexts (15/15)
SL-02 A/B equals B/A and scores stay 0–100 for all 15 demo pairs (15/15)
SL-sense a classic complement (Tobacco Vanille + Jazz Club) beats a tension pair (Tobacco Vanille + Acqua di Gio) (90 vs 51)

Limits of these evals

Personas and annotations are mine, not real users'. They test behaviour against the spec, not taste.
Offer titles are synthetic. Real listings are messier; live precision needs a sample of real SerpApi results, labelled.
Notes, accords and season votes are community data and can be wrong; the evals inherit that.