The classify CLI's lexical engine had zero lexicon coverage for "feature" and "praise" (only "bug" was covered) and none at all for generic sentiment (positive/negative/neutral), so bare-label runs of the CLI's own README example either tied on everything or confidently mislabeled a praise/feature row as a bug whenever a bug-shaped word like "crash" appeared. Add two small, additive synonym groups to the existing lexicon. Measured on a new 65-row synthetic gold set (examples/classify-eval.jsonl, run via `npm run classify-eval`, scripts/classify-eval.mjs): 32.3% -> 49.2% accuracy counting ties as wrong, 67.7% -> 71.1% on rows the engine actually committed to; the sentiment set alone went 5.0% -> 35.0%. The product-wide gold regression suite (`npm run gold`) still passes 61/61 — every change is a new synonym entry, not an edit to an existing one. README now documents the measured numbers, the --threshold tradeoff (0.55 raises decided-row accuracy to 83.8% at the cost of more abstains), and the real remaining limit: a bag-of-words engine can't fully resolve a praise sentence that mentions "crash" from an actual bug report. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
66 lines
11 KiB
JSON
66 lines
11 KiB
JSON
{"set": "ticket", "labels": "bug,feature,praise", "text": "The app crashes every time I open the camera screen on Android.", "label": "bug", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Please add a dark mode, the white background hurts at night.", "label": "feature", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Love the new dashboard, it saves me an hour every week.", "label": "praise", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Export to CSV fails with an error when the report has more than 1000 rows.", "label": "bug", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "It would be great if I could invite teammates with a link.", "label": "feature", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Support answered in five minutes and solved my problem. Thank you!", "label": "praise", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Login keeps failing after the latest update.", "label": "bug", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Can you add an option to schedule reports by email?", "label": "feature", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Best onboarding I have seen in a long time, well done.", "label": "praise", "shared_vocab": false}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Thanks for finally fixing the crash, the update feels so much smoother now.", "label": "praise", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "The new update broke dark mode, now everything looks washed out again.", "label": "bug", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Could you add a crash report email so we know when the app fails?", "label": "feature", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Dark mode crashes the settings screen every time I open it.", "label": "bug", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "I love that you finally added the export option, exporting reports is so much faster now.", "label": "praise", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Exporting to CSV still fails after the last update, please fix it.", "label": "bug", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Would it be possible to add dark mode scheduling, like automatically switching at night?", "label": "feature", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "The onboarding flow crashes on step two for new signups.", "label": "bug", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Your onboarding is incredible, we signed up five new teammates this week thanks to it.", "label": "praise", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Please let us invite teammates directly from the dashboard instead of email.", "label": "feature", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "The dashboard used to crash constantly but the latest fix made it rock solid, amazing work.", "label": "praise", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Scheduling reports by email keeps failing silently, no error shown.", "label": "bug", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "It would help a lot if failed report emails showed a clear error message.", "label": "feature", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Five minutes and the crash was fixed, support is incredible, thank you.", "label": "praise", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "The invite link feature is broken, teammates get an error when they click it.", "label": "bug", "shared_vocab": true}
|
|
{"set": "ticket", "labels": "bug,feature,praise", "text": "Can you add a retry button when the export fails instead of just an error?", "label": "feature", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "This is the best purchase I've made this year.", "label": "positive", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Absolutely terrible, I want my money back.", "label": "negative", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "The package arrived on Tuesday as scheduled.", "label": "neutral", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Good packaging, but the product itself stopped working after two days.", "label": "negative", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Not bad, does what it says, nothing special.", "label": "neutral", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "I was skeptical at first but now I genuinely love it.", "label": "positive", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "The color is fine but the zipper broke on the first use.", "label": "negative", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Delivery took three days, which matches the estimate given at checkout.", "label": "neutral", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Customer service was slow, but once they responded, they fixed everything perfectly.", "label": "positive", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "It works, I guess. Average, nothing to complain about.", "label": "neutral", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Great idea, poor execution, it looks good but constantly glitches.", "label": "negative", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Honestly one of the best support experiences I've had, five stars.", "label": "positive", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "The manual says it's waterproof, and the box included a charging cable.", "label": "neutral", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Good in theory, but in practice it's just frustrating to use every day.", "label": "negative", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Exceeded my expectations, works even better than advertised.", "label": "positive", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "It's okay. Not great, not terrible, does the basic job.", "label": "neutral", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "I had a good first impression, but after a week it stopped syncing entirely.", "label": "negative", "shared_vocab": true}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "The app has a dark mode toggle and a settings menu in the top right.", "label": "neutral", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "So happy I switched, this is miles better than my old one.", "label": "positive", "shared_vocab": false}
|
|
{"set": "sentiment", "labels": "positive,negative,neutral", "text": "Fine for the price, roughly what I expected based on the listing.", "label": "neutral", "shared_vocab": false}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Click here now to claim your free prize before it expires!!!", "label": "spam", "shared_vocab": false}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "We're running a 20% discount on annual plans this month, want details?", "label": "sales", "shared_vocab": false}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "My account got locked after I changed my password, can you help?", "label": "support", "shared_vocab": false}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Congratulations, you've won a gift card, click the link to claim it.", "label": "spam", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Hi, following up on our demo call, happy to send over a custom quote.", "label": "sales", "shared_vocab": false}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "I keep getting a 500 error when I try to log in, please help.", "label": "support", "shared_vocab": false}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "URGENT: your account will be suspended, click here to verify now.", "label": "spam", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Our enterprise plan includes a dedicated account manager, interested?", "label": "sales", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Can you reset my password, the reset link in the email isn't working?", "label": "support", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Limited time offer: buy one get one free, click to redeem.", "label": "spam", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Thanks for the trial signup, want to hop on a call to discuss pricing?", "label": "sales", "shared_vocab": false}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "The export button is greyed out and I can't click it, what's wrong?", "label": "support", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "You have unclaimed rewards, click this link to collect your winnings.", "label": "spam", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "I'd like a discount on the enterprise tier if we commit to a yearly contract.", "label": "sales", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "My invoice shows a double charge, can someone look into my account?", "label": "support", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Act now, this exclusive offer disappears in 24 hours, click to claim.", "label": "spam", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Our team would love to offer you a personalized walkthrough of the new plan.", "label": "sales", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "The help center link on the account page returns a 404 error.", "label": "support", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "Free trial expired but you can click here to reactivate instantly and save 50%.", "label": "spam", "shared_vocab": true}
|
|
{"set": "moderation", "labels": "spam,sales,support", "text": "We noticed you're on the free plan, want to talk about upgrading with a discount?", "label": "sales", "shared_vocab": true}
|