Concierge AI Incorrectly Flags Hotel Receipt as Having Prohibited Incidentals

[m-natarajan]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Version Number:
Reproducible in staging?: Needs Reproduction
Reproducible in production?: Needs Reproduction
If this was caught during regression testing, add the test name, ID and link from BrowserStack:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @joekaufmanexpensify
Slack conversation (hyperlinked to channel name): #Concierge

Action Performed:

1. Upload a hotel receipt or Pay for a hotel expense by Expensify card
2. Concierge AI reviews the receipt
3. Receipt is flagged for prohibited incidentals

Expected Result:

Concierge AI should only flag receipts when prohibited incidentals are actually present.
Valid hotel receipts without restricted charges should pass without warnings.

Actual Result:

Receipt is incorrectly flagged as containing prohibited incidentals.

Workaround:

Unknown

Platforms:

Select the officially supported platforms where the issue was reproduced:

• Android: App
• Android: mWeb Chrome
• iOS: App
• iOS: mWeb Safari
• iOS: mWeb Chrome
• Windows: Chrome
• MacOS: Chrome / Safari

Screenshots/Videos

The screenshot for the issue is in slack thread

View all open jobs on GitHub

[melvin-bot]

Triggered auto assignment to @jliexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[dangrous]

hrm well I uploaded the provided PDF and it did not mark it locally, or on production, as having prohibited expenses. I'm going to make this a monthly and keep picking away at it. It's definitely just flakiness from the LLM, but we might be able to improve that.

[melvin-bot]

@dangrous @jliexpensify this issue was created 2 weeks ago. Are we close to a solution? Let's make sure we're treating this as a top priority. Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[jliexpensify]

Btw @dangrous I'm assessing a candidate's THC and they reported this same issue 😄

[dangrous]

stupid non deterministic AI...

Any chance the candidate can send over the receipt? It would be helpful! (but if they don't want to for privacy, etc., that's totally fine)

EDIT: If they do send it over, remember not to post it here, since this is our public repo! I forgot where I was for a sec

[dangrous]

I'm wondering if we should just add text somewhere being like "prohibited expenses are not always accurate! Please review receipts closely" or something, basically covering issues like these?

[dangrous]

@MelvinBot any thoughts on this one?

[MelvinBot]

Here are my thoughts after digging through the codebase:

How Prohibited Expense Detection Works

The prohibitedExpense violation is entirely backend-generated. The App frontend has zero detection intelligence -- it only displays what the backend sends. The flow is:

1. Workspace admins enable prohibited categories (alcohol, hotel incidentals, gambling, tobacco, adult entertainment) via toggles in Workspace > Rules
2. When a receipt is scanned, the backend AI/LLM analyzes receipt line items against the enabled categories
3. Backend creates TransactionViolation records with name: 'prohibitedExpense' and sends them to the client
4. The App merges and displays them as "Prohibited expense: hotel incidentals"

The key evidence that this is backend-only: ViolationsUtils.getViolationsOnyxData handles client-side violation creation for many types (missing category, tag out of policy, over limit, etc.) but never creates prohibitedExpense violations.

Why False Positives Happen

This is classic LLM non-determinism. The backend AI is analyzing receipt images and trying to identify specific line items as "hotel incidentals." Hotel receipts are tricky because the AI needs to distinguish between:

• Room charges (valid) vs. minibar, spa, laundry, etc. (incidentals)
• Room service food vs. restaurant charges
• Resort fees vs. room rates

The same receipt can produce different results on different runs depending on how the LLM interprets ambiguous line items.

On the Disclaimer Idea

Adding a user-facing disclaimer is a reasonable mitigation. Currently there is no disclaimer, confidence indicator, or "review recommended" language anywhere in the violation display flow. The user just sees a flat assertion: "Prohibited expense: hotel incidentals" -- which reads as definitive when it's actually probabilistic.

The admin-facing description already says "will require manual review" (en.ts:6351-6352), but this only appears in workspace settings, not to end users.

Suggested approaches (not mutually exclusive):

1. Soften the violation message itself -- Change from "Prohibited expense: hotel incidentals" to something like "Possible prohibited expense: hotel incidentals" in en.ts:7753-7781. This is the simplest change -- just update the translation string in all languages. The receiptGeneratedWithAI violation already does this by saying "Potential AI-generated receipt" rather than asserting it definitively.

2. Add a disclaimer component near the receipt audit -- Similar to the existing GPS Disclaimer pattern (small icon + muted text), add a brief note near the receipt violations in MoneyRequestReceiptView when AI-detected violations are present. Something like "AI-detected items may not always be accurate. Please review the receipt."

3. Backend improvements -- The actual fix for reducing false positives would be in the backend receipt analysis pipeline (likely in Auth or Web-Expensify). Options include: adjusting the LLM prompt to be more conservative, adding confidence thresholds so borderline detections don't trigger violations, or requiring multiple LLM runs to agree before flagging.

My recommendation: Option 1 is quick and addresses the immediate UX problem -- the violation message shouldn't sound so definitive for an AI judgment call. Option 3 is the proper long-term fix but requires backend investigation. Option 2 sits in between and may be overkill if the message itself is softened.