How do I stop an LLM from making things up in my app?

You can't eliminate hallucinations entirely, but you can reduce them a lot and make them detectable:

1. Ground the model: give it the relevant facts (RAG over your docs and policies) and instruct it to answer only from the provided context. 2. Give it permission to not know: "If the answer isn't in the documents, say you don't know and offer to contact support." Models guess much less when abstaining is explicitly allowed. 3. Require citations: ask for the source passage or doc id for each claim. Verify programmatically that quoted passages exist, and drop claims that can't be backed. 4. Use structured output for anything actionable (refund eligibility, prices). Compute those with code or database lookups (tool use), not model memory. 5. Evaluate continuously: keep a test set of real questions, including ones with no answer in the docs. Measure groundedness on every prompt or model change. 6. Design the UX for error: show sources, and add an easy "this was wrong" button that feeds your eval set.