A Databricks Function called ai_parse_document turns messy PDFs into structured JSON with just a few lines of SQL, handling both digitally-born and scanned files by combining embedded text reads and OCR. You feed read_files() binary blobs into ai_parse_document, flatten the resulting document.elements into a full_text string, then run ai_query with a prompt that asks for specific fields—patient_ref, document_date, document_type, primary_diagnosis, follow_up_required. In a proof-of-concept, it nails the obvious fields and even guesses unstated ones, but wraps everything in markdown and prose, so you still need a cleanup step before parsing the JSON. Running that same pipeline in production surfaces four hidden costs. First, every re-run—whether fix the prompt or reprocess new batches—incurs model charges (~$4.20–$4.55 per 1,000 pages). A 30,000-page corpus costs about $120–$137 each time you iterate. Second, duplicate documents slip through: corrected or resent notes get stored again, skewing your patient cohort and diagnosis counts. Third, non-determinism creeps in. Even at temperature zero, LLM rounding and parallel GPU execution can flip a field like follow_up_required back and forth, undermining audit trails. And fourth, there’s no built-in checkpoint, so every small change forces you to reparse everything. The post walks through a simple two-step SQL example in a Databricks notebook and points to a companion GitHub repo for the full code. It stops short of deep dives into cost-tuning (Photon vs. serverless), failure modes, exactly-once semantics, or data governance, noting those deserve their own write-ups. If you need a system that runs reliably at scale and meets audit requirements, you’ll have to layer on deduplication, checkpointing, reconciliation steps, and possibly a deterministic fallback rather than lean solely on ai_query.