Data teams have long focused on execution—writing SQL, training models, shipping dashboards—with the assumption that good judgment would follow. That never worked. As AI takes over more of the execution layer, the real gap is between people who can build pipelines and those who can decide if those pipelines measure the right thing. The author calls this latter role “measurement engineer” to highlight its distinct skill set: choosing and validating metrics, spotting confounded tests, and guiding decisions when data is messy. Three common failures show why. First, teams often track hundreds of metrics and cherry-pick whichever ones support their story. A measurement engineer would work with product leaders to retire noise and focus on the handful of metrics that predict real outcomes. Second, internal model evaluations can tick upward while user satisfaction tanks. The evals were precise but measured fluency, not usefulness. Bridging that gap means validating that your tests actually predict user experience. Third, A/B tests with borderline results require more than p-values. You need power analysis to know if a 1.5% retention drop is real risk and the confidence to recommend holding off when evidence is inconclusive. Judgment isn’t just gut feel. It comes from disciplines largely missing in data science courses: construct validity (from psychometrics), measurement reliability, and decision theory under ambiguity. Asking if your metric truly captures engagement or if your eval suite consistently measures what matters can prevent costly mistakes. And when tests conflict or effect sizes lie near the detection threshold, decision theory teaches you how to weigh risks instead of defaulting to whichever direction seems appealing. This matters more now because AI handles the pipes and charts, but struggles with meaning. Evaluating AI systems is the hardest measurement problem most teams have faced: outputs are non-deterministic and subjective. At the same time, a wrong metric on a dashboard can skew a quarter’s strategy, and a faulty AI eval can send a hallucinating model live. As those costs scale, measurement engineering becomes the most valuable skill.