Changes in the profession of medicine are creating the demand for a substantive reexamination of current practices in medical education. Many of the major issues in addressing this challenge are structural and political rather than scientific; here we address one critical scientific issue that will be important to (though not by itself adequate for) guiding such changes. The factor we address is the development of a global performance assessment: a standardized mechanism to assess individual skills and abilities that can be used to evaluate alternative educational interventions. We discuss the issues involved in developing such a system, and describe a set of principles for defining desired outcomes and developing assessment tools, including (a) wide clinical scope, (b) direct relationship to actual clinical performance, (c) reliability and repeatability, and (d) non-longitudinal measurement. We discuss the role of simulation, reflective practice, and portfolios of work in developing such a system, and argue that whatever form such metrics take, a system of global performance assessment will develop from close collaboration between clinicians and educators with innovative ways of thinking about performance and expertise.