Uses a reliability framework to examine the role that relevance judgments play in quantitative information retrieval (IR) performance measurement with an eye toward issues introduced by Internet IR. Argues that a theoretical basis for reliable measurement is possible if the relevance yardstick is reconsidered in light of recent research and if the dynamic nature of cyberspace is taken into account. (Author/AEF)