This paper specifically reports on a comprehensive evaluation of competency based assessment within the mining industry in Queensland, Australia. However, it also addresses a broader problem associated with competency-based assessment already identified in the literature. The qualitative component of the study involved 83 people, nineteen mine sites and seven sites in Brisbane. Interviews and focus groups were used to obtain views about the strengths and weaknesses of assessment practices. A quantitative survey involved a different cohort of 83 people and sought information about the assessment competencies themselves. A number of problems were identified in the training of assessors and the implementation of assessment in the workplace. Given the complex environment in which assessment occurs these findings, like the extant literature, suggest that careful scrutiny and reconceptualisation of assessment practice may be warranted.