This study examines the roles and challenges of external quality assurance in reviewing learning outcome assessments of Japanese universities. Following criticism that Japan has overly relied on the difficulty of entrance examination as a quality metric for a given institution, rather than graduates' competencies, there is more pressure for transparency in higher education. Recognising the shift from the old dichotomy to a new 'trinity' (accountability, improvement and transparency), Japanese accreditation has pushed universities to define, assess and improve learning outcomes. Applying the double logic of quality assurance (bureaucracy and peer review) and employing interviews with key quality assurance agency personnel, this study shows peer reviewers' professional judgment as qualitative, contextual and even philosophical, thus helping enhance a university's uniqueness and diversity while maintaining objectivity and equity. Furthermore, accreditation needs to cultivate a 'culture of evidence' in universities to capture their initiatives and make such efforts empirically meaningful.