This paper unpacks our efforts as external evaluators to work toward decolonizing our evaluation practice. Undertaking this writing exercise as a form of reflective practice demonstrated that decolonization is much more complex than simply translating materials, organizing locals to collect data, and building participants' capacity around Western modalities. While this complexity is clear in the decolonization literature, practice-based examples that depict barriers and thought processes are rarely presented. Through this paper, we deconstruct our deeply held beliefs around what constitutes good evaluation to assess the effectiveness of our decolonizing approach. Through sharing our critical consciousness-raising dialoguing, this paper reports our progress thus far and provides information and provocations to support others attempting to decolonize their practice.