The concept of neurodiversity has fuelled a social justice movement advocating for the rights of those whose lives diverge from a socially-constructed default. However, deficit understandings of disability persist in educational settings and neurodivergent people continue to face disadvantage and discrimination in organisations constructed on normative understandings of the world. Although New Literacy Studies is concerned with ideas of power, dominance and worth, there is a notable lack of work that connects NLS with issues of neurodiversity. In this paper, I introduce the term 'neurodivergent literacies' to propose a field of study that links the ideological model of literacy with the neurodiversity paradigm. From this starting point, I outline a project that examined literacies around what are often referred to as the 'special interests' of autistic people. Presenting data from interviews with 13 neurodivergent adults, related to school experiences and the literacies they engage with around their self-defined 'ruling passions', I make recommendations for literacies practitioners, arguing that schools need to do more to take account of difference and disability. By describing how 'neurodivergent literacies' can help teachers harness their own critical literacy skills to challenge deficit models of difference in the classroom, this paper illuminates how an understanding of neurodiversity is essential for anyone teaching and researching literacies with a commitment to social justice.