Exercise referral schemes aim to increase physical activity amongst inactive individuals with or at risk of long-term health conditions. Yet many patients referred to these schemes (by health professionals) fail to take up the exercise opportunities on offer. Understanding factors influencing uptake to exercise referral schemes may help improve future attendance. Using the Socio-Ecological Model as a framework, this qualitative study aimed to explore factors influencing uptake to an exercise referral scheme based in the North West of England. Semi-structured interviews were conducted with referred patients (n = 38) about their reasons for referral, interactions with referring health professionals, events following referral and ideas to improve future uptake. Data were analysed thematically and mapped onto the constructs of the Socio-Ecological Model. Factors reported to influence uptake included intrapersonal (past PA experiences, motivation, competing priorities), interpersonal (scheme explanations, support) and organizational influences (scheme promotion, communication between service, cost). Whilst several intrapersonal-level factors influenced patient decisions to uptake the exercise referral scheme, modifiable interpersonal and organizational factors were identified as potential targets for intervention. Recommendations are made for improving awareness of exercise referral schemes and for enhancing communication between referring practitioners, patients and referral scheme staff.