Reaction mechanisms are a core component of organic chemistry. Being able to handle these mechanisms is a central skill for students in this discipline. Diagnosing and fostering mechanistic reasoning is hence an important branch of chemistry education research. When it comes to reasoning about mechanisms, students often experience difficulties because they either lack conceptual understanding, cannot make appropriate inferences, or struggle to link representations to chemical concepts. Instructional tools to bridge this gap are thus required in organic chemistry education. Recently, scaffolds to support students to make connections between properties and reaction pathways as well as connecting a representation to chemical concepts are documented to be helpful to foster students' reasoning. Although the advantage of scaffolds is reported, the question arises of how students are working with scaffolds and if scaffolds can influence students' scores in a conceptual knowledge test. In this study, we explored in a pre--post mixed methods approach how students recruited from an organic chemistry course are working with a written scaffold. We correlated their level of causal complexity and multivariateness expressed in the scaffolds with their score in a conceptual knowledge test pre and post. The task used consisted of scaffolded contrasting cases of two addition reaction steps of a nucleophile on a carbonyl carbon. The paper-pencil test used for pre-, and posttest covered the respective conceptual knowledge. We qualitatively identified patterns along the dimensions of causal complexity and multivariateness in students' written responses in the scaffold and looked for relationships between students' scores in the pre- and posttest and these two dimensions. We found five different patterns in students' responses and were able to show that the score they achieved in the pretest influenced how effectively students were supported by the scaffold. Thus, this exploratory study provides encouraging implications and insights into the use of scaffolds.