Many autistics report being distressed by eye contact, but eye-tracking studies suggest that eye contact is associated with hypo-arousal rather than hyper-arousal in autism. Within a live face-to-face paradigm combining a wearable eye-tracker with electrodermal activity sensors, 80 adults (40 autistics) defined words in front of an experimenter either staring at their eyes (direct gaze condition) or looking elsewhere (averted gaze condition). Autistics did not differ from neurotypicals in their eye behaviours nor their skin conductance responses. Autistics did not appear distressed when they were looking at the experimenter's eyes in the direct gaze condition. However, neurotypicals, compared to autistics, might experience more stress when looking at the experimenter in the averted gaze condition, even after controlling for social anxiety and alexithymia. In comparison to autistics, neurotypicals might be hyper-aroused when they look at someone avoiding eye contact. Based on a bidirectional perspective on interactional difficulties in autism, we speculate that the neurotypicals' distress when their attempts to eye contact are not reciprocated could make their behaviour insistent, which, in turn, could make the autistics uncomfortable. In our study, participants' partner remained passive, displaying no specific reaction when a mutual gaze was shared or not. Future studies should test different partner reactions to gaze in various social contexts.