This research evaluates the psychometric properties of short-answer response items under a variety of grading rules in the context of a mobile learning platform in Africa. This work has three main findings. First, we introduce the concept of a differential device function (DDF), a type of differential item function that stems from the device a student uses to take an assessment. Second, we identify a plausible mechanism for this DDF by examining the keystroke requirements of smartphone and basic mobile phone users. We identify a set of platform design rules to mitigates this bias. Lastly, we suggest that the edit distance of student responses can be used as a tuning parameter to optimize the Cronbach's alpha of the assessment. We find that literal string evaluation performs poorly compared to other grading rules. Partial string matching with an edit distance of two provides the highest reliability across exams. This is a simple yet effective rule, which performs well across a variety of assessments. [This paper is published in: "Proceedings of American Educational Research Association (AERA 2019)." ACM, New York, NY.]