Purpose: Oral diadochokinesis (DDK) is a standard dysarthria assessment task. To extract automatic and semi-automatic DDK measurements, numerous DDK analysis algorithms based on acoustic signal processing are available, including amplitude based, spectral based, and hybrid. However, these algorithms have been predominantly validated in individuals with no perceptible to mild dysarthria. The behavior of these algorithms across dysarthria severity is largely unknown. Likewise, these algorithms have not been tested equally for various syllable types. The goal of this study was to evaluate the performance of five common DDK algorithms as a function of dysarthria severity, considering syllable types. Method: We analyzed 282 DDK recordings of /ba/, /pa/, and /ta/ from 145 participants with amyotrophic lateral sclerosis. Recordings were stratified into mild, moderate, or severe dysarthria groups based on individual performance on the Speech Intelligibility Test. Analysis included manual and automatic estimation of the number of syllables, DDK rate, and cycle-to-cycle temporal variability (cTV). Validation metrics included Bland-Altman mixed-effects limits of agreement between manual and automatic syllable counts, recall and precision between manual and automatic syllable boundary detection, and Kendall's tau-b correlations between manual and algorithm-detected DDK rate and cTV. Results: The amplitude-based algorithm (absolute energy) yielded the strongest correlations with manual analysis across all severity groups for DDK rate ([tau][subscript b] = 0.7-0.84) and cTV ([tau][subscript b] = 0.7-0.84) and the narrowest limits of agreement (-5.92 to 7.12 syllable difference). Moreover, this algorithm also provided the highest mean recall and precision across severity groups for /ba/ and /pa/, but with significantly more variation for/ta/. Conclusions: Algorithms based on signal energy analysis appeared to be the most robust for DDK analysis across dysarthria severity and syllable types; however, it remains prone to error against severe dysarthria and alveolar syllable context. Further development is needed to address this important issue.