The current paper intends to exploit the Many Facet Rasch Model to investigate and compare the impact of situations (items) and raters on test takers' performance on the Written Discourse Completion Test (WDCT) and Discourse Self-Assessment Tests (DSAT). In this study, the participants were 110 English as a Foreign Language (EFL) students at Vali-e-Asr University of Rafsanjan in Iran. The students were asked to complete the WDCT and rate themselves on that test. Four raters scored the WDCT tests. According to the FACETS results, there were significant differences in students' performance between the two methods. The stable fit statistics and differing levels of difficulty measures for each test method indicated that each test had a unique way of differentiating the test taker's pragmatic ability. Based on the results, both DSAT and WDCT are acceptable measures for pragmatic ability; however, there are some fitness problems in DSAT. This shows the unpredictable pattern of ratings in the DSAT. It is recommended to have rater training to obtain more accurate results from the DSAT. Finally, the implications were discussed.