Background: For peer assessment, reliability (i.e., consistency in ratings across peers) and validity (i.e., consistency of peer ratings with instructors or experts) are frequently examined in the research literature to address a central concern of instructors and students. Although the average levels are generally promising, both reliability and…

Background: Learning analytics (LA) research often aggregates learning process data to extract measurements indicating constructs of interest. However, the warranty that such aggregation will produce reliable measurements has not been explicitly examined. The reliability evidence of aggregate measurements has rarely been reported, leaving an…

Background: Sound learning design should be based on the constructive alignment of intended learning outcomes (LOs), teaching and learning activities and formative and summative assessment. Assessment validity strongly relies on its alignment with LOs. Valid and reliable formative assessment can be analysed as a predictor of students' academic…

Background: In this technologically advanced era, media literacy is necessary to effectively evaluate the information and understand various biases inherent in media messages. Several media literacy (ML) tools are available; however, we need generic and objective tools that can be applied to all forms of media messages. Objectives: The current…

Background: Automated writing evaluation (AWE) systems, used as formative assessment tools in writing classrooms, are promising for enhancing instruction and improving student performance. Although meta-analytic evidence supports AWE's effectiveness in various contexts, research on its effectiveness in the U.S. K-12 setting has lagged behind its…

Background: A practical test that measures the information and communication technology (ICT) skills students need for effectively using ICT in primary education has yet to be developed (Oh et al., 2021). This paper reports on the development, validation, and reliability of a test measuring primary school students' ICT skills required for…

Background: Computerised adaptive tests (CATs) are tests that provide personalised, efficient and accurate measurement while reducing testing time, depending on the desired level of precision. Schools have different types of assessments that can benefit from a significant reduction in testing time to varying degrees, depending on the area of…

Background: One area that has been dramatically changed by artificial intelligence (AI) is educational environments. Chatbots, Recommender Systems, Adaptive Learning Systems and Large Language Models have been emerging as practical tools for facilitating learning. However, using such tools appropriately is challenging. In this regard, the…

Background: Digital-first assessments leverage the affordances of technology in all elements of the assessment process--from design and development to score reporting and evaluation to create test taker-centric assessments. Objectives: The goal of this paper is to describe the engineering, machine learning, and psychometric processes and…

Background: Critical thinking is essential in modern education, and artificial intelligence (AI) offers new possibilities for enhancing it. However, the lack of validated tools to assess teachers' AI-integrated pedagogical skills remains a challenge. Objectives: The current study aimed to develop and validate the Artificial Intelligence-Critical…

Digital competence is critical for university students to adapt to and benefit from digitally enhanced learning. Prior studies on its measurement mostly focus on educators and relied on factor analyses. However, there is a lack of valid and convenient tools to measure university students' digital competence. This study aimed to develop a digital…

Measuring cognitive load is important in virtual learning environments (VLE). Thus, valid and reliable measures of cognitive load are important to support instructional design in VLE. Through three studies, we investigated the validity and reliability of Leppink's Cognitive Load Scale (CLS) and developed the extraneous cognitive load (EL)…

The use of e-assessment methods raises important concerns regarding the reliability and validity of these methods. Potential threats to validity include mode effects and the possible influence of computer-related attitudes. Although numerous studies have now investigated the validity of online assessments in noncourse-based contexts, few studies…

Background: As educational institutions increasingly operate as multicultural hubs, leaders must navigate the complexities of cultural differences, language barriers and diverse learning styles in digital environments. These challenges are amplified by the lack of non-verbal cues and the asynchronous nature of online communication, which can lead…

Background: The global COVID-19 pandemic turned the adoption of on-line assessment in the institutions for higher education from possibility to necessity. Thus, in the end of Fall 20/21 semester Tel Aviv University (TAU)--the largest university in Israel--designed and implemented a scalable procedure for administering proctored remote…