Chapter 22: Evaluations with AIEd systems
Restricted access

This chapter surveys designs of evaluations of AIEd systems. It focuses on non-statistical design choices, such as what to compare one’s system to. Once the study procedure and data collection have been designed, there are textbooks and other resources that prescribe data analytic methods. In order to give the reader a sense of which evaluation designs are common, the chapter surveys evaluations used in recent AIEd publications. This small sample of publications contained three types of systems. (1) Instruction systems were used by students.  Their function was usually to help the students learn. Most evaluations measured the change in student knowledge during the instruction, using a variety of pre-tests and post-tests. Some evaluations also measured behavior during the instruction. (2) Assessment systems were also used by students. Unlike instruction systems, which take actions intended to help students learn, assessment systems only collect and analyze data on student performance. Most evaluations focused on comparing the system’s assessment to a gold-standard assessment from objective measures or expert judgements. (3) Teacher and developer aids were used by instructors and developers outside of class to author or analyze instruction, or by instructors during class to help with conducting the class. There was a large variety of aids, each with its own method of evaluation. Unlike the early days of AIEd research, all publications in the sample that described a system also presented an evaluation of the system. This is a welcome advancement in the maturity of the field.

You are not authenticated to view the full text of this chapter or article.

Access options

Get access to the full article by using one of the access options below.

Other access options

Redeem Token

Institutional Login

Log in with Open Athens, Shibboleth, or your institutional credentials

Login via Institutional Access

Personal login

Log in with your Elgar Online account

Login with your Elgar account