Publications

CTAT

• Aleven, V., McLaren, B.M., Sewall, J., & Koedinger, K.R. (in press). Example-Tracing Tutors: A New Paradigm for Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education (IJAIED), Special Issue on "Authoring Systems for Intelligent Tutoring Systems."
  Full-text article forthcoming—citation posted 2009-03-26
  Show Abstract
  Abstract. Key success criteria for an ITS authoring tool are that (1) the tool supports the creation of effective tutoring systems, (2) the tool can be used to build tutors across a wide range of application domains, (3) authoring with the tool is cost-effective, (4) the tool supports easy deployment and delivery of tutors in a variety of technical contexts, (5) tutors created with the tool are maintainable, and (6) if tutors are used in a research context, the tool must support research-related functionality. The Cognitive Tutor Authoring Tools (CTAT) address all of these requirements to a substantial degree, fully meeting most of them.
  
  CTAT supports the creation of both Cognitive Tutors (Koedinger & Corbett, 2006) and a newer type of tutors called example-tracing tutors. This paper focuses on the latter. Example-tracing tutors evaluate student behavior by flexibly comparing it against examples of correct and incorrect problem-solving behaviors. Example-tracing tutors are capable of sophisticated tutoring behaviors: they provide step-by-step guidance on complex problems while recognizing multiple student strategies and maintaining multiple interpretations of student behavior. On that basis, they should be deemed intelligent tutoring systems. Example-tracing tutors can be built without programming, through drag-and-drop techniques and programming by demonstration. Example-tracing tutors have been built and used in real educational settings for a wide range of application areas.
  
  Development time estimates from a large number of projects that have used CTAT suggest that CTAT improves the cost-effectiveness of ITS development by a factor of 4-8, compared to "historical" estimates of tutor development time. Although there is a lot of variability in these kinds of estimates, they nonetheless support our hope that lowering the skill requirements for tutor creation is a key step toward widespread use of ITS technology. The main contributions of the work are the example-tracing tutor technology and tools for building these types of tutors without programming.
• Aleven, V., Sewall, J., McLaren, B. M., & Koedinger, K. R. (2006). Rapid authoring of intelligent tutors for real-world and experimental use. In Kinshuk, R. Koper, P. Kommers, P. Kirschner, D. G. Sampson, & W. Didderen (Eds.), Proceedings of the 6th IEEE International Conference on Advanced Learning Technologies (ICALT 2006), (pp. 847-851). Los Alamitos, CA: IEEE Computer Society. PDF (2.5 MB)
  Show Abstract
  Abstract. Authoring tools for Intelligent Tutoring Systems are especially valuable if they not only provide a rich set of options for the efficient authoring of tutoring systems but also support controlled experiments in which the added educational value of new tutor features is evaluated. The Cognitive Tutor Authoring Tools (CTAT) provide both. Using CTAT, real-world ”Example-Tracing Tutors” can be created without programming. CTAT also provides various kinds of support for controlled experiments, such as administration of different experimental treatments, logging, and data analysis. We present two case studies in which Example-Tracing Tutors created with CTAT were used in classroom experiments. The case studies illustrate a number of new features in CTAT: Use of Macromedia Flash MX 2004 for creating tutor interfaces, extensions to the Example-Tracing Engine that allow for more flexible tutors, a Mass Production facility for more efficient template-based authoring, and support for controlled experiments.
• Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. (2006). The Cognitive Tutor Authoring Tools (CTAT): Preliminary evaluation of efficiency gains. In M. Ikeda, K. D. Ashley, & T. W. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS 2006), (pp. 61-70). Berlin: Springer Verlag. PDF (392 KB)
  
  Show Abstract
  Abstract. Intelligent Tutoring Systems have been shown to be effective in a number of domains, but they remain hard to build, with estimates of 200-300 hours of development per hour of instruction. Two goals of the Cognitive Tutor Authoring Tools (CTAT) project are to (a) make tutor development more efficient for both programmers and non-programmers and (b) produce scientific evidence indicating which tool features lead to improved efficiency. CTAT supports development of two types of tutors, Cognitive Tutors and Example-Tracing Tutors, which represent different trade-offs in terms of ease of authoring and generality. In preliminary small-scale controlled experiments involving basic Cognitive Tutor development tasks, we found efficiency gains due to CTAT of 1.4 to 2 times faster. We expect that continued development of CTAT, informed by repeated evaluations involving increasingly complex authoring tasks, will lead to further efficiency gains.
• Koedinger, K. R., Aleven, V., Heffernan. T., McLaren, B. & Hockenberry, M. (2004). Opening the Door to Non-Programmers: Authoring Intelligent Tutor Behavior by Demonstration. In the Proceedings of 7th Annual Intelligent Tutoring Systems Conference. Maceio, Brazil. PDF (404 KB)
  Show Abstract
  Abstract. Intelligent tutoring systems are quite difficult and time intensive to develop. In this paper, we describe a method and set of software tools that ease the process of cognitive task analysis and tutor development by allowing the author to demonstrate, instead of programming, the behavior of an intelligent tutor. We focus on the subset of our tools that allow authors to create “Pseudo Tutors” that exhibit the behavior of intelligent tutors without requiring AI programming. Authors build user interfaces by direct manipulation and then use a Behavior Recorder tool to demonstrate alternative correct and incorrect actions. The resulting behavior graph is annotated with instructional messages and knowledge labels. We present some preliminary evidence of the effectiveness of this approach, both in terms of reduced development time and learning outcome. Pseudo Tutors have now been built for economics, analytic logic, mathematics, and language learning. Our data supports an estimate of about 25:1 ratio of development time to instruction time for Pseudo Tutors, which compares favorably to the 200:1 estimate for Intelligent Tutors, though we acknowledge and discuss limitations of such estimates.
• Heffernan, N. T., Koedinger, K. R., & Aleven, V. A. W. M. M. (2003). Tools Towards Reducing the Costs of Designing, Building, and Testing Cognitive Models. The 2003 Conference on Behavior Representation in Modeling and Simulation, BRIMS 2003. DOC (291 KB)
  Show Abstract
  Abstract. We are developing a suite of Cognitive Tutor Authoring Tools (CTAT) intended to make tutor development both easier and faster for experienced cognitive modelers and possible for potential modelers who are not experts in cognitive psychology or artificial intelligence programming. Our concrete goal is to experimentally demonstrate a reduction in development time by a factor of three. We are employing Human-Computer Interaction (HCI) methods and Cognitive Science principles, as we have done before, to design development tools that reduce programmer time. Our preliminary analytic and empirical analyses compare use of CTAT with use of our current develop environment and indicate a potential reduction in development time by a factor of about two. These early quantitative results are less important than the specific guidance that such analyses provide as we iteratively converge on demonstrably more cost-effective cognitive tutor development tools.
• Koedinger, K. R., Aleven, V. A. W. M. M., & Heffernan, N. T. (2003). Toward a Rapid Development Environment for Cognitive Tutors. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 11th International Conference on Artificial Intelligence in Education, AI-ED 2003 (pp. 455-457). Amsterdam: IOS Press. PDF (412 KB)
  Show Abstract
  Abstract. We are developing a suite of Cognitive Tutor Authoring Tools (CTAT) intended to make tutor development both easier and faster for experienced modelers and possible for potential modelers who are not experts in cognitive psychology or artificial intelligence programming. Our goal is to demonstrate a reduction in development time by a factor of three. We employ Human-Computer Interaction (HCI) methods and Cognitive Science principles to design development tools that are both useful and useable. Our preliminary analytic and empirical analyses compare use of CTAT with use of our current develop environment and indicate a potential reduction in development time by a factor of about two.

Collaboration

• Harrer, A., McLaren, B., Walker, E., Bollen, L., Sewall, J. (2005). Collaboration and Cognitive Tutoring: Integration, Empirical Results, and Future Directions. In C.-K. Looi et al. (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp.266-273). Amsterdam: IOS Press. PDF (521 KB)
  Show Abstract
  Abstract. In this paper, we describe progress we have made toward providing cognitive tutoring to students within a collaborative software environment. First, we have integrated a collaborative software tool, Cool Modes, with software designed to develop Cognitive Tutors (the Cognitive Tutor Authoring Tool). Our initial integration provides a means to capture data that acts as the foundation of a tutor for collaboration but does not yet fully support actual tutoring. Second, we've performed two exploratory studies in which dyads of students used our software to collaborate in solving modelling tasks. These studies uncovered five dimensions of observed behavior that point to the need for abstraction of student actions to better recognize, analyze, and correct collaborative steps in problem solving. We discuss plans to incorporate such analyses into our approach and to extend our tools to eventually provide tutoring of collaboration.
• McLaren, B., Bollen, L., Walker, E., Harrer, A., Sewall, J. (2005). Cognitive Tutoring of Collaboration: Developmental and Empirical Steps Towards Realization. In the Proceedings of the Conference on Computer Supported Collaborative Learning Conference (CSCL-05). Taipei, Taiwan. PDF (295 KB)
  Show Abstract
  Abstract. In this paper, we describe developmental and empirical steps we have taken toward providing Cognitive Tutoring to students within a collaborative software environment. We have taken two important steps toward realizing this goal. First, we have integrated a collaborative software tool, Cool Modes, with software designed to develop Cognitive Tutors (the Cognitive Tutor Authoring Tool). Our initial integration does not provide tutoring per se but rather acts as a means to capture data that provides the beginnings of a tutor for collaboration. Second, we have performed an initial study in which dyads of students used our software to collaborate in solving a classification / composition problem. This study uncovered five dimensions of analysis that our approach must use to help us better understand student collaborative behavior and lead to the eventual development of a Cognitive Tutor for collaboration. We discuss our plans to incorporate such analysis into our approach and to run further studies.
• McLaren, B. M., Koedinger, K. R., Schneider, M., Harrer, A., and Bollen, L. (2004). Towards Cognitive Tutoring in a Collaborative, Web-based environment. In M. Matera, S. Comai (Eds.), Engineering Advanced Web Applications: Proceedings of Workshops in Connection with the 4th International Conference on Web Engineering (pp. 167-179). Princeton: Rinton Press. PDF (218 KB)
  Show Abstract
  Abstract. While intelligent tutoring has been applied to collaborative learning environments, it has met with little success so far because of the complexity involved in adding a tutoring component to a collaborative environment. We propose to tackle this problem by using Cognitive Tutors as the basis for our approach and by applying a technique we call Bootstrapping Novice Data (BND). The BND approach involves feeding student log les from a problem-solving tool into tutor development software to create the beginnings of a tutor for the tool. We describe an initial implementation of our approach in which Cool Modes, a collaborative software tool, is integrated with the Behavior Recorder, tutor-authoring software that supports development by demonstration. We show how our initial implementation provides a foundation for an intelligent tutor for collaboration but also discuss some of the challenges ahead.

Bootstrapping

• McLaren, B. M., Koedinger, K. R., Schneider, M., Harrer, A., and Bollen, L. (2004). Bootstrapping Novice Data: Semi-Automated Tutor Authoring Using Student Log Files. In the Proceedings of the Workshop on Analyzing Student-Tutor Interaction Logs to Improve Educational Outcomes. Seventh International Conference on Intelligent Tutoring Systems (ITS-2004), August 2004. PDF (278 KB)
  Show Abstract
  Abstract. A potentially powerful way to aid in the authoring of intelligent tutoring systems is to directly leverage student interaction log data. While problem-solving data has been used in the past to guide the development of tutors, such data has not typically been used as a means to directly construct an initial tutoring system model. We propose an approach called bootstrapping novice data (BND) in which a problem-solving tool is integrated with tutor development software through log files and that integration is then used to create the beginnings of a tutor for the tool. We describe an initial implementation of the BND approach in which Cool Modes, a collaborative software tool, is integrated with the Behavior Recorder, tutor-authoring software that supports development by demonstration. A key to this implementation is a component-based approach in which complementary pieces of software are integrated with little or no change to either software component. We argue that more tutors could be built, and with substantial time savings, using this approach. We discuss some of the lessons learned from this initial effort and from applying the component-based approach, as well as some data analyses that could eventually be performed using the data collected during BND.