Cognitive Tutor Authoring Tools 2.0 > Using the Tools > Example-tracing Tutors

2. Example-tracing Tutors

This section will walk you through the main phases of authoring an Example-tracing Tutor with CTAT. For a full, example-based tutorial, see the tutorials section of the CTAT web site.

There are four main phases for authoring an Example-tracing tutor:

  1. Demonstrate: in the student interface (typically Java or Flash), demonstrate examples of correct and incorrect behavior for each problem for which students will be tutored. Alternate ways of solving a problem are recorded as separate paths in the graph.

  2. Generalize: in the Behavior Recorder, define ways in which the tutor can recognize a broader range of behavior. Methods for this include specifying ordering constraints so that students can solve a problem in orders other than the one in which the problem was demonstrated; enhancing 'matching' to generalize the conditions under which a demonstrated step is matched; and marking certain steps (links) in the graph as optional.

  3. Annotate: define hints for correct student actions, feedback for student errors, and success messages for correct actions. Associate knowledge components, pieces of acquired knowledge, with steps in the graph, and view a knowledge component matrix of knowledge component occurrence by problem.

  4. Test: use the tutor, playing the role of student attempting to solve the problem. If possible, field the tutor with a user or number of users to get feedback early and often.

2.1. Demonstrate

Authoring a single graph (problem) is done by demonstrating (performing) problem-solving actions. During this phase of authoring, you will commonly have the Behavior Recorder window showing in CTAT along with a student interface with which you'll interact.

To record your problem-solving demonstration, set Author Mode to Demonstrate. Now, when you interact with the student interface, new action links and interface state nodes will appear in the behavior graph displayed in the Behavior Recorder.

Generally, you will demonstrate a preferred, correct solution path, and revisit points in the graph to add common student misconceptions in the form of both incorrect and suboptimal actions, and alternative correct paths. To jump to a state in the graph, click the state node.

When in Demonstrate mode, all steps are initially created as Correct Action links. To mark an action as incorrect or suboptimal, right-click the link's label and select an option from the menu Change Action Type.

The last step of a demonstration should be a done step, that is, a step where the student signals completion of the problem by pressing a Done button.

Save your graph by selecting File > Save Graph As.

2.2. Generalize

A complete graph can be generalized to account for various problem-solving strategies, as well as specific student input and ordering constraints.

2.2.1. Input Matching

In an Example-tracing Tutor, each action that a student performs is compared by the tutor against the set of potential actions described in the graph. Specifically, the tutor compares the student's 'selection' (widget), 'action' (interaction type), and 'input' (entered value) with those on the links of the graph. Whether this comparison is against all links in the graph or a specific subset of links depends on the graph's ordering constraints (see Ordering Constraints).

When CTAT compares a student action with that defined on a link, it can do so precisely (an exact match), or by another method (eg, a regular expression, range, wildcard, or any value). The type of comparison is specified on each link in the graph. By default, the type of comparison is an 'exact' match. Other types of input [action?] matching are available in CTAT and can be specified for a link.

Exact Match

The default matching type. Requires that the student's selection, action, and input match exactly with that specified on the link.

Any Match

Any input value for the specified selection and action is considered a match.

Range Match

An input value within the specified numeric range is considered a match. Includes the minimum and maximum values (eg, 0-100 would match inputs of 0, 2, 10.12, or 100). A range cannot be specified for selection or action—these are exact matches.

Wildcard Match

A value that matches the wildcard pattern is considered a match. A wildcard * denotes any number of characters. Special characters such as {, \, or & should be avoided. A wildcard expression can be used for selection, action, and/or input.

Example 1: "*_text" would match "_text", "this_text", or "123abc_text".

Example 2: "abc*xyz" would match "abcxyz", or "abc123xyz".

Regular Expression Match

A value that matches the regular expression is considered a match. Uses the Java regular expression engine. A regular expression can be used for selection, action, and/or input. Regular expression matching is the most powerful type of matching in CTAT.

Below are six examples that illustrate some of the regular expression conventions. For a complete reference and tutorial, see Sun's Java Regular Expression Lesson.

Example 1: The regular expression ^cat$ would match 'cat', but not 'a cat', 'cat one', or 'Cat'; the '^' character specifies beginning of a line, and '$' specifies end of a line.

Example 2: The regular expression animal. would match 'animals' or animales', but not 'animal'; the '.' character represents any single character.

Example 3: The regular expression [THK]im would match 'Tim', 'Him', or 'Kim', but not 'Jim'; any single character from the set of characters between the square brackets can be used in the match.

Example 4: The regular expression a{3} would match 'aaa' or 'aaaaaa' (it would match this twice), but not 'aa'; the number between the braces specifies the number of times the preceding character must occur to match.

Example 5: The regular expression .*hello would match 'hi hello', 'xhello', or 'hello'; the '*' character represents zero or more occurrences of the preceding character, and '.' represents any character.

Example 6: The regular expression \d\w\W would match '0z!' or '09@', but not 'b1+' or '4!!'; \d represents a digit (0-9), \w represents a word character (a-zA-Z_0-9), and \W represents a non-word character.

2.2.2. Student- and Tool-Performed Actions

Steps in a behavior graph can be either student-performed (steps the student should perform) or tool-performed (steps that the tutor itself should perform). While a student-performed step specifies behavior the student should perform while using the tutor, a tool-performed step specifies an action that the tutor software will perform on the interface. Tool-performed steps are therefore powerful links that can be used to complete problem-solving steps, make changes to the student interface such as hiding or showing a widget, or perform other actions that you specify.

By default, demonstrated steps are stored in the behavior graph as correct steps that a student should perform when using the tutor. Steps such as these have an actor of student; that is, they are performed by the student. For steps where the tutor performs the step, the actor is the tool itself—a tool-performed action.

To specify a tool-performed action:

  1. Create a new link (optional) - while in Demonstrate mode, demonstrate a step in the student interface. For steps that cannot be demonstrated (e.g., hiding an interface widget), create a blank step in the graph by right-clicking a node in the graph and choosing Add Blank Step.

  2. Right-click (Windows) or Control-Click (Mac) on the action label of the link in the graph for which you'd like to specify a tool-performed action.

  3. Select Edit Student Input Matching.

  4. To the right of Actor, select Tutor (see Figure 2.1, “Specifying a tool-performed action”).

Figure 2.1. Specifying a tool-performed action

Specifying a tool-performed action

Note

Tool-performed actions can only be specified for links where the matching type is Exact Match.

Timing of tool-performed actions

A tool-performed action link in a graph is activated by the tutor whenever the state before the tool-performed link becomes the current state. In an ordered group of links, this process is intuitive: when the state preceding the tool-performed action link becomes the current state, the tool-performed action link is then activated. Slightly less intuitive is that this criterion—that a state with a single outgoing tutor-performed action link becomes the current state—applies for unordered groups of links as well (see Section 2.2.3, “Link Groups and Ordering Constraints” below).

2.2.3. Link Groups and Ordering Constraints

In an Example-tracing tutor, student problem-solving steps are represented in a behavior graph (BRD) as links in a graph. Also represented in the graph is the notion of order: in what order (if any) should the correct steps be performed?

In earlier versions of CTAT, a Behavior Recorder graph could be one of two types: ordered or unordered. In an ordered graph, the student had to perform the entire series of steps (links) in the graph in order, on a single path from the start state node to the final node, for the problem-solving behavior to be considered correct. In an unordered graph, the student could perform the steps in the graph in any order for the behavior to be considered correct; for sections where the graph branched, the student had to perform all steps on the branch they chose and could do so in any order. Later, the ordered graph type was extended to include the notion of an unordered group of links—a set of links that, once encountered, could be performed in any order. Groups could only be of the unordered type, and could not be nested.

In CTAT 2.0, the concept of ordering constraints has been extended to allow for an arbitrary number of link groups, each being either ordered or unordered. The root, or universal, group of the graph must also be either ordered or unordered; therefore, a graph with a single group of links emulates one of the simpler graph types (ordered or unordered) of the past.

See the CTAT tutorial on Fraction Addition to see how ordering constraints might be used in a real-world tutoring scenario.

To define link groups:

Note

This task requires editing an XML file in a text editor. As of CTAT 2.0, only a user interface for adding unordered link groups to a single ordered group exists. Although arbitrary nesting of groups is possible, creating such groups can only be accomplished by editing the BRD file in a text editor.

  1. In a text editor, open the desired behavior graph (BRD file) for which you'd like to create link groups.

  2. Identify the link IDs of links that you'd like to group. To do so, locate <edge> elements in the file. Each <edge> element describes a link in the graph. Examine the <edge> element for identifying characteristics such as selection, action, and input values, or hint messages. When you've found an edge (link) that you'd like to group, note the value of its <uniqueID> element—this will be used later when defining groups.

  3. When you've noted the <uniqueID>s of a set of links, you can define a group using syntax similar to that shown in the code example below.

    At the bottom of the BRD file, add an <EdgesGroups> element, including its closing tag, </EdgesGroups>. Within this element, add a <group> element, giving it attributes name and unordered. Name is a descriptive name of your group; unordered is a boolean variable (true or false) distinguishing the group as unordered (true) or ordered (false). Be sure to close to the group element with </group>.

    Within the group element, add <link> elements for each link that you'd like to make a member of that group. Identify the link using the id attribute—this is the value of <uniqueID> that you noted when browsing the <edge> elements. Note that link elements are empty elements (self-closing elements).

  4. Save your BRD file and open it with CTAT (File > Open Graph).

  5. Test the tutor in Test Tutor mode. If the tutor does not work as expected, examine the group syntax in the BRD file to verify that it conforms to the example below. Also verify that the link IDs map to valid (and expected) edge uniqueIDs.

<EdgesGroups>
  <group name="OnesTens" unordered="true">
    <link id="1"/>
    <link id="3"/>
  </group>
  <group name="Others" unordered="true">
    <link id="5"/>
    <link id="7"/>
    <link id="11"/>
  </group>
</EdgesGroups> 
Editing unordered link groups in an ordered graph

Using the Edit Groups dialog, you can define multiple unordered groups of links within a single ordered group. That is, at the highest level, the behavior graph is ordered; and series of links can be grouped to form unordered groups.

In an unordered group, the student can visit the states in that group in any order. The groups themselves, however, must be visited in the order they are shown in the graph. States that are not part of any group must also be visited in the order shown in the graph.

Note

Although arbitrary nesting of groups is possible in CTAT 2.0, the Edit Groups dialog only supports a single level of nested groups: unordered groups within the graph's root ordered group. Also, a path from one node to another can be used in an unordered group only if there are no other paths coming off of it (other than incorrect action or untraceable error links).

To define a new unordered link group:

  1. Select Graph > Edit Groups ...

  2. Click New.

  3. Enter a Group name.

  4. Select a First link description from the first combobox. This is the first link of the group of unordered links.

  5. Select a Last link description from the second combobox. This is the last link of the group of unordered links.

  6. Click Close.

To edit an unordered link group:

  1. Select Graph > Edit Groups ...

  2. Select the name of the group that you'd like to edit.

  3. Make any changes to the Group name, First link, or Last link options.

  4. Click Close.

To delete an unordered link group:

  1. Select Graph > Edit Groups ...

  2. Select the name of the group that you'd like to delete.

  3. Click Delete.

  4. Click Close.

2.3. Annotate

2.3.1. Hints

When a student requests a hint, the first (top-level) hint from the relevant link is displayed. A link can have multiple levels of hints, where each successive hint request by the student displays the next level of hint.

Tip

The last hint is commonly a 'bottom-out' hint, a hint that informs the student of the correct answer so that they can continue. This approach, however, can facilitate students 'gaming' of the tutoring system by clicking through to the final hint for each step of the problem.

Note

Only correct action links can have hints associated with them. If you switch the action type of a correct action link to one of the other types, you will be prompted to choose what to do with the existing hints—you can convert them to buggy (error) messages or remove them from the link.

To define hints for a correct action link:

  1. Click a correct action link in the Behavior Recorder.

  2. Select Edit Hint and Success Messages.

  3. Enter hints into the hint message text areas. Alternatively, you can copy hints from another link or rule by selecting the link or rule name from the combo boxes at the bottom of the Edit Hint and Success Messages dialog box.

  4. (Optional) Click Add Hint Level to define more than three levels of hints.

  5. Click Done.

Figure 2.2. Edit Hint and Success Messages dialog

Edit Hint and Success Messages dialog

2.3.2. Feedback

Feedback for a student action can be specified on all types of links. It is given after a student action is evaluated by the tutor.

For correct action links, feedback is a success message. For an incorrect or suboptimal action, feedback is an error ("buggy") message.

To define a success message:

  1. Click a correct action link in the Behavior Recorder.

  2. Select Edit Hint and Success Messages.

  3. Click More Options.

  4. Enter a success message in the Please Edit Success Message text area.

  5. (Optional) Choose whether or not the success message should be copied to links with a) the same selection/action/input, and/or b) the same production rule.

  6. Click Done.

To define a 'buggy' (error) message:

  1. Click an incorrect or suboptimal action link in the Behavior Recorder.

  2. Select Edit Bug Message.

  3. Enter a buggy (error) message in the text area of the dialog box. Alternatively, you can copy the buggy message from another link in the graph by selecting the link's description from the combo box.

  4. (Optional) Choose whether or not to copy the buggy message to all links with the same selection/action/input.

  5. Click OK.

Figure 2.3. Edit buggy message dialog

Edit buggy message dialog

2.3.3. Skills [Knowledge Components/Labels?]

Each link on a behavior graph can have any number of skill labels associated with it. A skill label is a tag used to describe a piece of acquired knowledge that is associated with a step (link) in the graph. When applied consistently to steps in a problem, skill labels facilitate many types of analysis on the data generated by student use of the tutor. Such analysis based on knowledge components can be accomplished using the PSLC DataShop, a web application that facilitates learning science research through advanced reports and queries. Using the DataShop, you can easily produce learning curves and other reports from log data coded with knowledge component labels.

To collect this data for later analysis, logging must be turned on for the tutor. See Logging for instructions on configuring and enabling logging, including logging to the PSLC DataShop.

Note

If Knowledge Component labels are not visible in the graph, click Graph > Show Skill Labels.

To associate a knowledge component with a step:

  1. Single-click the default skill ('unnamed') or another existing skill that you'd like to change.

  2. Select Edit Skill Name.

  3. In the Edit Skill Name dialog (figure below), enter a new skill name in the combo box or select an existing skill name from the drop-down menu.

  4. (Optional) Enter a new skill set name or select an existing one from the drop-down menu.

  5. Click OK.

Figure 2.4. Edit Skill Name

Edit Skill Name

To disassociate a knowledge component from a step:

  1. Single-click the knowledge component that you'd like to disassociate from the step.

  2. Select Delete Skill Name from Link. You'll be asked whether or not you'd like to delete the skill name.

  3. Click Yes to remove the skill name from the link; to leave the skill association as is, click No.

With knowledge component labels applied, you can view a matrix of knowledge component usage by problem.

To view a knowledge component matrix:

  1. Select Windows > Skill Matrix.

  2. Using the Problems Organizer, select a subset of problems (or a single problem) by selecting course, unit, section, and problem name from the drop-down boxes.

  3. Click OK to view the skill matrix for the problems you've selected.

The Skill Matrix (pictured below) shows problem names vertically along the left-hand side of the table and skill names horizontally across the top. As skill names are often long, shortened identifiers are used (S1, S2 ...) to represent skills; these are defined below the table. The numbers in the table are counts of the number of times each skill appears in each problem.

Figure 2.5. Skill Matrix (Knowledge Component Matrix)

Skill Matrix (Knowledge Component Matrix)

2.4. Test

At any time during tutor authoring, you can test the behavior of your Example-tracing tutor. To do so, make sure the Tutor Type is set to Example-tracing Tutor and Author Mode is Test Tutor (see Figure 2.6, “Tutor Type and Author Mode for testing an Example-tracing Tutor” below).

Figure 2.6. Tutor Type and Author Mode for testing an Example-tracing Tutor

Tutor Type and Author Mode for testing an Example-tracing Tutor

In this mode, you can use the tutor, playing the role of a student attempting to solve a problem. Feedback, hints, and other messages will be enabled.

To jump to a state of the problem—partially solved, for example—click a node in the behavior graph. The student interface will update to reflect that state. You can then continue testing the tutor from that state.

To reset the problem, click on the start state of the behavior graph (the first node in the graph).

Changing the tutor's feedback and hint messages can be done while in Test Tutor mode. To demonstrate new behavior for the problem, you will need to switch to Demonstrate mode