The production rules file, productionRules.pr, contains the Jess production rules that model student procedural knowledge and misconceptions.

Jess functions are typically defined at the top of the production rules file so that production rules can use them. Alternatively, functions can be defined in the Jess templates file, or in a separate file which is referenced via a (require* ) function that appears at the bottom of the templates file. See the Jess manual for more on the require* function and its reciprocal function, provide.

CTAT loads the production rules file whenever a new behavior graph is loaded, or when the start state of a loaded behavior graph is clicked. You can also load the production rules file at any time while in CTAT.

The templates file, wmeTypes.clp, contains definitions of Jess templates. A template in Jess is a description of a fact type; every fact in working memory has a template. See the Jess reference documentation on templates for more on this construct.

CTAT loads this file when a behavior graph is loaded; or when the start state of a loaded behavior graph is clicked.

The templates file typically ends with the following line to notify Eclipse that Jess templates have been parsed:

(provide wmeTypes)

CTAT creates an initial set of Jess templates when the start state of a new graph is created. This representation is based on the visual elements—the widgets—that appear in the student interface. CTAT creates a Jess template for each widget type in the interface.

In addition, CTAT defines the following templates in the templates file:

The facts file contains the Jess facts that make up working memory for a given problem. CTAT loads this file (if it exists) when a behavior graph is loaded; otherwise, facts are created based on the current interface and templates. One facts file (with file extension .WME) should exist for each behavior graph (with file extension .BRD).

CTAT creates an initial representation of working memory when the start state of a new graph is created. This representation is based on the visual elements—the widgets—that appear in the student interface. CTAT creates a Jess fact for each widget in the interface. This initial structure is typically expanded to account for non-visual structures or more complex facts (eg, facts composed of other facts). The initial working memory contents are not saved to file, but loaded into working memory.

The facts file typically begins with the following line to specify to Eclipse the templates file to read:

(require* wmeTypes "wmeTypes.clp")

The behavior graph is the file that represents the starting state of a problem, and, optionally, the correct and incorrect steps that comprise student problem-solving behavior for that problem.

In a cognitive tutor (Jess), the start state of the behavior graph is the only required node: the start state provides the initialization information necessary for CTAT to create a Jess representation of the problem. Tutoring is provided to the student based on the cognitive model and model-tracing algorithm; therefore, no other graph information is required. A complete behavior graph, however, can be used for semi-automated testing.

In addition to helping you to construct an Example-tracing Tutor, the Behavior Recorder can aid in cognitive model planning, development, and testing.

As a planning tool, the Behavior Recorder allows you to:

As a testing tool, the Behavior Recorder allows you to:

3.2.1.2. Testing with the Behavior Recorder

Behavior graphs can also serve as test cases for a cognitive model. In this way, a behavior graph is a specification for how the model should behave on the steps of the problem.

To test a cognitive model against a behavior graph:

1. Load the behavior graph into the Behavior Recorder (File > Open Graph).

2. Check that the cognitive model has loaded by entering the command (rules) in the Jess Console. The console should print the names of your production rules and end with a count of the total number of rules.

3. Select Cognitive Model > Test Cognitive Model on All Steps, or press CTRL+T.

Two indicators will appear notifying you of the results of the test. The first is the test report window (shown below).

Figure 2.7. Production Model Test Report Window

This report describes the results of a comparison between the graph's specification of correctness for an ordered list of steps and the model-tracing algorithm's evaluation of that same list of steps. Here, the term 'step' refers to a student action (technically represented by a selection-action-input triple).

The test operates by first determining the possible paths (or path) from start state to done state, the last state of the graph. For each of these paths, the model-tracing algorithm traces the path to each step, and presents its evaluation (incorrect, correct) to the test. The test compares the link type in the graph to the results of the model-tracing algorithm's trace. A comparison is consistent if the link type defined in the graph matches the evaluation by the model-tracer; it is inconsistent if the two do not match, or if a state in the graph is unreachable by the model-tracing algorithm. A state is unreachable if it appears beyond a buggy (incorrect action) link in the graph—a step by the model-tracing algorithm not present in the graph—or if appears in the graph but is not traced by the model-tracing algorithm.

The report also references good and bad changes. As the report indicates, a 'good' change is from inconsistent to consistent; a 'bad' change is from consistent to inconsist. This comparison is presented if the Test Cognitive Model on All Steps command has been run previously during the authoring session. Typically, you run the test and upon finding inconsistencies, modify the production rules and run the test again.

The second indicator is color changes to the links in the behavior graph. [TBS: how are colors determined? dotted for inconsistent? grayed for unreachable?]

To reset link colors modified by the test:

• Select Cognitive Model > Reset Link Colors.

Lastly, the Behavior Recorder allows you to jump to recorded states, moving both working memory and the student interface to the desired state. To jump to a recorded state, click the desired state in the behavior graph. Note the updates to the student interface, to the working memory window, and to the conflict tree as the cognitive model is traced against the steps outlined in the graph.

The working memory (WME) editor allows you to inspect and modify a cognitive model's working memory at any time.

Figure 2.8. Working Memory (WME) Editor

The Conflict Tree is a debugging tool that shows you which rules correctly predicted the student's selection/action/input (S/A/I) and which rules fired, but only partially activated. Its purpose is to show the space of rules that were explored by the production system interpreter as it tried to find a "chain" or "path" of rules that correctly predicted the S/A/I. This space is always in the form of a tree.

The Conflict Tree displays rule activations in terms of 'chains' of rules formed during the model trace. In Jess model tracing, a chain is a point in the model-tracing search where the effects of one rule's activation to working memory cause another rule to fire. The chaining point is represented by a folder icon in the tree.

In addition, the Conflict Tree is the launching point for 'Why Not?' inquiries (e.g., 'I see that a rule did not fire at all, but why not?').

For a rule that predicts student S/A/I (via the test-SAI function on the right-hand side of the rule), the results of that prediction are shown to the left of the rule name in the columns S, A, and I. A green checkmark indicates that the selection, action or input was predicted correctly. A red X signifies that the selection, action or input was not predicted correctly. As soon as the production engine encounters a rule activation where the left-hand side (LHS) matches the asserted facts and the selection/action/input (S/A/I) were all correctly predicted, it fires that rule and stops in that state. Hence an entry in the conflict tree having three green checkmarks signifies the rule activation that fired and ended a successful model-tracing search for the student's action; the production engine's current state should reflect the actions of that rule's RHS.

Figure 2.9. Conflict Tree: Rule's predicted SAI vs. student's actual SAI

A rule can be in any of the following states at a given point in Conflict Tree:

The Conflict Tree distinguishes only Not Activated ("Failure to match LHS"), Activated But Not Fired ("SUCCESSFUL MATCH OF THE LHS"), Fired But Incorrect S/A/I prediction ("Failed to Match SAI"), and Fired, Correct and Complete S/A/I Prediction.

There are two ways tutor can fail to match what the student did:

You may have a rule that was partially activated (i.e., an instance of a rule where a rule has some, but not all, of its conditions satisfied) or a rule that did not activate at all. In these cases, you may want to see more details concerning the match. Alternatively, you may want to explore the details of a matched rule (one depicted with three green checkmarks in the conflict tree). The Why Not? window provides further detail about each search node depicted in the Conflict Tree.

3.2.4.1. Interpreting the Why Not? Window

The top third of the Why Not? window (Figure 2.10, “Why Not? Window: rule definition”) displays the production rule that you're examining. Here, all variables are given a background color, which corresponds to the color used in the table below the rule definition (Figure 2.11, “Why Not? Window: variable values”).

If you hover over a variable in the top window with your mouse cursor, a tooltip will appear displaying the variable's value (in the case of a simple variable) or table of fact information (in the case of a fact reference). Simultaneously, a black outline will appear around the corresponding variable row in the middle area.

Figure 2.10. Why Not? Window: rule definition

Partial Activations shows the various activation attempts by Jess. Each activation is an attempt to match the LHS of the rule with the facts in working memory; and all variables must be 'bound' (matched) to activate the rule. If all variables could not be bound for a given attempt, that attempt results in a partial activation. Green indicates that all variables were bound successfully; red indicates that some variables were not bound.

Click a partial activation in the list box on the left to see the alternate mappings; the variables table to the right will update to reflect the particular activation, as will the highlighting in the rule definition window above.

Of particular importance is the line to the right of Partial Activations (depicted in Figure 2.11, “Why Not? Window: variable values”). Below, this line reads 'LHS matched'. When you click on a partial activation, this line will update to reflect the first disparity in the comparison between the rule's LHS definition and working memory.

Figure 2.11. Why Not? Window: variable values

Note

You will often see more than one activation listed in Why Not? because the pattern-matching algorithm in the Jess rule engine almost always makes multiple attempts to bind variables, as it has to try different values for those variables.

The Jess console is a text window and command line for interacting directly with the Jess interpreter.

You can define breakpoints for debugging a cognitive model.

Figure 2.12. Defining Breakpoints