Choosing between machine learning and knowledge engineering is not only a question of predictive accuracy. For teams building behavior models from digital interaction logs, the right approach also depends on development effort, interpretability, and whether the model must work in systems or populations beyond the original training context. A comparison using detection of âgaming the systemâ in intelligent tutoring environments illustrates how these trade-offs can play out.
Machine learning vs knowledge engineering for behavior modeling
Knowledge engineering turns expert understanding of a behavior into explicit rules. Machine learning instead uses labeled examples and measured features to identify relationships between observed actions and a target behavior. A hybrid approach combines expert-derived behavioral knowledge with algorithms that can identify useful patterns in data.
The comparison focused on âgaming the systemâ: using a learning environmentâs support functions to reach answers through guessing, copying, or inappropriate help-seeking rather than working through the material. Examples include rapidly requesting hints to reveal an answer, entering sequences of similar guesses, or reusing answers across problem steps.
Although the case concerns digital learning rather than factory automation, its methodological lesson is relevant to any team developing predictive models from operational logs: a model that performs well in one setting may be difficult to explain or may not transfer reliably to another.
How the three approaches were evaluated
The models were developed using interaction data from 59 students working in Cognitive Tutor Algebra over a school year. Human-coded samples of five consecutive actions were divided into a training set and a held-out test set. The models were compared on predictive performance in the original environment, interpretability, and transfer to other intelligent tutoring systems.
Two measures were used for predictive performance. Cohenâs Kappa measures agreement beyond chance, while the area under the receiver operating characteristic curve (AUC) measures how well a model distinguishes examples from two classes across decision thresholds. The knowledge-engineered model produced binary predictions, so an AUC was not reported for it.
Machine learning: less elicitation, limited transfer
The supervised machine-learning model used 26 features describing actions, timing, help requests, errors, and estimates of student knowledge. A J48 decision tree learned a sequence of decisions from the labeled examples. On the held-out test data, it achieved a Kappa of 0.218 and an AUC of 0.691.
In this case, machine learning required fewer resources to develop than the approaches involving expert knowledge elicitation. Once labeled data and features were available, the model could be trained without a lengthy process of observing and interviewing a domain expert. However, its performance declined when applied to other environments, particularly ASSISTments. Its decision tree was also complex, with 448 nodes, making the full model difficult to interpret despite the familiar if-then structure of individual branches.
This approach may suit projects focused on detecting a defined behavior within a stable context, especially when development resources are constrained. The result also cautions against treating held-out testing within one environment as evidence of transfer to new contexts.
Knowledge engineering: explicit rules and stronger generalization
For the knowledge-engineered model, a specialistâs judgments were examined through observations and interviews. The process identified behavioral cues such as pauses around actions, repeated or similar answers, and switching problem steps without completing the current one. These cues were organized into interpretable patterns; a clip was flagged when it contained one of the identified patterns.
The model achieved a test-set Kappa of 0.330 in Cognitive Tutor Algebra, above the machine-learning modelâs 0.218. More importantly, it was the most interpretable approach: its rules corresponded to patterns that could be inspected and discussed. When tested in other tutoring environments, it showed relatively stable performance and, in some comparisons, outperformed the other approaches.
That transparency and transfer came with a cost. Developing the model required access to an expert and a knowledge engineer, as well as a careful, iterative elicitation process. Explicit rules also reflect what the expert was able to articulate; they may not capture meaningful behavior outside that elicited knowledge.
Hybrid models combine expert knowledge with pattern discovery
The hybrid approach used expert-identified behavioral constituents to create features, then applied machine learning to select and combine them. This allowed the algorithm to search for combinations of actions that might be difficult for an expert to specify directly.
On the original test set, the hybrid ensemble achieved Kappa of 0.376 and AUC of 0.876. It also transferred comparatively well to the scatterplot and ASSISTments datasets. The hybrid models retained interpretable elements because their features were grounded in identifiable action patterns, although algorithmically selected patterns are not guaranteed to match an expertâs explanation of behavior.
Building the hybrid model required the knowledge-elicitation work plus additional effort to develop methods for combining that knowledge with machine learning. It therefore offered a strong balance across performance, interpretability, and transfer, but was the most resource-intensive option.
Choosing an approach for the intended use
- Choose machine learning when the main objective is detection in a specific context and development resources are limited, while recognizing potential limits in interpretability and transfer.
- Choose knowledge engineering when explanations and application across different environments are priorities, and expert time is available for structured elicitation.
- Consider a hybrid model when strong performance across multiple criteria is important enough to justify additional development work.
These findings come from one behavior and a particular set of learning systems, so they should not be treated as a universal ranking of modeling methods. Performance and transfer can depend on the behavior being modeled, the features available, the data, and the development process.
Conclusion: match model design to operational needs
The practical takeaway is to define how a behavior model will be used before selecting its development method. If it must support decisions across changing systems, explain its classifications, or inform further analysis, transfer and interpretability matter alongside predictive scores. Machine learning can reduce the effort of model construction, knowledge engineering can make assumptions explicit, and hybrid methods can combine structured domain knowledge with data-driven pattern discovery. The best choice is the one whose trade-offs fit the modelâs intended role and the resources available to build and validate it.
Access Official Technical PDF