Homepage --
Objectives --
Guidelines --
Table of contents --
Advisory board --
Contact us
Model-based evaluations
David Kieras
This chapter will include a critical evaluation and survey of systems
for modeling performance that are realized in computer tools or
systems (as opposed to paper-and-pencil). The following systems will
be covered:
- Two modeling systems that are enjoying current success in
industry/military applications. These systems are marked both by their
relative lack of scientific basis combined with a long track record of
application.
- MA&D's IPME(a newer version, basically, of MicroSAINT) and
- COGNET/IGEN,
- 2 or 3 GOMS modeling tools, varying in their basis and history, and
- at least 2 high-end cognitive-architecture systems (e.g., EPIC,
SOAR (or EPIC-SOAR) and ACT-R.
Three of these systems recently participated in an AFOSR modeling
"bake-off" - the AMBR project presented at HFES last year.
For each system, the discussion will include a short summary
description of the system, focussing not on the mechanics of using it,
but rather on the psychology theory that provides the basic "engine"
for using the model, and which governs its coverage. Dimensions of the
discussion will include:
- When in development the systems can be used: E.g. SAINT-class systems
are actually quite agnostic of many human performance and interface
issues, and so can be applied with only the crudest of available
information - like rough SME estimates of task performance times.
Detailed or architectured systems normally work at the actual
interface "keystroke" level of detail and so require a candidate
design.
- Predictive power: does the system have parameters that have to be
estimated or tweaked before credible performance predictions can be
obtained? What can be done to get true design-time prediction?
- Theoretical coherence: is there a clear psychological theory involved
with definite roots in the scientific literature, or is there a lot
of ad-hocery? Does it reflect current thinking on human cognition and
performance? Does it contain an architecture for the human system, or
does it just interconnect black boxes?
There will also be a section on gaps - critical phenomena that need to
be better understood both empricially and theoretically before models
can be deployed confidently in important areas. Two of these are
critical to modelling display-based tasks, common in military
applications: one is "change blindness" and related Visual WM
phenomena - we really don't understand what people remember about the
contents of a display between looks. A second display-related
problem is we need good models of acuity and recognition - if you are
looking at one object on a display, what can you see about other
objects? A third has to do with the speech modality and how it relates to
concurrent and team task performance - e.g. can somebody listen,
talk, and use verbal stm at the same time? It seems doubtful, but
many esp. military situations seem to require this. What are good
models for what can and cannot be done?
Homepage --
Objectives --
Guidelines --
Table of contents --
Advisory board --
Contact us