1 This appendix provides information on Usability Testing (UT) and uses ECDIS as a
closely aligned example relevant to future e-navigation systems. This UT example aligns
with Stage 4 of the HCD process for evaluating the performance of essential tasks by
competent users. The selection of test participants is important and has a bearing on
the quality of test results.
2 If tasks require operations based on navigational experience or knowledge,
then appropriate participants should be selected. Tasks that are generally performed by
less experienced or knowledgeable personnel should be similarly tested.
3 The UT activity involves the following steps:
4 Only the steps related to planning and evaluation of results are explained
in this appendix since these steps are the most important.
5 A UT plan should be developed by defining scenarios and identifying the
most important or critical tasks that users must perform. Users and the test environment
should also be identified.
6 A goal-based approach should be used when setting the tasks with the aim of
facilitating flexible yet practical assessment of the target system.
7 The following steps can be part of the goal-based approach:
-
.1 definition of goals based on the context of use of the system, which may
come from functions stipulated in internationally agreed performance
standards;
-
.2 specify functional requirements or the criteria to be satisfied in order
to conform to the goals, taking into account the relevant performance
standards and user requirements;
-
.3 specify "usability" requirements that must be achieved during testing,
based on the aspects of effectiveness, efficiency and satisfaction; and .4
prepare tests that will assist in verifying the extent to which the system
conforms with the identified goals.
8 In the case of ECDIS goals could include "to plan and display the ship's
route for the intended voyage and to plot and monitor positions throughout the voyage",
based on SOLAS regulation V/19.2.1.4.
9 Similarly, functional requirements for ECDIS could be defined based on the
IMO's ECDIS performance standard
(resolution MSC.232(82)). The following example of ECDIS functional
requirements relates to nautical data handling necessary for safe navigation, with the
following sub-requirements:
-
.1 chart data handling (example: change display orientation, mode, etc.);
-
.2 own ship data handling (example: read position, speed, etc.); and
-
.3 tracked target (TT) and radar data handling (example: show TT symbols
overlaid on ECDIS chart screen, etc.).
10 In the case of ECDIS, "usability" can be evaluated in terms of user
effectiveness and efficiency for each of the tasks and overall satisfaction of the
system (for example through subjective evaluation). As highlighted in table 1, measures
of effectiveness relate the selected user goals to the accuracy and completeness with
which these goals can be achieved. In this example, the achievement rate is used as a
measure of "effectiveness". The four levels and their criteria are listed in table 1.
Usability outcomes can be based on the "dialogue principles", as identified under ISO
9421-110, using UT methods based on ISO/TR16982. It is important that methods for
evaluating usability are selected when devising the UT plan.
11 Scenarios and test tasks can also be created to satisfy the functional
requirements. The following are examples of tasks for a basic display handling scenario:
-
Task 1: Adjust display modes and scale to meet operator's needs
-
Task 2: Obtain information about a lighthouse
-
Task 3: Measure the bearing and distance to a landmark
- Task 4: Overlay a tracked target symbol and obtain information about the
target
12 Criteria should be set to establish the degree to which tasks are achieved
and also to capture user feedback on satisfaction with the operation of the system.
Table 1 provides simple examples of achievement criteria for each task. Quantitative
performance criteria such as time taken to complete tasks can also be included.
13 For the evaluation of system performance the level of task achievement can
be useful (e.g. the time required to complete tasks). Questionnaires can assist with
overall subjective system evaluation.
Table 1: Examples of achievement criteria for measures of effectiveness
| Achievement
level
|
Criteria
|
| Achieved
|
1
|
✓ Participants understood the information
correctly and operated properly with confidence.
✓ In
case participants made some mistakes but noticed the mistakes
immediately and achieved the goal smoothly, this should be considered
"achieved smoothly".
|
| 2
|
✓ Participants completed the task properly byt
hemselves, even with some hesitation or confusion.
✓ In
case participants took time to find the first action or to recover from
errors but completed the task, this should be considered "achieved not
smoothly".
|
| Not achieved
|
3
|
✓ Even if participants completed the task
properly, it should be considered "not achieved with errors" if the
participants could not understand the information correctly or if
achievement took a large number of interactions.
|
| 4
|
✓ Participants could not complete the task by
themselves and needed suggestions from the moderator.
|
14 To satisfy quality management system requirements a UT report should be developed.
ISO/IEC 25062 provides an example for a template that can be used for a UT report.
UT methods that can be applied at various stages in the life cycle (based on ISO/TR
16982)
| Name of the method
|
Direct involvement of users
|
Short description of method
|
Life cycle stage
|
| Observation of users
|
Y
|
Collection of information in a precise and
systematic way about the behaviour and the performance of users, in the
context of specific tasks during user activity.
|
4
|
| Performance-related measurements
|
Y
|
Collection of quantifiable performance
measurements in order to understand the impacts of usability
issues.
|
4
|
| Critical incident analysis
|
Y
|
Systematic collection of specific events
(positive or negative).
|
1
|
| Questionnaires
|
Y
|
Indirect evaluation methods which gather users'
opinions about the user interface in predefined questionnaires.
|
1 and 2
|
| Interviews
|
Y
|
Similar to questionnaires but with greater
flexibility involving face-to-face interaction with the
interviewee.
|
2
|
| Thinking aloud
|
Y
|
Involves having users continuously verbalize
their ideas, beliefs, expectations, doubts, discoveries, etc. during
their use of the system being tested.
|
3 and 4
|
| Collaborative design and evaluation
|
Y
|
Methods which allow different types of
participants (users, product developers and human factors specialists,
etc.) to collaborate in the evaluation or design of systems.
|
Any
|
| Creativity methods
|
Y/N
|
Methods which involve the elicitation of new
products and system features, usually extracted from group interactions.
In the context of human-centred approaches, members of such groups are
often users.
|
1 and 2
|
| Document-based methods
|
N
|
Examination of existing documents by the
usability specialist to form a professional judgement of the
system.
|
1 and 2
|
| Model-based approaches
|
N
|
Use of abstract representations of the
evaluated product to allow the prediction of users' performance.
|
2 and 3
|
| Expert evaluation
|
N
|
Evaluation based on the knowledge, expertise
and practical experience in ergonomics of the usability
specialist.
|
Any
|
| Automated evaluation
|
N
|
Algorithms focused on usability criteria or
using ergonomic knowledge-based systems which diagnose the deficiencies
of a product compared to pre-defined rules.
|
4
|
| Simulation
|
N
|
Use of computer simulation modelling tools used
for initial evaluations.
|
2 and 3
|