Interpretation
A systematic review by Ahmed et al.31 concluded that a
combination of global and task-specific assessment tools appears to be
the most comprehensive solution for observational assessment of
technical skills. This is supported by findings in the RHAS, CAT-LSH and
OSALS, tools which all consist of a general and procedure-specific
checklist and are validated in studies with relatively strong
methodology. It has been shown in a simulation setting that evaluation
of a clinical competence solely using a procedure-specific checklist
does not preclude incompetence in terms of technical ability and
safety.32 Identifying safety issues requires the
inclusion of assessment using a global rating scale. By adding GERT the
operative substeps prone to errors, can be identified.
Savran et al.23 asserted that their assessment tool
meets the criteria for summative assessment, using the contrasting group
method to set a pass/fail score. Similar to most studies in our scoping
review, the authors grouped the surgeons according to surgical load,
with experienced or expert surgeons defined according to the number of
cases performed, even though this is not an objective measure of
competency, just as a pre-set standard must exist to establish summative
assessment.1
Focused on formative feedback, high-stakes assessment and programme
evaluation, Hatala et al.13 used Kane’s framework to
evaluate OSATS and found reasonable evidence in terms of scoring and
extrapolation for formative and high-stakes assessment. For programme
assessment, there was validity evidence for generalisation and
extrapolation but a complete lack of evidence regarding implications and
decisions based on OSATS scores. This calls for more research.