Interpretation
A systematic review by Ahmed et al.31 concluded that a combination of global and task-specific assessment tools appears to be the most comprehensive solution for observational assessment of technical skills. This is supported by findings in the RHAS, CAT-LSH and OSALS, tools which all consist of a general and procedure-specific checklist and are validated in studies with relatively strong methodology. It has been shown in a simulation setting that evaluation of a clinical competence solely using a procedure-specific checklist does not preclude incompetence in terms of technical ability and safety.32 Identifying safety issues requires the inclusion of assessment using a global rating scale. By adding GERT the operative substeps prone to errors, can be identified.
Savran et al.23 asserted that their assessment tool meets the criteria for summative assessment, using the contrasting group method to set a pass/fail score. Similar to most studies in our scoping review, the authors grouped the surgeons according to surgical load, with experienced or expert surgeons defined according to the number of cases performed, even though this is not an objective measure of competency, just as a pre-set standard must exist to establish summative assessment.1
Focused on formative feedback, high-stakes assessment and programme evaluation, Hatala et al.13 used Kane’s framework to evaluate OSATS and found reasonable evidence in terms of scoring and extrapolation for formative and high-stakes assessment. For programme assessment, there was validity evidence for generalisation and extrapolation but a complete lack of evidence regarding implications and decisions based on OSATS scores. This calls for more research.