Table 2: Rating system that individuals in journal teams used
to assign their own self-assessment scores (SA-score)
In order to synthesise and describe processes that define best practice
in peer review and to define potential obstacles encountered by journal
teams, five raters (TG, CG, ECM, EM, MW) also rated the answers to each
question using the qualitative feedback provided by the journal teams.
The journal responses were anonymised with a journal number assigned to
each journal. The questions were then divided between the raters
according to the five Essential Areas, with one rater assessing and
rating every journal response within a specific Essential Area. The
raters followed a consistent rating system (Table 3) devised after the
questions and after the self-assessments had been completed. The raters
scored the responses from one to three, and in addition rated a
non-applicable answer as ‘N/A’ and an unaccountable answer as ‘U’. The
raters’ scores are referred to as Rated scores (R-scores).