On the other hand, the project has included the evaluation of itself as a transversal process. The aim of this evaluation is to gather enough evidence regarding the educational innovation project development. In fact, a formative evaluation is proposed, implying a continuous monitorization of it, allowing to reorientate the action depending on the follow-up results. The aim of this article is to register the evaluation protocol that will be developed and end up after the project will be done.
Method
Design and key questions
The evaluation is being developed according to the contribution analysis approach (Mayne, 2008) basing on the Theory of Change of the project. Regarding the scope, the evaluation is focused on the degree of compliance with the project targets. That is:
- What extent the project has found an alternative teaching evaluation process more contextualized and flexible than the current process at the University of Málaga.
- What extent it is from the lecturer and students debate.
- What extent it uses various gathering data instruments and different analysis techniques.
- How useful it is for lecturers, students, and for the institution indeed.
In a coherent way, the purpose of the evaluation is to analyze the goodness of the project, its usefulness, and the degree of compliance of its goals. In this sense, the initially intended evaluation users and stakeholders are:
- The project team project (see annex): the project coordinator, lecturers and collaborators.
- The other University of Málaga lecturers.
- The University of Málaga quality department.
The results from the evaluation will be useful for the project coordinators to know what extent the partial results will be reaching, monitoring the developing, making decisions about changes or adaptations if need, and analyzing the outcomes, the degree of compliance, fails and success, and planning new ways of continuity. Another highlight consequence is the chance to export this new methodology of teaching evaluation to other universities. In the same way, the evaluation could be useful for lecturers to change their perception about how useful the teaching evaluation could become.
Taking into account the scope of this evaluation, the main key questions proposed are the following:
- Is the project relevant to current university needs?
- What is the project design quality?
- What are the barriers to reach the project aims?
- How valuable will be the results to stakeholders?
- Does the project produce the intended results in the short, medium and long term?
- If so, for whom, to what extent and in what circumstances?
- What unintended results – positive and negative – does the intervention produce?
- If so, how do these occur?
- What extent are the project using the available resources to achieve the best results to participants and university community?
In accordance with OECD-DAC (1991, 2008) this evaluation observes the following principles:
- Impartiality and independence. In order to contribute to the credibility of evaluation and avoiding bias in findings.
- Credibility. To ensure transparency, the evaluation will be as open as possible with findings made widely available.
- Usefulness. The effort of the evaluation team is targeted to reach relevant and useful results for whole stakeholders.
- Collaborative. All participating parties will be involved in the evaluation process.
Indicators and instruments
Taking into consideration the Theory of Change of this project, the indicator to monitor its development are:
- Number of meetings. They will be one per quarter. Besides, the project designers set that meeting will have 90% of attendees at least.
- At least 80% of subjects by lecturers involved in the project will be evaluated with the new method
- The assessment of the whole project by participants.
- That the new method became requested by other lecturers or university staff in a time period upper one year.
Minutes and documents from meetings will be taking into consideration to get data for indicators. At the same time, other instruments could be used, such as interviews or scales, if need.
Another important element is the objectivity of the project evaluation process. In this sense, it will be formed a group of external experts. The role of this will be to analyze the evaluation and inform respect the validity, utility, and make a collective and consensual judgement regarding the evaluation.
As it could be not otherwise, data and information will be collected with the express permission of participants. The evaluation data will be stored and maintained by the evaluation team coordination following the current regulations about personal data escrow. The data will be used only for purposes set out in the project, that includes academical purposes. In this regard, data from evaluation could be used in academical meetings, scientific workshops, academical papers, and other dissemination academical and scientific methods, always ensuring anonymity.
Roles and responsibilities in the evaluation
The evaluation is the responsibility of all member of the project (see annexe) where the coordinators are the signer of this article. The evaluation data will be collected by the team members, although, people trained for that purpose will be enrolled if necessary.
After analysing data, the reports will be written by coordinators. They will be turned into public by the coordinators, in several ways: the project blog, specific brief report for mass-media, University of Málaga Education School social networks, meetings, etc.
Eventually, any stakeholders interested in the evaluation and their results can contact with the coordinators in order to ask for information about.
Plan and time
The project was launched throughout the 2019-20 academic year. Due to the Covid situation, its initial plan was delayed for a semester. In the first semester of the 2020-21 academic year, group work is carried out to define the new teaching evaluation methodology. During the second semester of the 2020-21 academic year (January to June 2021) the application will be carried out in the subjects that are available, depending on the academic situation. For the second half of the first semester of the academic year 2021-22, the final results of the evaluation are expected.
Conclusions
We live at a turning of the teaching paradigm, mainly in Higher Education. During the last months, lecturers have been dealing with social demands from institutions, as well as syllabus and technology challenges along the pandemia. In this vein, it is a need to think about how to face so accelerate changing avoiding the reductionism and teacher's malaise either. We observe astonished how many researching proposals reach our emails, besides assessments and educational evaluation in pandemia time based only on questionnaires. Having said that, evaluations based on questionnaires can not become the only method to embed improvements or making decisions regarding lecturer careers.
Teaching evaluation is a powerful key for mediation and awareness into the Educational System. In this project, using the Theory of Change as a referent, we are conscious about how important the impact of this evaluation approach could have, mainly in the case of accountability and public scrutiny, among others. The teaching evaluation quality is used as criteria to facilitate, or not, the promotion into the teaching career, or, to analyze critically the lecturers' potentialities and weaknesses themselves.
Using the teaching evaluation in different areas, for different aims, including monetary incentives, or highlighting social recognition and status could generate suspicion, distrust, and misunderstanding among lecturers. In this sense, agencies, public administration, managers, so on, are really concerned and bet for a teaching evaluation of quality as an indisputable factor in the educational system warranty.
In this sense, not only the evaluative process control is externalized at companies and private or independent institutions, but we can discover models encouraging student participation to assess the teaching activity too, filling any kind of questionnaires, and appraisal scales. Although we do not rule out these instruments and devices, this project of innovation assumes the relevance of the self-evaluation, internal audit, and self-critic from the group itself into a democratic framework, looking for sustainable alternatives, more empathic, and with more projection, as intra as inter institutions. Evenly, it is highlighted that teaching quality evaluation needs to be focused on specific projects, initiatives, or actions and not only on merely individual or comparative factors. Besides, in this case, the appraisal is not from the final result of a traditional questionnaire, but it is used inclusive evaluative designs which let analyze more coherent and balanced project developments, at the same time that causal attribution is sought in a cooperative way.