Our Assessments Failed Validation, What Now?
I hear this question often. However, my question in reply – is validation a process that is pass or fail? For this we need to look at the purpose of validation and what we intend to achieve by following this process. First, consider the current definition of validation in the Standards for RTOs 2015:
“…is the quality review of the assessment process. Validation involves checking that the assessment tool/s produce/s valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the training package or VET accredited courses are met. It includes reviewing a statistically valid sample of the assessments and making recommendations for future improvements to the assessment tool, process and/or outcomes and acting upon such recommendations.”
Source: Users’ Guide to the Standards for RTOs 2015
Considering this definition, my viewpoint is that we aren’t making a judgment of pass or fail on validation. We are following a process that enables us to suggest how we can make improvements to assessment tools, processes and outcomes.
Image by Serj Marco
As a vocational education and training (VET) consultant, I conduct validations of assessment all the time. I don’t think I have ever put on a validation report that the assessment was absolutely perfect and recommended zero changes at all. Does that mean every validation I have ever conducted has resulted in assessments that have ‘failed’ validation? No, it means there are opportunities to make improvements and changes moving forwards, sometimes small changes and sometimes huge changes. If there are major gaps in coverage of the unit, the recommendation may include conducting reassessment of some students.
The other thing to consider when engaging VET consultants to conduct validation is that often they are not subject matter experts, and they do not always know your clients or target audience. This is why it is important that RTOs also engage a subject matter expert (SME) to participate in the validation activity (and is also a requirement of the Standards to have a vocationally competent and current person).
A process I find useful to follow, and time effective, is for the VET consultant to conduct an initial validation and be thorough and meticulous in noting what they perceive to be gaps in evidence (essentially, be totally black and white in what is in the unit and what is in the assessment). This is then followed up by a session with the SME to discuss each item that has been identified. This allows the SME to clarify where something may be there but using different terminology to the unit itself, or why a task needs to be contextualised in a particular way for that client group or industry. I see this discussion between the consultant and the SME as the link between RTO-land and industry, and ensures we aren’t too heavily focussed on the words in the unit, but achieve a balance in ensuring the tools are also meeting current industry requirements. A validation conducted by a VET consultant that misses this step is a validation that is only half-done, unless they are also a SME in that area and can make the inferences needed.
When conducting this final step with the SME, I also find it useful to include the people who developed the assessment, or who are conducting assessment. Not to influence the recommendations of validation, but to be able to put those recommendations in place. The final phrase of the definition of validation is to act on recommendations, and it isn’t the person conducting validation who needs to act on recommendations, it is the people who are developing and conducting assessments, so it is imperative they are part of the conversation rather than trying to interpret a report after the event.
Ensuring people understand the purpose and process of validation makes it less daunting and they are more comfortable with idea that it is all about improvement rather than their work being judged.
