Zoe Handley in the Department of Education used a locked-down version of the VLE to deliver a summative examination to postgraduate students on the MA Research Methods in Language Learning and Teaching. She used the VLE’s assessment engine to serve up a combination of multiple choice style questions and open questions, which were presented to students in a randomised order. Students logged in to the VLE exam environment using their Student ID and University of York examination number. This approach enabled open questions to be marked anonymously and combined with the marks the VLE’s assessment engine automatically generated for the multiple-choice questions to provide an overall test score.
Key words: summative assessment, exam, feedback
Aims and Objectives
This case study focuses on the use of the VLE to support an online summative assessment for postgraduate students on the MA Research Methods in Language Learning and Teaching at the University of York. Students were assessed in Week 1 of Spring Term (January 2015) through an online examination of two hours duration, completing a series of short answer multiple choice questions and longer answer open questions.
A cohort of around 160 students is enrolled each year on the Research Methods module. The module was previously assessed through a pen-and-paper examination, made up of a combination of multiple-choice style questions (two-thirds) and open questions (one-third). Multiple-choice questions were used to assess knowledge and understanding of a wide range of methods for carrying out and analysing the data from research in language learning and teaching. Open questions requiring students to write a short paragraph, i.e. around five sentences, were employed to assess students’ higher order thinking skills. For example students might be asked to critique an instrument or coding framework, or evaluate a study design. The module team found that the marking effort was challenging to complete in the required 6-week period, when staff were busy teaching other modules and were keen to explore another approach.
The rationale for the switch to an e-assessment approach was to continue to assess the learning outcomes as before, but at the same time improve the efficiency of the delivery of the examination and consistency of marking for multiple-choice style questions. It was anticipated that this would reduce the marking load and ensure that teaching staff could continue to teach and complete their marking duties over the busy spring term.
Zoe met with members of the E-Learning Development Team and colleagues from IT Services and the Exams Office to discuss the design and delivery of the online examination. A separate version of the University’s VLE was set up to provide a dedicated examination environment, with attention to security and performance issues. An existing bank of questions which had been developed in 2013 in an earlier e-assessment trial – funded by the University’s Rapid Response fund – was updated and then converted for use within the VLE exam environment.
Once the test had been quality checked by the module team, it was set up within the VLE exam. IT Services locked this environment down, so that it would provide access to the VLE’s assessment engine only, with users unable to browse to the web and make use of other applications. Equally, this set up meant that off-site users could not access the exam remotely. The ELDT managed the availability of the test environment, ensuring that it was only made available for approved Exam IDs for a time-limited period. Zoe booked four PC classrooms for the Week 1 exam of Spring Term (January 2014) and informed the Exams Office of the arrangements, including invigilation arrangements. IT Services converted each of these PC classrooms into an examination mode, with log in access to the VLE exam only.
Students were briefed about the VLE exam format and provided access to a video tutorial and set of dummy questions within the VLE which they could use to familiarise themselves with the exam format. They also sat a mid-term mock exam in November 2014, which replicated the exam conditions that they would face in the January exam.
The VLE exam was successfully delivered to students located in four separate PC classrooms across the Heslington West campus in January 2014 and again in January 2015. The marking was completed and results communicated to students within the 6-week time-frame.
The mid-term examination and video tutorial of the exam format provided a useful orientation to students, who felt well prepared for the spring term exam. Inevitably though, some students were undecided about the e-assessment approach over paper-based examinations and would like to have received further practice with the question types and typing under examination conditions. Time management and exam technique were also areas where students would like to receive further support and guidance– specifically in terms of how they navigate their way through the question-set, and draft and complete their answers.
Overall students appreciated the security and reliability of the online examination environment and the scope to revise their responses to open questions with ease. They also welcomed the fact that they no longer needed to worry about presentation issues and poor handwriting negatively influencing the way that their work was marked.
Having struggled to pitch the paper-based version of the examination, previously finding the questions were too easy and then too difficult, staff appreciated the facility within the VLE’s assessment engine to automatically produce an analysis of the difficulty and discrimination level of the individual questions which could be used to inform the design of future versions of the examination.
Transferable lessons learned
It is important to prepare students for an online assessment and give them opportunities to practise and familiarise themselves with the assessment format before the examination, so that the assessment targets the knowledge and understanding that you wish to evaluate rather than technical skills and aptitude for online assessment. This is particularly important when students are encountering e-assessment methods for the first time.
Do ensure that all key stakeholders are in the communications loop on e-assessment plans, including exam invigilators, so that procedures and contingency plans are clearly understood by all parties and in place before an exam is delivered. A cross-services team approach involving the ELDT, IT Services’ Digital Workspace Group and the Exams Office holds the key to a successful and a smooth delivery of an online examination. A clear benefit of this arrangement is that technical and security issues are managed by central services and the academic convenor can focus on the assessment design and the marking.
Time savings can be achieved ‘downstream’, once a question bank has been developed and the assessment design quality checked. There is, however, an initial investment to be made in designing good questions and developing them within the VLE.
The VLE exam approach has also been adopted successfully by the BA Educational Studies and BA English in Education programme team (spring term 2015).
- Link to the presentation on VLE Exam to the Higher York E-Learning Network.
- You can learn more about the topics discussed here in our TEL Handbook:
- Contact the E-Learning Development Team via email at firstname.lastname@example.org to find out more.
- Visit our Development Opportunities page to view upcoming webinars on a range of TEL topics and to access recordings of past ones.
- Follow us on Twitter for TEL-related news and tips.
Case study published: June 2015