The E-Learning Development Team (ELDT) has established a protocol for delivering online closed examinations using a dedicated instance of the University’s centrally supported virtual learning environment Yorkshare – VLE Exam – for examinations only. VLE Exam is most appropriate for the delivery of examinations, comprising mainly multiple-choice style and short answer questions, with up to 20 different question types supported. VLE Exam can be used to assess cohorts of up to 200 students sitting an examination at the same time.
The protocol of the delivery of online closed examination is broadly the same as that for standard closed examinations, as are the regulations. Key differences between online and standard closed examinations relate to: (i) guidance to students; (ii) preparation of the exam questions and configuration of the test environment; (iii) mitigation of performance risks; (iv) security of examination materials; (v) materials and resources permitted in examinations; (vi) behaviour in examinations.
Guidance to students
Staff coordinating the examination should ensure that their students are familiar with the VLE exam environment prior to undertaking a summative assessment. It is recommended that opportunities for students to familiarise themselves with the examination environment be provided through the provision of a formative assessment of a similar format, incorporating question types and content relevant to the specific module of study. (The provision and delivery of a formative exam should be arranged with the ELDT, following the same protocol for summative exams, as outlined below.)
Students should be clear about the format of the assessment, the number and type of question items to be used, the time limit they will be facing in the actual exam. Special arrangements for students that require additional requirements should also be addressed prior to the exam taking place (e.g. extra time or a separate work station environment with accessibility controls).
Preparation of the exam questions and configuration of the test environment
Academic staff must ensure that examination questions are prepared so that they are suitable for the examination platform that they will be presented in. For VLE exams, the question-set should be shared with the E-Learning Development Team (ELDT) at least four weeks in advance of the date of the examination, so that the appropriate checks can be performed to ensure that the question items are fully compatible with the examination environment and that there are no usability issues (e.g. problems with the rendering of images or challenges with navigation between question items) which might impact on the running of the examination and adversely affect students in their completion of the exam. Consideration should also be given to accessibility requirements for the examination and any reasonable adjustments for students with special requirements should be made in advance of the examination. The ELDT should be informed if students require additional time to complete the exam, which will need to be factored into arrangements on the day of the exam, regarding the cut-off point at which the VLE Exam test site will be made unavailable to staff and students.
Mitigation of performance risks
It is of the utmost importance that the platform used to deliver online examinations is technically reliable and robust and that the examination environment (PC classroom) is correctly set up for the delivery of a computer-based examination. To ensure that adequate preparations are in place, It is the responsibility of the staff coordinating the examination to:
- Book the PC classrooms that will be used for the examination, ensuring that rooms are also booked an hour before and after the examination duration, so that IT Services staff have control of the room to prepare it for the examination and then return it to its original state;
- Notify IT services of the date of the examination and the PC classrooms that will be used, requesting that the classrooms be prepared so that individual work stations are locked down and with browser controls presented to students in examination mode with restricted access to the test environment.
These tasks should be done at least eight weeks prior to the date of the examination.
IT Services will also need to ensure that no major maintenance work on the IT network is scheduled to run during the examination, which might affect the performance of the examination environment and connectivity between individual work stations and the test environment. The ELDT may also need to apply updates to the VLE Exam environment in the weeks before the examination, as well as run a range of stress tests and checks to the hardware and software that will be used.
Whilst these measures should address most performance issues and ensure that the exam runs smoothly, it should be recognised that no computer-based system is 100% reliable and technical failure either at the network level or with the examination software may still occur. An individual computer or hardware associated with an individual computer may fail, and in the worst case the network may go down. Spare keyboards should therefore be provided in each examination room – as a rule of thumb 10% of the machines in each examination room should be reserved for contingency use – to allow candidates to be moved to another machine if an individual computer fails. Back-up paper copies of the examination should also be provided by paper setters via the Examinations Office in case of widespread network or electrical failure.
Security of examination materials
The security of examination materials is of the utmost importance. Examinations are therefore delivered using a separate dedicated version of Yorkshare (VLE Exam). VLE Exam can only be accessed from university supported machines on campus. The test site which is used to deliver a specific examination is only made available to students for the one/two hour duration of the examination. Outside that time, the test site can only be accessed by tutors involved in setting and marking the examination. Students cannot access the examination questions until they have entered the unique password for the examination. This unique password, which is only known to the tutors setting the examination and the ELDT, is only released to students immediately before the start of the examination.
In the week leading up to the examination, the examination coordinator should ensure that a spreadsheet of students who will be taking the exam is shared with the ELDT, so that the ELDT can enrol them on the test site within the VLE Exam environment. Student accounts are created by the ELDT, based on a list of usernames and Exam IDs provided by the Department
The ELDT will ensure that the test site remains unavailable until the exact date and time of the examination, when it will be released – with invigilators then guiding students through the log in process to the VLE Exam environment and providing students with the correct password to access the test.
Immediately after the examination has been completed and all submissions have been received and verified by the ELDT’s VLE Application Manager, the examination site will be made unavailable to all users. Should the exam include open answer question types, which require manual marking, a copy of the exam will be prepared for the marking and moderation activities with Exam IDs removed and then shared with the exam coordinator. Otherwise marking is automated and the VLE Application Manager will prepare a spreadsheet of marks which will then be shared with the exam coordinator.
Materials and resources permitted in examinations
Online delivery of closed examinations opens up the possibility of students accessing other materials and resources through collaboration, communication and discovery. This includes the use of email and social networks to communicate and collaborate, and the use of search engines such as Google to locate materials. Computers used to deliver online closed examinations are therefore ‘locked-down’ before the start of the examination, that is, access to the internet and non-permitted software is blocked – in fact, normally access is provided to the test site within the VLE Exam environment only.
Behaviour in examinations
Regulations for online closed examinations are the same as those for standard closed examinations with the following exceptions:
- Candidates must arrive 15 minutes before the published start time of the examination
- Candidates are not permitted to leave the examination room before the end of the examination
- Summary checklist of key responsibilities that need to be addressed in the preparation and delivery of a computer-based exam: the Learner Engagement with e-Assessment framework http://tinyurl.com/LEeAP-framework [pdf]
- A video case study of a VLE exam design approach: https://elearningyork.wordpress.com/learning-design-and-development/case-studies/vle-exam/
- Information on the technology set-up that is used to support the VLE Exam service: https://elearningyork.files.wordpress.com/2016/01/durham2016_vle-exam.pptx
- Information on item analysis techniques for reviewing question-items (screen cast): https://elearningyork.wordpress.com/2013/08/13/running-item-analysis-on-a-test
For more information and consultation over examination plans, please contact the E-Learning Development Team at firstname.lastname@example.org