Day 2 of the Durham Blackboard Users’ Conference offered further opportunities for reflection on challenges to the provision and delivery of learning technology services to academic staff and students. The conference is aimed at Blackboard UK users within the further education and higher education sectors – a diverse community reflecting the eclectic range of tools and services that the company now supports – with the conference organising team inviting presentations from vendors and client institutions alike. Predictably vendors showed the most reluctance to embrace the conference theme, which this year focused on ‘learning from failure’ by sharing candid reflections on the effectiveness of learning technology strategies and services.
Urkund focused on the failings of a rival text matching service rather than their own, with the presentation highlighting the superior search coverage of their databases over competitors and the reliability of their solution. As a new entrant to the UK HE marketplace, there is growing interest in this solution as an instructor-centric plagiarism detection service, yet it is one that fails to embrace academic skills support for student text-matching queries; individual searches always lead to results being shared with the instructor as part of an audit trail. This punitive emphasis on text-matching does not sit well with the current philosophy of UK education, based on trust and collaboration with students as partners – ironically the theme for last year’s Durham conference.
Alan Masson’s Blackboard keynote slot explored the road we have travelled together – sharing client-vendor enhancements and innovations in institutional learning and teaching processes supported through the use of Blackboard products – a noteworthy twist to the conference theme that he had been asked to address! Cited examples ranged from the underwhelming (e.g. use of Blackboard’s community system to provide a virtual space for university committees and dedicated tab for employability information – neither particularly new or path-breaking) to genuinely innovative practices – such as Aston University’s use of Blackboard Collaborate to support virtual doctorates and vivas and Edinburgh’s use of the solution to support virtual open days. When questioned at the end of his session on the failings that the company had learned from over the years, Alan made passing reference to communication issues with clients, with no mention of the usability and accessibility challenges which have dogged product development. With Blackboard Ultra still some way off, the ‘bolt-on’ approach to tools and services and lack of integration of the core tool-set within Blackboard Learn remain challenges to the user experience – reflecting the clunkiness of the design which users frequently allude to in their feedback, despite the undoubted value of individual tools to their learning. We can only hope that Jay Bhatt’s vision of a more coherent and usable architecture (Blackboard’s new vision for e-learning 2014) remains intact and will soon be realised with a consolidated design to the Learn platform, and will not be undermined given the recent changes in leadership within the company.
Institutional representatives were more open to highlight the challenges they have faced in establishing services for staff and students, with useful contributions from Al Holloway (Northampton) on the setbacks he has faced in embedding accessibility standards in online course provision across the institution and engaging academic staff along the way and James Leahy (Regent’s University) on design challenges for VLE course site to enthuse students and get them to make use of online course materials, which he is now addressing through the use of interactive media to support more engaging VLE course sites.
Andy Parkinson and I took a different angle – reflecting on the challenges that we have faced in setting up a high stakes computer-based testing service.
- Establishing high stakes computer-based testing through Blackboard [PowerPoint slides from Richard Walker and Andy Parkinson]
- Abstract: Establishing high stakes computer-based testing through Blackboard [PDF]
This has been based on an iterative design and implementation approach, starting with technical trials with QMP in 2009-10 through to our current service, which is run on Blackboard software as a dedicated service and one that is separate from our learning and teaching VLE platform. Along the way we have encountered a wide range of technical and support challenges, from hard failures (connectivity issues and submission errors) through to soft failures – glitches which have affected the assessment experience on individual work stations but not fundamentally undermined it (e.g. latency issues when navigating between questions). On the spectrum of failure there may be a clear distinction between soft and hard technical issues, but from a student perspective they amount to the same – an unacceptable distraction when their attention should be focused entirely on the assessment that they are being asked to tackle.
We have learned from our experiences that mastering the technology is only one part of the equation in delivering an effective service – with cross-service support issues (managing people and processes) requiring equal if not more attention. In terms of risk management there is a low threshold for error and therefore a greater chance of things going wrong when you are introducing unfamiliar technologies and support processes at the same time. The ‘silence of success’ when everything smoothly depends on stakeholder engagement, well-established support processes, underpinned by proven technologies with clear contingencies in place as back-up.
Through a stakeholder engagement approach, we have addressed the fundamentals of the computer-based testing service to establish a robust service which can now support simultaneous testing to scale in a secure fashion across multiple PC classrooms. From a technical perspective, the use of a separate and dedicated instance of Blackboard software has been key to this success, as we are working with a ‘known’ technology on a day-to-day basis, and one which students, assessment designers and to a lesser extent invigilators are familiar with too. Whilst the flexibility of Blackboard’s assessment engine may not support different pathways in assessment (routing based on the responses a student provides) or offer a rich diversity of question types to assessment designers, it passes the 80:20 test in providing an acceptable solution for the majority of test scenarios and can support high-order testing through the judicious use of short-answer questions. (See our blog post from ALT-C 2015 for a discussion of the scope of the service to support testing for postgraduate Education students.)
Challenges remain, though, in the way that we prepare and support students to take computer-based assessment (a focus for a future talk), as well as in how we ‘normalise’ the academic culture of assessment within higher education institutions so that academics, invigilators and support staff are unfazed by this way of working. Arguably this is a more challenging undertaking than mastering the technology and a more pressing concern for institutions to address. How far are we off students expecting to sit electronic rather than ‘pencil and paper’ tests and seeing this as an entitlement for all assessments that they take at university? A thought to ponder!
Cover image photo credit: ADTeasdale on Flickr. Photo licenced under CC Attribution licence https://flic.kr/p/jwgYBD