Smart Sparrow Adaptive Tutorials as Summative Assessments

Since 2014, Gary Velan has been deploying Smart Sparrow adaptive tutorials as summative assessments to replace phase 1 Medicine practical examinations in histology, embryology and pathology. For each examination, students are provided with secure access to a unique Smart Sparrow tutorial comprised of drag and drop and dropdown list questions, requiring students to identify regions on high resolution images. These time-limited (30 minutes) invigilated assessments take place on campus in a computer lab.  

The primary advantage for teachers compared with traditional paper-based exams is that all exams are instantly marked, which previously took around 40 hours to complete. Even with the increased time needed to set up these exams, time saved for the teacher is at least 30 hours per exam. The other advantage is the rich in-built analytics which can be used to provide rapid feedback to students. 

Each Adaptive Tutorial is a component of a larger practical examination for Phase 1 medical students. A typical cohort consists of 280 students who are split into four groups (each approximately 70 students). Each group then rotates through the different components of the examination over the course of a morning (3-4 hours).  

Once students reach the computer lab, they each use a secure examination login, which locks down the PC to enable only access to a Moodle page containing link to the exam. Each computer terminal is locked to the Mozilla Firefox browser and the ability to open new windows or tabs is disabled, so students are unable to access any other programs or content throughout the exam. The lab is also invigilated the same way a standard paper-based exam would be.  

As students begin the exam, a timer that has been built into the tutorial starts individually for each student restricting them to 30 minutes. If the timer elapses, they are immediately pushed to the end of the exam and their answers are automatically submitted. Students must complete 6 sections in the exam which each comprise of different sets of questions, but each section is worth equal marks. Students can only go forward in the exam and are only allowed 1 attempt at each question. Students submit their responses on completion of the exam and their marks are synchronised with the Moodle gradebook. 

Pros

• Automatic marking saves extensive time (up to 30 hours per exam); 
• Grades automatically synced via Moodle Gradebook;
• Students enjoy the greater interactivity of the assessment; 
• Digital format saves time and effort involved with manual management of paper exams; 
• Drag and drop questions are accessible even for students with colour blindness – hovering over the star reveals its colour name; 
• Relatively easy to create a copy of the exam from the previous year and change the questions.

Cons

• Took considerable time initially to build the first “pilot” exam and work out logistics; 
• Requires some coordination with central IT to ensure the computer lab is set up to deploy exam via the lock-down browser; 
• Images occasionally might load slowly which may disadvantage some students; 
• Technical difficulties may occur such as a computer failing to power on; 
• User difficulties may occur such as students accidentally skipping questions; 
• Editing via Smart Sparrow’s authoring tool can be difficult for first-time users. 
• The specific Smart Sparrow plugin used to build the questions uses Flash technology, meaning the exam must be deployed on Flash-enabled browsers and devices (iPads for example do not support Flash at all).

• All teaching staff at UNSW can use Smart Sparrow as part of the enterprise license – if you require an account, see this page; 
• If you are new to Smart Sparrow, it might be best to first get familiar with the instructor side of the platform via their knowledge base; 
• If you are experienced using Smart Sparrow, you may use the demo link here to create a copy of this lesson and re-purpose each screen with your own questions; 
• Once your lesson is complete, ensure the logistics of running your exam has been handled including booking an appropriate computer lab, coordinating with central IT for deploying via a locked-down browser, the exam link has been integrated with Moodle and not visible to students and students are given enough information prior to the exam so they know what to expect. 

The following tips have been found to assist but not all may be completely necessary for a successful deployment: 

• Selected response type questions are ideal as they can be automatically marked by the platform; 
• Book your computer lab as far in advance as possible (at least a few months); 
• Ensure you contact central IT and coordinate the deployment with them at least 2-3 weeks beforehand; 
• Have a member of IT onsite to monitor for any technical issues; 
• Give buffer time to the students – for example, for a 30-minute exam, allocate an hour. Students can finish the exam early but once they leave the lab, they cannot come back in. Buffer time helps for students who may face any sort of issue and needs more time; 
• Create a practice exam in the same format, accessed via Moodle, so students will be completely familiar with the process beforehand; 
• Do a “dry-run” of the exam onsite with yourself and a member of IT, simulating exactly what the students will be doing from start to finish; 
• Ensure that there are extra computers available in case of computer crashes; 
• Smart Sparrow does not necessarily have to be used – the same deployment strategy can be applied with other platforms like Moodle Quiz for example. 

Experience the demo version here. 
(Your browser or device must have Flash enabled in order to view the demo - iOS devices are not Flash-compatible)


 

Contact Gary Velan ([email protected]) for further information about his experience.