The PVC Education Office is calling for expressions of interest (EOI) to participate in TELT pilots to evaluate the use of selected tools for digital assessment. We welcome those new to digital assessment and those already conducting digital assessments (we are keen to collaborate!). The current pilots include:
- Safe Exam Browser for Moodle: this software integrates with Moodle (specifically Quiz) to lock down the computer thereby restricting access to authorised resources during an assessment. This includes the specified Moodle quiz plus any 'whitelisted' software applications and internet sites selected for the particular assessment. SEB works with all Moodle quiz question types including STACK used in STEM disciplines. Separate pilots are planned with a given pilot generally focusing on one of three contexts: a) SEB in on-campus labs, b) SEB in exam halls and c) SEB on BYO laptops for in-class use, although a mix of these may be desirable in a given course pilot.
- Digital Inking in Moodle: This will uses a) Poodll Quiz question type and b) 'Poodll anywhere' TinyMCE simple whiteboard tool. The digital inking capability for Moodle quiz provides the ability for students to respond within a Moodle quiz using free hand drawing for diagrams, Mathematical formulae, or to annotate an image provided by the teacher. While the simple digital inking tool in the TinyMCE editor allows digitally inked responses in forums, wiki, blogs and assignment form based submissions. Note: The digital inking tools require a digital pen input device for each student in order to be effective (e.g. USB graphics tablet with pen or touchscreen laptop-tablet hybrid with pen. The provision of these devices is a matter to be arranged on a case by case basis. It must be noted that students who cannot afford to purchase such a device may apply to the university equity access office and the cost may fall to faculties or schools to fund.
- Remote online invigilator services: this service integrates with Moodle. The service provides either live human invigilator to observe and record an assessment session (exam) where the student is located off-campus. This includes web cam and screen recording. Students will be required to run through a series of pre-session steps including registration with the external invigilator service provider via Moodle, conduct computer and internet speed checks, book the exam session at least 48 hours prior to the event and agree to the recording of the session. Initial pilots are solely being conducted with 'UNSW Online' programmes however expressions of interest are being collected to evaluate possible extension of the service. Specifically, this pilot does not include the broader courses offered by the UNSW Business School.
The key purpose of these pilots is to test the utility of these tools at UNSW and to obtain feedback on the staff and student experience. These pilots will be conducted under controlled conditions to avoid any adverse impact on student assessment.
To get involved:
- Please read the detailed information below, then,
- Complete the expression of interest form
We will discuss with you the best way to proceed according to your context and experience with the selected digital assessment tool/approach.
- In general we plan to run pilots as either:
- Proof of concept done as a mock assessments or formative assessment i.e. the assessment cannot contribute any mark, irrespective of weighting, to the course result. We recommend that those new to digital assessment try a mock assessment first before moving on to undertaking a real assessment.
- Low stakes pilot assessments (real assessments) i.e. the assessment will contribute to student's mark in the course. We recommend this for those already conducting digital assessment or those who have already run a proof-of-concept.
- The types of computer devices may include:
- Desktop devices in existing computer laboratories; this will restrict participation to those with access to computer laboratories and appropriate local IT support.
- Students' own devices; while this will allow greater flexibility and broader participation, it will also entail the additional local responsibility of supporting students' devices. The PVCE Office is unable to provide backup devices or desktop support.
- The types of assessment items (dependant on the tool chosen):
- Selected response items e.g. MCQs, T/F Qs, graphical interactions.
- Constructed responses involving short text answers or simple numerical responses.
- Constructed responses involving long text answers (essays).
- Digitally inked responses.
- Subject to local support availability, the preferred number of students per setting is less than 100.
Staff participating in the pilots will be responsible for:
- Writing the assessment items in the selected platform; basic support will be provided to staff in the use of these platforms.
- Implementing the preferred tool to deliver the assessment; the School’s professional and technical staff will be required to implement the assessment; basic support will be provided to staff in the use of these tools.
- Recruiting students to participate in the pilot; student participation may be part of the usual course activities or may be a separate activity. If the pilot is conducted as a course activity, students who do not wish to participate or who are unable to participate must be offered the same assessment under usual conditions (e.g. an option for pen-on-paper).
- Providing technical support to students using their own devices prior to and during the assessment; local IT staff will be needed for this purpose. The PVCE Office will provide generic resources to inform students.
- Locating the physical space for the pilot; dedicated physical spaces for the pilot will not be available; the pilots will need to be conducted in available teaching spaces.
- Providing invigilation, if required.
- Encouraging students to complete evaluation forms and surveys.
- Completing staff evaluation forms after the pilot.
- The status of ethics protocol approval will be discussed as part of our preparatory discussions. As of August 2019 an ethics protocol is in place with a defined set of data collection tools. Individual academics may want to be added to the protocol and/or suggest changes to the data collection tools. With that in mind the time line for changes to ethics protocols needs to be kept in mind.
- Provide support to staff on the use of tools and their associated integrations with TELT.
- Provide information to staff for supporting students and provide an escalation point on difficult problems.
- Manage application capacity and service availability with the vendors.
- Conduct evaluation, including providing evaluation forms, and report back to participants.
Pilots with Moodle: Safe Exam Browser or Digital Ink Poodll
a) Safe Exam Browser (SEB) has been selected as the preferred add-on application for securing devices using Moodle digital assessment tools. SEB can lock down the device so that the student cannot access unauthorised resources during the assessment. SEB is integrated with Moodle. The SEB application must be installed on the computer being used for the assessment (i.e. in a lab or on the student's device prior to the assessment). Additional restrictions may need to be established - such as to limit Internet access in on-campus labs.
Further information about SEB is available here https://safeexambrowser.org/about_overview_en.html
b) Digital inking using Poodll has been implemented and is now available for use in UNSW Moodle - however this is still in pilot phase and as such limited support is available at this time. UNSW has only implemented the 'drawing tools from the Poodll suite. With that in mind information about Poodll anywhere (only the simple whiteboard tool) https://moodle.org/plugins/tinymce_poodll and the Poodll recording question type (only the 'picture' function) https://moodle.org/plugins/qtype_poodllrecording Note: Digital Inking can be combined with SEB for use in summative assessment.
- The pilot assessment must be delivered using Moodle assessment tools. If SEB is being used then this must be using Quiz. This is to allow SEB to be configured.
- Students should be informed about the purpose and nature of SEB; information will be provided by the PVCE Office for this purpose.
- SEB can be used with Windows, Mac and iOS devices. However full computers or laptops are preferred.
- SEB can be configured to allow access to specified resources; staff wishing to use this function should seek advice from the PVCE Office and their local IT support staff.
- Use of Digital Inking tools require the use of digital pen input devices. Examples include commodity USB graphic pen tablet (approx $60) or touchscreen laptop with pen input device. Schools and faculties may be called upon to fund the purchase of these devices for use by students.
Pilots with remote invigilator services
Two service providers are available for pilot use
a) Examity has been selected as the preferred for real-time live human invigilated remote assessments. Examity is only supported for use at UNSW if used via the Moodle LTI connector. Separate, direct use will not be supported.
Further information about Examity is available here https://examity.com/
b) RP-Now (by PSI) currently provides a record and post review option for remote online assessment. Currently use at UNSW is restricted to pilots run as a "proof of concept" where no stakes can be attached to the activity. Negotiation is continuing to permit the use of for-stakes assessment pilots. RP-Now is used by the Plus Alliance partner Arizona State University. Note: other PSI products are not available for use at UNSW at this time.
Further information about RP-Now is available here https://www.psionline.com/en-gb/platforms/rpnow/
- The pilot assessment must be delivered via Moodle LTI connectors only - direct use is not supported.
- Students should be informed about the purpose and nature of the remote invigilator service and tools; information will be provided by the PVCE Office for this purpose.
- The remote invigilator services require the use of Windows and Mac laptops. Mobile or tablet devices are not supported in pilots.
- The remote invigilator services and individual assessments must be configured per exam; staff wishing to use this function should seek advice from the PVCE Office and their local IT support staff.
- In participating in pilots of remote invigilator services mean that timelines in the run up to the pilot assessment must be met by staff involved in the pilot, including the specification and submission of assessment material/exam questions well in advance of the date of the assessment.
For further information on the Digital Assessment Project please contact Karsten Sommer [email protected]