The Framework itself incorporates continual effectiveness re-assessment of the evaluation processes: how well they identify suitable technology applications and tools for inclusion in the platform, and to what extent they improve technology supported learning and teaching. Thus, the Evaluation is conducted over the entire life-cycle of the technologies, delineated across three stages:
Stage 1: An initial vendor assessment: to determine the suitability, stability and viability of the prospective vendors/open-source solutions.
Stage 2: A pre-pilot evaluation score-sheet: to determine the suitability of technology for full pilot evaluation testing in Stage 3. Stage 2 assesses the features and functionality of the application from educational and technical perspectives.
Stage 3: An on-going full-pilot evaluation: to determine suitability for the TELT Platform, as measured by the educational value of the technology application.
Categories of TELT Applications
To permit relevance to different types of TELT applications (categorised for example as large-scale LMS, medium-scale virtual classroom or collaborative suites, and small-scale special purpose software), evaluation methodologies are to be identified to address the specific requirements of each category.
Categories of TELT Applications that have been identified, include:
- LMS—Learning Management System
- CMC—Computer-Mediated Communication Systems
- PDS—Plagiarism Detection Systems
- CAA—Computer Assisted Assessment Tools
- EMS—Educational Multimedia Software
- LC—Lecture Capture Systems
- Smaller-scale Special Purpose Software
Evaluation Framework—Stage 3
Stage 3 begins an ongoing L&T focused evaluative sub-cycle that aims to facilitate a culture of evidence-based excellence at the faculty and institutional levels to improve UNSW technology supported L&T practices.
Stage 3 determines the suitability of the TELT technology for inclusion in the TELT Platform as a whole. Stage 3 consists of three layers:
- Pedagogy ("L&T Critera" in the following diagram)
- Business
- Technical
The Pedagogy layer is emphasised first and foremost as it is directly informed by the UNSW Graduate Attributes, Learning and Teaching Principles and the 16 Guidelines for Learning and Teaching Practices.
The evaluation criteria specified for this layer are also guided by the teaching approaches and models applied in faculties and schools.
The pedagogical criteria for evaluating a TELT technology are further divided into three sub-layers:
- The educational value and usefulness of the selected application
- The learning effectiveness of the TELT technologies
- The learning environments that use TELT technologies
The Business layer reflects the formalised processes of organisational change at UNSW that are informed first by pedagogical and then by technological factors.
The Technical layer reflects the need to account for the technical requirements, standards and protocols established at UNSW that are informed first by the pedagogical and then the business process criteria.
Generic Usability Evaluation
In conjunction with the overarching concerns over evaluating the actual pedagogical (learning) effectiveness of integrated eLearning environments as assessed through various educational theories and methodologies, it is also important to begin considering usability and usefulness as two fundamental factors that create the foundation for eLearning applications to be both conducive to effective learning and supportive of positive pedagogical strategies, thus having education value. Prior to undertaking a comprehensive assessment of the learning effectiveness of an eLearning application, a preliminary examination should consider the application’s usability, usefulness, and educational value as a function of usability and usefulness within a learning and teaching context.
If the usability of a particular technology application is a hindrance to the users’ learning and teaching processes, and if it is not evaluated for separately, the entire evaluation could present a biased impression of the overall educational value and subsequent learning effectiveness of the application.
On a practical level, the six-stage heuristic life-cycle forms an evidence-based starting point for the creation of tailored and thematically grouped questions revolving around the entire application usability experience, for surveys and other instruments that would physically empower the Evaluation. The questions would assess the most common usability, usefulness and educational value attributes, and typical usability problems, as identified through our extensive evaluation literature review, which builds on the review undertaken in the supporting document Evaluation of the TELT platform: Essential elements and methodologies.
It is the nexus of various ad-hoc usability elements and assessable attributes identified through research that creates the ephemeral, intuitive "look and feel" of user experience and satisfaction that has been so inherently difficult to isolate and evaluate.
The Generic Usability Evaluation relies heavily on educational and technology literature—please find literature references and supporting documentation for each heuristic usability element, in the comprehensive reports available for download.
