"Research is formalized curiosity. It
is poking and prying with a purpose."
Prior to publishing a course, two basic types of testing is recommended:
1) Quality assurance
2) Formative evaluation also know as usability studies
Quality assurance testing is done at the final stages of your project and is designed to catch technical problems and typos.
Formative evaluation helps you find out if you are achieving your goals and objects in the formative stages of your project, instead of waiting until you are have finished the project.
Formative evaluation provides:
The American Dental Association describes formative evaluation as:
Formative evaluation is an important method to assure that the developed software meets requirements on several levels, such as usability, functionality, and instructional effectiveness. Formative evaluation allows developers to assess the program during development. For instance, developers may elect to test an early version of the program with users to identify weaknesses in the user interface or other areas. http://www.ada.org/prof/prac/stands/index.html
An important part of formative evaluation for online courses are usability studies. "In Don't Make Me Think: a common sense approach to Web Usability" Steve Krug defines the importance of user testing:
"If you want a great site, you've got to test. After you've worked on a site for even a few weeks, you can't see it freshly anymore. You know too much. The only way to find out if it really works is to test it.
Testing reminds you that not every one thinks the way you do, knows what you know: uses the Web the way you do."
Ease of Use
Three questions developed by Robert Mager in "Making Instruction Work" help define key items evoked by formative evaluations:
Additional testing of ease of use ensures the interface doesnt get in the way of the learning:
One of the most powerful reasons to conduct formative evaluation is to determine that your message is clear.
In designing your plan for conducting formative evaluation, strive to develop test methods that will answer the following questions:
"The true worth of a researcher lies in pursuing what he did not seek in his experiment as well as what he sought."
Claude Bernard, (18131878) French physiologist
During formative evaluation, be alert to notice the unintended consequences, which can be more powerful than the intended consequences. Unintended consequences are when you had the intention of providing one service or message and users interpret and practice it in ways you didnt think of.
Examples of unintended consequences
1) During the early days of anti-drug films, young people learned new-and-improved ways to actually use drugs by watching movies that depicted behavior that was, ironically, designed to discourage drug use.
2) Under the Americans with Disabilities Act, television broadcasters are required to provide closed-captioned programming to make broadcasts accessible to people who are hearing impaired or deaf. As it turned out the appeal of closed-captions goes far beyond its intended use and is also enjoyed by people learning to read, people learning a second language, and people who want to watch TV but dont want to disturb others with the sound in places like restaurants, bars and bedrooms.
The unintended consequences often have more significance than one might think.
When to Test
Formative evaluation should be conducted at every stage of development to ensure the course is achieving your goals and objectives. The earlier you begin formative evaluation the more likely you'll be to have an effective course in the end. Dr. Gerold Lesser, from the Harvard Graduate School of Education, recommends allocating ten to twenty percent of your project budget to formative evaluation.
What a Test Plan Should Include
Examples of commonly used testing methods:
Designing & Conducting Formative Evaluation
Examples of Formative Evaluation Reports
This reports are much broader in scope than the reports required for this course but provide some examples of why people have conducted formative evaluation and how they write up their results.
A formative evaluation of distance education:
Formative Evaluation of a Multimedia Program (pdf file)
Formative Evaluation for an Innovative Course in Social Work (pdf file)