PART 5
DEVELOPING REGULATIONS
The Basic Steps and the Plain Language Approach

5. TEST THE DRAFT REGULATION

5.1 Why Test?

An important step in the plain language drafting process is testing. Testing means asking a sample of the people who are likely to be working with the regulation, the regulation's "users", to provide you with feedback on it. However, if you are making only minor amendments to a regulation, it may not be worthwhile to test.

The feedback you get from testing can help you to correct problems with structure, wording and policy which you might not otherwise detect. Testing recognizes that policy experts, legal specialists and drafters are not the ultimate users of a regulation. In fact, their detailed knowledge of the issues and policies involved can prevent them from adequately assessing how clearly the regulation states its purpose and how easily users can work with it.

Testing is neither a technical review of the regulation nor a popularity contest on its policy objectives. The purpose of testing is to identify problems users may have in finding information in and understanding the message of the regulation, so that these problems can be corrected before the regulation is brought into force.

5.2 The Benefits of Testing

Usability testing has become an essential part of computer hardware and software development during the last several years. Companies in this highly competitive market recognize the value of finding out how well a user can work with a piece of equipment or software package before they put it on the market. They realize that the programmers and engineers who create the products are not the users and cannot tell them what they need to know about the product's usability. By testing beforehand, they avoid consumer complaints, product failure and a poor product image. Similarly, testing a regulation with users and then making appropriate changes to correct the problems that they identify before the regulation comes into force can have many positive results. It can:

• reduce the time spent by government employees answering phone calls and questions from people who cannot understand the regulation's requirements;

• increase compliance with the regulation, because people are more likely to obey the law when they understand it;

• decrease the need to amend a regulation because of unintended consequences or overlooked problems; and

• improve the overall quality of the regulation.

5.3 When to Test

Testing should take place when the draft regulation is close to its final form. The project team should be confident that they have completed the policy development and initial writing stages and made all the technical corrections that they can make. They should review the sequence of section numbers, and check for such things as the consistent use of defined words and the logical flow of ideas.

However, there is no point in testing a draft regulation if there is no time or intention to make changes to it. Testing provides valuable insights into how users read and work with the draft, but it is unfair to the test participants, and to the project team, to gather test results when the timetable or administrative realities do not allow these insights to be used.

5.4 Who Should Take Part in a Test

In section 2.1, we discussed the need to consult with users at a preliminary stage, before drafting begins. By reviewing the categories of users identified at that early stage, you can select a representative sample of users who, until the test, have not been involved in any way in the consultation or preparations leading up to the draft regulation. You may also want to show copies of the draft to people who were involved in earlier discussions; while that might be very helpful as a continuation of the consultation process, it would not constitute testing.

Users of regulations are likely to include people with and without legal training. Your test participants should be from both groups.

5.5 What to Test

Testing provides the opportunity to check with users on a variety of matters. Since you cannot test for everything, you need to decide what information is most important to gather. You can test, for instance, whether users can understand certain key sections of the regulation. Or, you may wish to select areas with the greatest impact on health and safety. You can test the users' ability to read the regulation and then to do what it requires. You can test whether they are able to use the table of contents efficiently to find the information that they need. You can also use the testing environment to find out which of two wording options or structures best meets the users' needs.

5.6 How to Test

The type of test you choose will depend on a variety of factors, including: (1) the information you hope to gain from testing; (2) the time available for testing; (3) the resources available; (4) the nature of the regulation; (5) the length of the regulation; and (6) the characteristics of the users.

If your goal is to find out how well users understand the regulation, you could set up a read-aloud or think-aloud test. In a read-aloud test, a participant reads a sentence of the regulation aloud and then explains its meaning in his or her own words. Problems with comprehension become evident, especially when the same problems recur among participants. In a think-aloud test, a participant is asked to read through the regulation aloud and to voice his or her thoughts about it along the way. The participant's comments, for example, "didn't I just read something about that?", "I wonder what that means?", provide insights into difficulties users have understanding the regulation.

Read-aloud and think-aloud tests are useful for identifying problems with wording and structure. Usability tests clarify how efficiently and effectively users can find information and correctly interpret it. For a usability test, participants are usually asked to answer scenario questions which are based on the types of situations that they are likely to encounter. Testers watch participants work with the regulation during the test and keep track of, for instance, how many times the participants re-read a section, how often they refer to the table of contents, and how frustrated they appear to get. In some test situations, testers may sit down with participants one by one and ask them to talk about what they are thinking and doing as they try to answer the questions. In other situations, participants may be asked to note when they began working on a question and when they finished it and to write down their comments.

Usability tests provide information on how well participants are able to find the relevant sections in the regulations and how well they understand the provisions. For example, when participants find the right section but give the wrong answer to a scenario question, it suggests a problem with the regulation's wording. When they cannot find the right section, it can suggest a problem with the structure, the table of contents, the headings or the marginal notes.

During all these tests, testers make an audio and sometimes a video recording. The tapes are played back later to review participant comments and ensure that valuable information is not lost. For instance, a problem which may have seemed insignificant with the first participant may afterwards appear as part of a pattern.

Participants may also be asked to complete a questionnaire on the regulation as part of the test. They might be asked if they thought the regulation was easy to understand, if the table of contents was easy to use, if it was easy to find necessary information and so forth.

All the information collected during testing is analyzed to identify ways in which the regulation could be written more clearly and be better organized for users' needs.

5.7 The Cost of Testing

Testing does not have to involve hundreds of participants or fancy lab equipment: testing with five to eight participants is often sufficient to provide significant information about one language version of a draft regulation. Experienced testers can assist in designing the test protocols and conducting the test. Even the simplest test process can provide invaluable results, at a low cost. The key is to think about what you want to find out and then to listen carefully and with an open mind to what the test participants say.

REFERENCE DOCUMENTS

For more information on testing draft regulations for usability, readability and comprehension, refer to the following:

• A Practical Guide to Usability Testing, Dumas, J.S. and Redish, J.C., Ablex, Norwood, N.J., 1993. This 412-page basic text explains the principles behind usability testing, describes how to plan and prepare a test and explains how to conduct a test and evaluate the results. Its authors are pioneers in the testing field who have experience testing a variety of "products", including government documents.

• Technical Report: Consumer Fireworks Regulations Usability Testing, Schmolka, Vicki, Department of Justice, Ottawa, 1995. This paper documents the federal government's first usability test of a draft regulation and includes the test results and a complete collection of the materials used in the usability testing.

• Consumer Fireworks Regulations: Final Report, Trevethan, Shelley; Gordon, Wendy; and Roy, Marie- Andrée, Department of Justice, Ottawa, 1995. This paper summarizes the steps that were taken to prepare a plain language version of a regulation, including usability testing. It offers an assessment of the process and a copy of the draft regulation that was prepared incorporating the test results.

• Practical Approaches to Usability Testing for Technical Documentation. Velotta, C., ed., Society for Technical Communication, Virginia, 1995. This 100-page booklet covers usability testing basics, with sample checklists and questionnaires. It has a comprehensive, 14-page bibliography of usability testing literature that is an excellent guide to available research materials.

• [ Previous Page 
• | Table of Contents 
• | Next Page ]