Concluding Comments

Each task you looked at enables us to learn something about a different aspect of people's literacy and numeracy: reading simple, familiar words; writing simple, everyday words; writing a short sentence about a familiar topic; reading instructions; comprehending everyday prose; handling a simulated computational problem; or extracting information from a document. However, no single task covers all aspects of literacy.

Some important differences between tasks are the nature of the cognitive skills demanded, and the realism of the task. For example, Task 1 involves recognition and decoding of single words with different degrees of familiarity. In contrast, in Task 5 (about the swimmer) a person has to interact with a complex, natural text in a specific cultural context. Both Task 1 and Task 5 involve reading, but of a very different nature. The kinds of inferences we can draw from a "correct" performance on each task (regarding a person's "literacy") are quite different.

Another difference is the logistics of administering each task, recording the response, and scoring it. Some tasks are simple and the answer is either correct or incorrect (Task 1 or 2), so it is easy and cheap to use them, even on a large scale. Yet, they may not capture much of what we mean by literacy. Other tasks (e.g., Task 3, writing a sentence about what you do in the morning) require the creation of scoring rules that describe different levels of performance. Clearly, what will count as "good" performance in one country may be considered "minimal" in another.

When answers cannot be simply scored as "correct" or "incorrect," more training for testers and scorers is needed, and the assessment will cost more. Yet, tasks that lead to answers with different levels of quality (such as Task 3) or that present realistic texts on different levels of difficulty (such as the many tasks used in the International Adult Literacy Survey, of which Tasks 5 and 6 are examples) are more in agreement with a view of literacy as a relative and dynamic collection of skills that form a continuum, rather than using a simple "literate" and "illiterate" dichotomy.

Ideally, we want to measure how well each citizen in a country performs on a reliable and valid test of literacy. This involves three challenges:

1. A comprehensive set of tasks (test items) has to be developed to cover all aspects of adult literacy and examine people's ability to perform a range of functional tasks that make sense within a specific culture. Many choices are called for in this process: which definition of literacy to use, whether to use functional or context-free items, and so forth.
   
   
2. Because of all the cost, time, and training issues mentioned above, comprehensive, direct assessments of literacy cannot normally be given to all citizens, even in "rich" countries. So, a (large) sample of citizens has to be tested, and the results generalized to the whole population. This raises new issues that have to be considered when interpreting results, such as how well this sample represents the population.
   
   
3. Results from the test have to be comparable across sub-populations. This becomes complicated in countries that have multiple official languages or spoken dialects. Comparability across countries is also a challenge, as tests have to be accurately translated and have comparable difficulty levels, and still maintain the same level of realism or familiarity within each culture (think of translating Task 5 and using it in different countries).

For all these reasons, many countries have been forced to use indirect measures of literacy, which are far less valid, but have some logistical advantages.

Now learn about Indirect Assessment.

Back to Assessment Issues or move on to The Limitations of Literacy Statistics.