Framework for Measuring Functional Literacy" discerns that as compared to the attention given in the World Declaration on Education For All at Jomtein(1990) to the new perspective on basic education including adult literacy, the question of evolving an appropriate framework for measuring achievement in basic literacy and allied areas like life skills, functional literacy, etc., remains a much neglected but critical area. Even m ore problematic, varying and vague definitions of literacy across different countries make comparability difficult, even raising doubts about the wisdom in investing efforts in literacy. In the broadened vision now prevalent, literacy is perceived, as not merely the ability to read and write but the skills to use written information to function in society. This perception of literacy raises two important questions: (i) if literacy is not to be equated with mere ability to read and write, how does one measure it?; and (ii) what are the broad parameters to define the contours of measuring functional literacy? UNESCO/PROAP proposes a Project to address these questions and sets up three broad objectives, viz.,

- evolving a commonly acceptable framework for defining a broadened vision of adult literacy with focus on functionality dimension;

- providing training to literacy personnel to design instruments for measuring literacy, develop scales of literacy performance and other abilities, and also for assessing impact, etc.; and

- building national capacities for monitoring and evaluation so as to enhance programme effectiveness.

The Project strategy has four training cycles for literacy personnel in the planning, management, monitoring and evaluation of literacy programmes.

The Project is proposed to be implemented in three phases, spread over two years. The first -- Preparatory -- phase entails preparation of concept papers explicating research and training components envisaged in the Project.

In the second -- Implementation -- phase, a comprehensive working paper on literacy definitions and measurement practices would be prepared, shared in a national workshop of experts so as to arrive at a framework for defining literacy and formulate draft instruments for assessing literacy status in the country. Field validation studies are envisaged to test the draft instruments' suitability and a final prototype of set of instruments would be developed in a second workshop.

The Project suggests that each of the four activities should be preceded by intensive training courses, aided by regional/intentional experts, in the theory and practice of literacy planning and management, monitoring and evaluation, development of impact indicators and designing of core instruments of measurement, data collection, validation, etc.

The expected outcomes of the Project include:

(i) training of literacy personnel in planning, management, monitoring and evaluation aspects of literacy programmes as well as in designing core instruments of literacy assessment and information on literacy programmes;

(ii) a study on impact of literacy programmes;

(iii) core information(quantitative and qualitative) on literacy programmes including cost-effectiveness and cost benefit analysis; and

(iv) a regional network of planners and experts of literacy programmes.

Project Implementation Modality

The basic responsibility of monitoring the process and quality of research and training will lie with participant countries, through their Steering Committee, while UNESCO/PROAP will do the final evaluation.

UNESCO/PROAP's financial support would be about US $40,000 for the two year duration of the Project, basically for technical and management support while the participant countries will host the training programmes and workshops, besides undertaking pilot and impact studies.

Rationale of India's Participation in UNESCO/PROAP's Project

In the context of PROAP's proposed Project, and as regards India's participation in it, two issues are pertinent: (i) what is the scene in India about the perspective and definition or norm of functional literacy and how to measure it; and (ii) what does the Indian literacy movement stand to gain from an exercise of appraising and sharing its perception and definitions and evolving a broadened definition as well as a framework to measure its attainment? Both these questions involve (i) a study and assessment of definition or norms of functional literacy and the procedures/instruments for measuring the learning attainment; and (ii) training of literacy personnel on the theory and practice of literacy programmes and to the evaluators, on evaluation of learning outcomes.

Perception of literacy in India, given the difference between Census and NLM(National Literacy Mission) definitions, perfectly fits the UNESCO/PROAP's finding of different and divergent definitions. It is also true that the NLM definition of literacy eminently aligns itself to the broadened vision of literacy with focus on functionality that PROAP advocates. At the same time, even within the Indian literacy movement, a commonly accepted framework for measurement of literacy, including the functionality components, and more importantly, its universal adoption without distortion, is still eluding India. Reasons may be many, such as inadequate level of understanding or appreciation of the definitions of literacy and insufficient training of literacy personnel and evaluation agencies about the theory and practice of literacy programmes, and the ability to design core instruments for assessing learning and other outcomes.

This, however, is no reflection on the clarity of definition or norm of literacy, including functionality, as envisaged by NLM or the evaluation framework explicated by Expert Groups set up by NLM from time to time. Yet, despite all the clarity about the broadened vision of literacy, very akin to the new perspective on basic education and adult literacy, there is as yet no study in India either on the comparability or differences of literacy definitions or any effort towards a regionally or internationally comparable framework of assessment of either literacy with functionality or the other outcomes, including impact, etc. Even less is the evidence about training of literacy personnel at regional/international level on the theory and practice of literacy programmes or on the designing of core instruments of measurement, information collection, etc. This is quite aside from the non-comparability of Census and NLM definitions of literacy. Thus, on both counts, India stands to gain by sharing and benefiting from experiences of other countries as this Project would offer.

II

LITERACY: INDIA's PERCEPTION AND EXPERIENCE IN ASSESSMENT

India's perception about literacy varied over time, but prior to the formation of NLM in 1988, it was the product of the thinking at international levels, especially UNESCO and the imperatives of national development as Indian leaders perceived[1]. But with the birth of NLM, the perception of literacy, its definition and norm, as would be evident later, eminently presaged the vision discernible with the World Declaration of Education For All at Jomtein in 1990. This note attempts to provide an appraisal of Indian perception about literacy and its measurement experience in relation to the PROAP's proposed Project on "evolving a framework for measuring functional literacy".

(1) Evaluation of Learning Outcomes in Literacy Campaign

The NLM, launched in 1988, aimed at imparting functional literacy[not mere literacy ability] to 100 million non-literates in the 15-35 age group by 1997. The competencies(cited later in this note on p.5) to be achieved at the end of the basic literacy phase were also specified. However, in the initial phase of the literacy movement marked by the Total Literacy Campaign(TLC) approach, "Total Literacy Declaration" became a sensitive and tickling question and opinions were sharply divided for and against the issue of `declaration'. The NLM set up a Group, headed by Dr. R.H. Dave(whose report came to be known as "The Dave Committee Report")[2], to examine the need and modality of `declaration'(what and how to declare)as well as the procedures for precisely evaluating the learning outcome.

In the early phase of the literacy movement, declaration of `total literacy' status became a ruling passion, something of a status symbol, with TLCs. The methodology of evaluating the learning outcome which the Dave Committee spelt out was necessitated because of the differences and deficiencies found in the procedures adopted by TLC organisers in assessment of literacy attainment.

The Committee opined that sharing and informing the performance of TLCs are desirable and important but, this should not be confused with declaration of total literacy achievement. Declaration should be more in the nature of sharing what has been accomplished against original goals and the shortfalls still needing attention. The attainments, based on systematic and sound evaluation of learning outcomes, should be presented in terms of percentages by ranges such as:

Below 50% X% of enrolled

50-59% Y% of enrolled

60-69% Z% of enrolled

70-79% A% of enrolled

80% and above B% of enrolled

Showing the learning attainment scores in terms of ranges will indicate the pattern as well as the backlog.

The Committee also advocated uniform evaluation of learning outcomes for all categories irrespective of gender, community and regional considerations. Besides functional literacy, many TLCs also set up related objectives, such as UPE, including enrollment, retention and better performance, activising NFE, greater participation in health care programmes of children, utilization of services of development programmes, small family norms, etc. The level of achievement of these objectives also need to be shared.

Rather than presenting the level or extent of achievements in all these, an unqualified statement of `total literacy' as having been achieved in a particular District is bound to evoke doubts and criticism. Even the term `total' is misunderstood. "Total literacy", said the Committee, refers to the particular target age group taken up in the programme and a level of achievement, namely, 80-90%, by the targeted learners. The basis of declaration should be a proper evaluation of learners' achievement, based on a test design and procedures which would precisely measure and describe the learning outcomes. However, a comprehensive statement of the project's outcomes would include the teaching-learning activities and process, participation rates, etc., to convey the results and provide input into policy, planning, remediation and reinforcement of the programme. The Commettee's emphasis on proper evaluation of learning outcome and other impacts was not merely in the case of external evaluation but was equally addressed to internal evaluation, which it said, should be aligned as closely as possible to the external evaluation. Thus, having dealt with the approach and procedures of sharing or conveying TLC outcomes and the criteria of declaration, the Committee dwelt in length on the methodology of evaluating the learning outcome.

Methodology of Evaluation of Learning Outcomes

Laying down the guiding principles, the Committee said that the evaluation scheme has to be: (i) simple, intelligible and non-threatening; (ii) systematic and progressional; and (iii) technically sound. The evaluation process especially the design of the test has to be aligned to the norms of literacy acquisition envisaged by NLM. The broad elements of learning and a `Blue-Print' of test design was provided(Annexure I) to help evaluation teams to develop appropriate capabilities for testing and also to design parallel tests.

The learning outcomes to be tested, as envisaged in NLM's objectives of imparting functional literacy, imply:

• achieving self-reliance in literacy and numeracy,
  
• becoming aware of the causes of their deprivation and moving towards amelioration of their conditions through organisation and participation in the process of development,
  
• acquiring skills to improve the economic status and general well-being,
  
• imbibing the values of national integration, conservation of the environment, women's equality, observance of small family norm, etc.

In the context of UNESCO/PROAP's emphasis on broadened vision of literacy, with focus on functionality, it is pertinent to note how the Dave Committee viewed the feasibility of evaluating the `functionality' and `awareness' components. Even while conceding that `functionality' and `awareness' are important components of `functional literacy' and need to be evaluated, the Committee felt that

in the present stage of programme, it may be enough to evaluate the learning outcomes in 3 Rs, and assume that these will automatically develop if the learners reach NLM literacy norms. Moreover, if the IPCL package is properly used and tests organised[i.e. passages meant for Reading and questions on Comprehension], these would indirectly indicate the level of awareness and functionality attained.

As for literacy learning, the Committee was of the view that no person should be declared literate unless he/she has reached the NLM norms. The test design worked out by the Committee, refers to the competencies expected in Reading, Writing and Numeracy, the manner of testing, scoring, etc. Keeping the total score as 100, 40 points are assigned for Reading, and 30 each for Writing and Arithmetic. In each of the competencies, the minimum score should be 50% of the maximum and for being declared literate, a person should have a minimum of 70% in the aggregate.

The test should be aligned to the competencies enumerated by NLM and the topics covered in the primers should be the basis for testing every competency. This would allow flexibility within the basic framework.

As regards the scores obtained by the learners, description of their performance by ranges was considered desirable such as below 50%; 50-59; 60-69; 70-79; and above 80%. It was considered unfair to categorise learners into `pass' or `fail', `literate' or `illiterate' and `successful' or `unsuccessful'.

The Dave Committee Report on "Evaluation of Learning Outcomes in Literacy Campaign" which also provided a Blue-Print" of Test Construction for measuring literacy achievement remained the guideline for External Evaluation(EE) since 1992.

(2) External Evaluation of TLCs: Approach and Methodology

The absence of a standing guideline on the sample size and design led to, what the Expert Group[3](set up in 1993) on "Evaluation of Literacy Campaign in India"(known as Ghosh Committee) found in some cases of EEs "deficient in terms of coverage and sample design(less than 1% of the target group considered for test), and above all, in the estimation procedure adopted to arrive at district level estimates"(extremely partial coverage of the areas and unrepresentative sample).

The Dave Committee which laid down the testing procedure, besides various other points, did not, however, specify whether the EEs should cover all learners in a TLC or only a part of them. Given the unfeasibility of organising simultaneous EE for the entire target universe, even if a sample test was implicit, neither the sample size nor the sample design and procedures were indicated.

Devious on the unambiguous but cashing on the un-elaborated -- this is how best the distortions crept into EEs of TLCs between 1992-94 could be described. What lent poignancy and urgency to the issue was the scale of distortions found in the evaluation methodology and procedures adpted by EEs. Considering the significance attached to streamlining the methodology of EE, the Expert Group's recommendations may be deemed as a watershed.

The Expert Group found that these "deficiencies" were not only in the area of what the Dave Committee did not specify, but also in respect of literacy norms(so clearly stated the methodology and procedure of literacy measurement), considerably lowering the qualifying scores/standards in declaring a person or district literate. Moreover, most evaluations reports were found to have focussed only on the percentage of success of literacy as per the Dave Committee norms and the social(or other) impacts were generally ignored.

The concern of the Expert Group for an evaluation that is representative, random and adequate in sample design and size must be appreciated: the TLCs, 275 when the Expert Group deliberated (Aug. 1994), and ever growing in number [nearly 450 by Aug. 1997, covering more than 80% of the Districts in India], implemented in districts as the basic unit, often with .2 to .4 million non-literates target size cannot lend itself for external evaluation of all its learners. But at the same time, a number of factors such as the high prestige TLC enjoyed, a high visibility, media attention, etc., led to `targetitis'[4] and the tendency to inflate significantly achievement figures. The pre-occupation with achievement of `total literacy', led to one-sided focus on quantitative aspects, with a pressure to report high percentage of achievements. Therefore, the Group attached "special importance to its recommendations on tightening up the procedures of both external and internal evaluations".

The Expert Group viewed that External Evaluations should provide:

• an objective and reliable assessment of literacy achievement and other social impact;
  
• feedback to local organisers about the present status, its strengths and weaknesses for remedial measures;
  
• academic inputs to improve policy and planing at District, State and Central levels.

The approach should be participatory, involving participants of the programme and non-threatening. EEs must adopt a multi-dimentional research approach, supplementing testing and interviews by secondary data, observations and group discussions.

The sampling design suggested for EE is based on the principle that every learner gets an equal chance of being included in the sample, while also covering the varying social demographic and regional characteristics of the district. In other words, the selected sample should be representative of the target population and one should be able to draw valid conclusions, based on the sample of learners tested, about the characteristics of the entire target population.

Following this principle, 5-10% of the target learners by stratified random sampling is suggested as the size of the sample, besides a sub-stratum within the sample to represent the socially disadvantaged groups, religious minorities and rural-urban variations. Village(ward in urban areas) would be primary unit of sample. All learners in the sampled villages/wards are to be covered under external evaluation. Based on the performance of learners in sampled villages/wards, the success rate at the stratum level, i.e., Panchayat, Block, could be worked out, and for the entire district, by taking weighted average of success rates of different strata. At least 80% of the learners of the sampled villages/wards must be tested including home visits to catch up with absentee learners. External evaluation was considered opportune when 60% of the learners have completed the third primer.

Much like the Dave Committee, the Expert Group also had serious reservations about the reliablibility of internal evaluation and dilated on the need to bring its procedure as close as possible to the EE. The Expert Group was more forthright than the Dave Committee for abandoning the whole business of declaration of total literacy status as it believed that total literacy at 80% level of success of the target number of learners to be nearly impossible.

In the Expert Group's view, EEs are to be done by persons trained in social science research methodology and statisticians and must adopt a multi-dimensional research approach, with testing, interviews, secondary data, observations, group discussions, etc.

The Expert Group's concern was not with the problems of compromises in the rigour and fairness of testing or what all aspects to be measured and with what test design framework etc. On these, the Expert Group went by what the Dave Committee had recommended. Its concern was mainly about the unrepresentative, non-random and inadequate sampling design.

About literacy norm, the Ghosh Committee felt that the level needed should be less than what the Dave Commettee recommended. Regarding awareness and functionality dimensions, it did not suggest either the parameters/elements or the framework for measuring. The Ghosh Committee found many EEs have gone beyond mere literacy achievement testing and tried to capture functionality and impact dimensions, but the most essential minimum must be the measure of literacy attainment.

Two issues remained in the aftermath of the Ghosh Committee report. One is an issue of morality, not professional competence, but uncompromising sincerity in the rigour and fairness in testing learning attainment including functionality and impact of TLCs. The other is professional competence about external evaluation with sound methodology. A sound methodology may remove the element of subjectivism and bias, thereby eliminating compromises and unfairness in evaluation. Therefore, close on the heels of the Ghosh Committee recommendations, submitted in 1995, the NLM launched on (i) a countrywide dissemination of the Expert Group's Report to all the TLCs, and adult education authorities of the State Government; (ii) organised reigonal level workshops with evaluation agencies, TLC organisers and adult education officials of of State Governments; and (iii) fostered attempts to work out evaluation methodology and sampling design based on the Ghosh Committee recommendations[5].

The NLM also constituted a Core Group to update and orient evaluation agencies based on their track record and/or potential for TLC evaluation.

The unambiguous and pervasive concern behind all these efforts to streamline External Evaluation, to align the modality of internal evaluation as close as to the EE as possible, and the introduction of Concurrent Evaluation, mandatory for all TLCs, is to ensure a standardized or uniform evaluation framework and its sincere application in evaluation so that (i) a comparable evaluation methodology is evolved and applied; and (ii) a reliable information about the literacy status is obtained.

The latest in the series of NLM efforts is an attempt to impact evaluation of TLCs[6]. However, there is as yet no prototype set of measurement instruments, as the UNESCO/PROAP Project proposes, which incorporates measurement of literacy and other learning outcomes(such as functionality and awareness) and impact(with indicators on development and improvement in the individual and family well-being of the learners. Such an effort as UNESCO/PROAP Project proposes is crucial as the Indian literacy movement is graduating into the more difficult and complex domain of Post Literacy and Continuing Education phase for which reliable information about the literacy status and achievement as well as the impact, in terms of improvement in the well-being of the lerners will be a vital input in the planning and management of literacy programmes.

III

TOWARDS A COMPARABLE DEFINITION AND MEASUREMENT FRAMEWORK:

INDIA's PROJECT STRATEGIES

Without belittling the efforts to imbue the objectives of NLM and as amplified and refined according to the socio-cultural contexts of each TLC, as also the recommendations of Expert Groups, endorsed by NLM about the approach and methodology of evaluation and training of evaluators, strict adherence to the common framework cannot be said to existing today all over India. Willful deviation by evaluation agencies has no cure, except by not allowing them in future. But assuming that the main problems lay in the inadequacy of sensitization and training on designing evaluation procedures and methodology, as suggested by the Expert Groups, the country's participation in UNESCO/PROAP's Project would be beneficial. There would also be a genuine satisfaction in sharing the perception about literacy and its measurement procedures as sought to be provided by NLM through the various sensitisation/training programmes and as sincerely followed in most cases. Thus, both as a learning and sharing experience, India would be keen to participate in the Project.

Towards this end, India would:

(i) carry out desk studies appraising literacy definitions and approach and methodologies adopted in TLC evaluations;

(ii) design a commonly accepted framework for measurement of literacy, with focus on functionality and impact, field test it through pilot studies and evolve a prototype of set of measurement instruments comparable with and adaptable to the varying cultural contexts in different countries;

(iii) organise sensitization programmes for decision makers and public awareness about the importance of literacy programmes, undertake research studies and take up training of literacy personnel;

(iv) organise technical workshops to consider, finalise, field test, authenticate and prepare prototype of sets of test designs and organise intensive training courses at each stage of project for literacy personnel and evaluation agencies;

(v) strengthen national level institutional arrangement with a National Steering Committee of top officials and experts for a more professional and effective monitoring and evaluation of the programme; and

(vi) develop a network of literacy planners and experts to assess and share Indian experience in the theory and practice of literacy movement as well as the state of the evaluation, and plough back regional/international experiences and advantages to enrich literacy efforts in India.

References

1. S.N.Saraf, Literacy in a Non-Literacy Milieu(IIEP, UNESCO, Paris, 1980).

Tom Steele and Richard Taylor, "Against modernity: Gandhi and adult education", International Journal of Lifelong Education, Vol. 13, No. 1, Jan.-Feb. 1994, pp. 33-42.

-----------., "Indian adult education the post-colonial legacy", International Jounal of Lifelong Education, Vol. 13, No.6, Nov.-Dec.1994, pp.451-64.

Anil Bordia, "An Agenda for Adult Education Research: The South Asian Perspective", International Jounal of University Adult Education, Vol. XXXV, No. 3, 1996, pp.8-20.

A. Mathew, Adult Literacy and Development: Report of a Seminar(India International Centre, New Delhi, 1991).

A. Mathew and C. Krishna Mohan Rao, "Divergent Percpeitons of Literacy Campaigns: Towards a Balanced View", Mainstream, Vol. XXXII, No. 10, Jan. 1994, pp. 15-23.

2. Evaluation of Learning Outcomes in Literacy Campaign(Report of Dave Committee)(National Literacy Mission, DAE, GOI, New Delhi, 1994 edn.).

3. Evaluation of Literacy Campaign in India(Report of Expert Group(National Literacy Mission, DAE, GOI, New Delhi, 1994).

4. Venkatesh B. Athreya and Sheela Rani Chunkat, Literacy and Empowerment(Sage Publications, New Delhi, 1996).

5. Denzil Saldanha, "Literacy Campaigns in Maharashtra and Goa: Issues, Trends and Direction, )Economic and Political Weekly,(Special Articles Vol. XXX, No. 20, May 1995, pp. 1172-96.

6. Mushtaq Ahmed, How to Evaluate Learning Outcomes of Total Literacy Campaigns(National Literacy Mission, DAE, GOI, New Delhi, n.d.).

7. Centre for Media Studies Adult Education & Development: The Impact of Adult Education Programme on Socio-Economic Development of Individuals and Communities: A Study -- INDIA REPORT(New Delhi, 1997).

Contact Info:

Khanam:050997
UNESCO/PROAP Project Proposal