Government Funded Research into Fantastic Phonics-downloadable print books

Government Funded Research into Fantastic Phonics

USAID Worldbank literacy intervention into Liberia

EGRA  Plus: Liberia
Program Evaluation Report

Summary:

In 2010, USAID completed a research trial in Liberia, using Early Grade Reading Assessment (EGRA) Plus, which applies a standardised measurement method to benchmark children in grades 2 and 3.

180 schools were randomly selected, and divided into three different groups; no treatment, light treatment, and full EGRA treatment, using a reading program (Fantastic Phonics) of 60 graded stories, with teacher guides.

The children were benchmarked before commencement, and tested on several literacy criteria. It was found that students in Grade 2 (on average) read close to 18 correct words per minute. In comparison, the United States benchmark for “severe risk” is less than 70 correct words per minute.

After 18 months, the children in the full treatment group showed a 250% improvement, compared to their nil- and light-treatment peers.

Bullet Summary
 

  • Cchildren who received the full program performed nearly two and a half times better (250%) compared to their peers who did not.

They achieved …

  • 1.9 school years jump in phonemic awareness,
  • 1.8 school years jump in familiar word reading,
  • 8.0 years jump in unfamiliar word fluency,
  • 1.9 years jump in oral reading fluency,
  • 2.0 years jump in reading comprehension, and
  • 1.8 years jump in listening comprehension.

Executive Summary

1.    Executive Summary

Building on the success of the Early Grade Reading Assessment (EGRA) as a measurement tool, many countries have begun to show interest in moving away from assessments alone and toward interventions focused on changing teacher pedagogy, and as a result, increasing student reading achievement. Liberia, for example, began an EGRA-based intervention, called EGRA Plus: Liberia, in 2008. The results from the EGRA Plus midterm evaluation showed very promising results on a variety of learning outcomes.[1] This report is an impact evaluation of the EGRA Plus program at project completion, and it presents compelling evidence that a targeted reading intervention focused on improving the quality of reading instruction in primary schools can have a remarkably large impact on student achievement in a relatively limited amount of time.

Program Design

Liberia’s path toward intervention started with a World Bank-funded pilot assessment using EGRA in 2008, which was used as a system-level diagnosis. Based on the pilot results that showed that reading levels of Liberian children are low, the Ministry of Education (MOE) and USAID/Liberia decided to fund a two-year intervention program, EGRA Plus: Liberia, to improve student reading skills by implementing an evidence- based reading instruction program. EGRA Plus: Liberia was designed as a randomized controlled trial. Three groups of 60 schools were randomly selected into full treatment, light treatment, and control groups. These groups were clustered within districts, such that several nearby schools were organized together. The intervention was targeted at grades 2 and 3.

The design was as follows: The control group did not receive any interventions. In the “full” treatment group, reading levels were assessed; teachers were trained on how to continually assess student performance; teachers were provided frequent school-based pedagogic support, resource materials, and books; and, in addition, parents and communities were informed of student performance. In the “light” treatment group, the community was informed about reading achievement using school report cards based on EGRA assessment results or findings and student reading report cards prepared by teachers.

Comparisons at Baseline

Schools in all three groups (control, full treatment, and light treatment) were assessed three times. The baseline measurement took place in November and December 2008,[2] the midterm in May and June 2009,[3] and the final assessment in May and June 2010.[4] Students were assessed on a variety of essential early grade reading tasks, including letter naming fluency, phonemic awareness, familiar word fluency, unfamiliar word fluency, connected-text oral reading fluency, reading comprehension, and listening comprehension. The tests used for the midterm and final assessments were equated to the baseline assessment in order to ensure comparability of data, and the reliability of thetests was calculated.[5]

Steps also were taken to ensure that the treatment groups were comparable. To illustrate, Table 1 shows the scores for each EGRA section at baseline. Note that this table presents combined scores for grade 2 and 3. The column “Comparison to Control” presents the results of t-tests comparing whether the outcome measures for full treatment and light treatment scores were higher or lower than they were for the control schools at the baseline. For full treatment, we found that before the intervention, the full treatment schools had higher average scores on familiar word and unfamiliar word fluency, oral reading fluency, and reading comprehension (at the .10 level). For light treatment, this table shows that light treatment schools outperformed their control school counterparts in oral reading fluency (at .10 level), reading comprehension (at .10 level) and listening comprehension.[6]

Table 1 - EGRA / Liberia / Fantastic Phonics Literaacy Invention

Program Impact

Table 1 also shows the increase in scores over the baseline for each group. Note that these estimates come from a simple tabulation of the data.

  • Letter naming fluency. At the baseline, Liberian children were capable of identifying the names of letters, with the average control child identifying 60.7 letters in a minute. At the baseline, the letter naming scores were good, which suggests that program impacts were not likely to be very large. At the final assessment, students in full treatment schools showed a 59.2% increase in letters read, while light treatment schools increased in letter fluency by 46.0%. Control schools also increased their scores, by 35.9%. This is evidence of the learning effect, since the final assessment was held at the end of the academic year while the baseline was at the beginning of the year. These were larger impacts on letter naming than we expected, and with respect to program impact, the increases for full treatment were 0.52 standard deviations (SD) and 0.21 SD for light treatment.[7]
  • Phonemic awareness. Program impact on phonemic awareness was also large. The combined scores for grades 2 and 3 show that the number of sounds identified increased by 67.4% and 39.3% in full and light treatment schools respectively, compared to 26.4% for control schools. This equates to an effect size of 0.55 SD in full treatment schools and 0.18 SD in light treatment schools. This represented a substantive increase of 2.4 words correct (out of 10) for full intervention schools and 1.4 for light intervention schools.
  • Familiar word fluency. For familiar words, children in full treatment schools increased by 247.8% and light treatment schools by 113.5%. Since control schools increased their skills as well, the effect size was 0.78 SD for full treatment and was statistically insignificant for light treatment. This is because control- school children increased their scores by 121.2%. This represents an increase of 24.9 and 10.5 words per minute.
  • Unfamiliar word fluency. For unfamiliar words, control and light treatment schools had very limited changes in outcomes. Control schools increased by 0.9 words (49.2%), while light treatment schools increased by 1.0 word per minute (42.2%). For full treatment schools the increase was 485.7% (from 2.5 words per minute to 17.3 words per minute). The effect size was a very large 1.23 SD for full treatment and insignificant for light treatment.
  • Oral reading fluency. The impact was also quite large for fluency in oral reading of connected text. Compared against baseline, full treatment children increased the number of words read correctly by 138.2%, light treatment schools increased by 41.3%, and control schools by 39.0%. Substantively, this means that full treatment schools increased their number of words read from 20.8 to 49.6 words per minute, while light treatment increased from 19.8 to 27.9. Compared against the gains for control schools, these effect sizes are positive for full treatment (at 0.80 SD) and insignificant for light treatment. This means that at the final assessment, children in full treatment schools were reading nearly two and a half times as fluently as they were at the baseline.
  • Reading comprehension. Comparing the final and baseline assessment scores in reading comprehension, we find that full treatment schools increased their scores by 130.1% over baseline, while light treatment scores increased by 33.4% and control schools by 32.9%. This means that, at the final assessment, children in full treatment schools scored 33.6 percentage points higher than control-school children scored at the baseline, with students in light treatment schools scoring 8.8 percentage points higher. The program’s effect size was 0.82 SD and the effect was statistically significant for full, but insignificant for light treatment schools. This is more than a doubling of the reading comprehension percentage rates for full treatment children.
  • Listening comprehension. For listening comprehension, the increases for full and light treatment schools were 148.8% and 108.0%, respectively. It should be noted that control schools increased their scores by 112.7%, so only by taking into account the baseline scores can a true program effect be estimated. Substantively, full treatment schools increased by 49.9% and light treatment schools by 37.3% over baseline, an effect size of 0.39 SD and no effect, respectively.

Comparing across EGRA sections, we find that the EGRA Plus full treatment program had moderate impacts on listening comprehension, large impacts on phonemic awareness, letter fluency, familiar word fluency, oral reading fluency, and reading comprehension. We found very large impacts for unfamiliar word fluency, indicating that the EGRA Plus program had particularly large impacts on improving children’s ability to manipulate sounds to make words.

Sex and Grade Differences
On all EGRA sections, grade 3 students scored statistically significantly higher than grade 2 students, with more than 11 additional words read correctly per minute on the oral reading fluency section. This is a measure of standard intergrade improvement, which we note for the sake of comparison: The project impact was much bigger than the standard intergrade improvement. In other words, the project was able to boost children’s learning by much more than one grade. On the other hand, there were no differences between boys’ and girls’ achievement, except for unfamiliar word fluency, where girls outperformed boys. This appears to be largely because the EGRA Plus program had a slightly larger impact for girls than for boys, partly because scores were lower for girls at the baseline. This suggests that more work is necessary so that the program, when it is folded into other efforts in Liberia, increases the skills of boys in the more complex portions of reading, and careful attention must be given to ways to ensure that boys and girls benefit equally from the program.

Overall Program Impact

This report presents the effect sizes from a more sophisticated analysis using differences- in-differences analyses.8 These are presented in Table 1 above, in the effect size column. These analyses show that the full treatment group increased student achievement for every section of the EGRA, often with quite large impacts on student achievement. In fact, the overall EGRA Plus effect size was 0.79 standard deviations, which is enormous in social science. When the program impacts are expressed in terms of grade effects, the full treatment increased letter naming fluency by 1.2 times the effect of being in school for one year. Amazingly, this was the smallest effect size for any of the skills assessed.

The EGRA Plus full treatment effect was the equivalent of 1.9 school years in phonemic awareness, 1.8 school years in familiar word reading, a remarkable 8.0 years in unfamiliar word fluency,[9] 1.9 years in oral reading fluency, 2.0 years in reading comprehension, and 1.8 years in listening comprehension. The light treatment group also increased student achievement in letter fluency and phonemic awareness.

This report shows that full treatment schools dramatically accelerated children’s rates of learning. Our regression estimates show that full treatment children increased their word naming fluency by 2.1 words per minute per month, while the associated rate for control schools was an increase of 0.8 words per minute per month.

The rate, then, in full treatment schools, was 2.6 times as fast. For unfamiliar word fluency, we find that the increase in fluency scores was 12.4 times faster in full treatment schools than in control schools, which suggests that a primary entry point for improving reading outcomes for students was through improved decoding skills. The relationship between full treatment and control schools for oral reading fluency of connected text was 4.1 times faster, and 4.0 times faster for reading comprehension.

This shows that the EGRA Plus program did not simply increase the learning outcomes for children; it dramatically accelerated children’s learning to an extent seldom found in educational or social science research.

  • In summary, given the existing literature on effect sizes in literacy interventions, EGRA Plus: Liberia far exceeded expectations with respect to impact on student achievement, particularly in the full treatment schools. Note that the effects were most often large in full treatment schools, with some moderate effect sizes. The range of effect sizes for full treatment was from 0.39 to 1.23 SD. Oral reading fluency scores at the final assessment were 238% of what they were at the baseline, reading comprehension scores were 230% of what they were at baseline, and decoding fluency scores were 585% what they were at the baseline assessment. These impressive results were found on all of the essential early reading components. Most critically, the EGRA Plus program accelerated the learning of children so much that children learned the equivalent of three years of schooling in one year.

Program Impact Compared with Expectations
When compared against the Performance Management Plan (PMP) of February 2009, the results from the EGRA Plus program are very strong. The PMP noted that the impact over baseline two years later would be a 35% increase for oral reading fluency and reading comprehension in full treatment schools, while light treatment schools would see a 10% increase for those same tasks. For both boys and girls, for both grade 2 and grade 3, the light treatment schools made their target in oral reading fluency. By the same token, for both sexes and both grades, the full treatment schools increased by more than 30%, and for each disaggregated level, the increase was more than 100%. The results were similar for reading comprehension. The increases for both boys and girls in grades 2 and 3 in light treatment schools were more than 10%, and the impacts for all groups in full treatment schools were over 80%.[10]

___________________________________________________________________________________________________

1 Piper, B., & Korda, M. (2009). EGRA Plus: Liberia data analytic report: EGRA Plus: Liberia mid-term assessment. Report prepared under the USAID EdData II project, Task 6, Contract No. EHC-E-06-04-00004-00. Retrieved September 21, 2010, from https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=200

2 Baseline: 176 schools were assessed, including 57 control, 59 full treatment, and 60 light treatment schools, for a total of 2,988 students.

3 Midterm: 175 schools were assessed, including 56 control, 59 full treatment, and 60 light treatment schools, for a total of 2,805 students.

4 Final: 175 schools were assessed, including 58 control, 57 full treatment, and 60 light treatment schools, for a total of 2,688 students.

5 Analysis of the assessment tool showed that it was reliable. The Cronbach’s alpha results for baseline, midterm, and final assessments showed reliability of 0.85 or higher, which is quite good. Cronbach’s alpha is a measure of how well a set of variables measure an underlying construct (in this case, early grade reading skill).

6 Because of these slight differences at baseline, our program impact analyses account for the differences at baseline in all of the models assessed. This is done in two ways, first, by the random selection of schools, and second, by the differences-in-differences identification strategy “removing” the baseline differences statistically.

7 Note that the effect sizes reported here are Cohen’s d from the differences-in-differences analyses presented in sections below. Small effect sizes are from 0 to .40, moderate from .40 to .75, and large higher than .75

8 Differences-in-differences is an identification strategy that attempts to make causal inference about a treatment effect by removing the secular trend using a pre and post treatment-and-control design. It is preferable to use three waves of data, if possible, as this particular data set allows. See Skoufias, E., & Shapiro, J. (2006). The pitfalls of evaluating a school grants program using non-experimental data. Working paper. Washington, DC: World Bank. Retrieved September 30, 2010, from http://www.webmeets.com/files/papers/LACEA- LAMES/2006/390/pec_eval.pdf

9 Note that the comparison is between the effect of moving from grade 2 to grade 3. For unfamiliar words, one must assume that the rate of learning to decode will increase as children get older; that is, the program effect is not linear. That said, children in full treatment schools benefited a significant amount in this section

10 Note that while these increases were quite high, the scores for control schools also increased, and at nearly the same rate as those of light treatment schools. This is why many scholars prefer reporting the impact of a program over the baseline and over control schools, to remove the “secular trend.”

Recent Posts

Credits

Early Grade Reading Assessment

(EGRA) Plus: Liberia

Early Grade Reading Assessment (EGRA) Plus: Liberia

EdData II Task Number 6

Contract Number EHC-E-06-04-00004-00

Strategic Objective 3

July 2009

This publication was produced for review by the United States Agency for International Development. It was prepared by the Liberian Ministry of Education, Liberian Education Trust, and RTI International.

  Prepared for  USAID/Liberia and Ministry of Education, Republic of Liberia with partial funding from the Presidential Initiative for Expanding  Education

Prepared by
RTI International
3040 Cornwallis Road
Post Office Box 12194
Research Triangle Park, NC 27709-2194

Reading program prepared in Collaboration with
Early Reading.com “Fantastic Phonics”.

RTI International is a trade name of Research Triangle Institute.

This publication is made possible by the support of the American People through the United States Agency for International Development (USAID.) The contents of this publication are the sole responsibility of RTI International and do not necessarily reflect the views of USAID or the United States Government.

Acknowledgments

In September 2008, the Liberian Ministry of Education (MOE) organized a week-long workshop during which a draft Scope and Sequence for teaching reading in English for Liberia was started and has now been developed. The MOE representatives and other participants provided the authors with sufficient written material to develop a teacher manual that is custom-made for Liberia. The workshop was funded by the World Bank and facilitated by RTI International and the Liberian Education Trust. The finalization of this teacher manual, the training in the use of the manual, and the overall implementation of the EGRA Plus: Liberia project is made possible by USAID/Liberia, the Presidential Initiative for Expanding Education, and the generous support of the American people.

The EGRA Plus: Liberia Project was requested by the Ministry of Education, and funded by the World Bank between June and September 2008. As of October 2008 and until October 2010, the Project will be funded by USAID/Liberia with partial funding from the Presidential Initiative for Expanding Education.

Early Grade Reading Assessment (EGRA) in Liberia The ability to read and understand a text is the most fundamental skill a child learns. Without literacy there is little chance a child will escape the inter-generational cycle of poverty. Yet, in many countries, students enrolled for as many as four years, are unable to read and understand even simple texts. Psychometric evidence indicates that learning to read both early and at a sufficient rate are essential for learning to read well. Acquiring literacy becomes more difficult as students grow older; children who do not learn to read in the first few grades are more likely to repeat and eventually drop out, or will fall behind others for the rest of their lives, and countries where the population cannot read well will tend to lag behind the more educated countries.

What is EGRA?

Concerned with the state of reading in early grades in developing nations, international agencies decided to fund an early-grade reading assessment (EGRA). Most national and international assessments are paper-and-pencil tests used in grades four and above; they essentially assume students have already master the basics of reading. For the few low-income countries participating in international tests, the median child performs at about the 3rd or 4th percentile of a developed country distribution. From these results it is often difficult to tell whether the most basic skills are absent or present to enable the student to understand the test, or whether the children cannot perform the tasks in the test. EGRA was designed to assess the foundation skills for literacy acquisition in grades 1 through 4 orally, including pre-reading skills such as listening comprehension.

The utility of EGRA (and its ramifications) lies in two areas. First, it highlights reading problems, drawing policy makers’ attention to the issue, and helps teachers track performance. Second, early grade reading is a “leading indicator” for the functioning of a school or school system. If reading is not being taught well in a school or a district, it is a safe bet that other things are not being taught well. Moreover, lack of reading instruction and skill is relatively easy to detect, as opposed to a lack of appropriate instruction and skill in, say, social studies.

EGRA in Liberia

While EGRA started out as a measurement tool, many countries have shown an interest in using it as a springboard to improving reading and teacher training around reading. EGRA Plus: Liberia is a good case. The tool itself will be used to lay a baseline on reading. But the main emphasis of EGRA Plus: Liberia will be to improve student reading skills by implementing evidence-based reading instruction. The tasks and activities that would provide information on causes of poor reading levels similar to the opportunity-to-learn literature will be included. Systematic training, support, and supervision will be provided to teachers, along with toolkits and plenty of reading materials for the students. The project will also include training and collaboration with Ministry of Education staff in areas of early grade assessment, development of skills in early grade reading improvement, and the use of data to drive teaching improvement. The project will work on a pilot basis using a thorough evaluation approach.

EGRA Plus: Liberia began in October 2008 and will end in October 2010. It is implemented by RTI International and Liberian Education Trust, with leadership from the Liberian Ministry of Education. EGRA Plus: Liberia has been funded by the World Bank and USAID/Liberia and the Presidential Initiative for Expanding Education. For further information, contact us at egraplusliberia@gmail.com.

How to Use the EGRA Plus: Liberia Teacher Manual and Other Resources

Teachers in Grades 2 and 3 in participating schools of EGRA Plus: Liberia project will use the following four main sources for teaching reading.

A.        EGRA Plus: Liberia Teacher Manual – Volume 1 + 2

The EGRA Plus: Liberia Teacher Manual – Volume 2 presented here consists of two main parts: (1) overview of Scope and Sequence for teaching reading for a year, and (2) weekly lesson plans with detailed daily sequences and instructions on what to teach that day and how to draw on other resources that have been provided to teachers (e.g., decodable books and supplementary materials).

Finally, the EGRA Plus: Liberia Teacher Manual will include the following teacher resources: pocket charts, letter cards, and flash cards.

B.        Decodable books

The EGRA Plus: Liberia project will provide every child in Grades 2 and 3 in participating schools with a number of decodable books. The books are compiled into three compilations and each compilation provides the weekly schedule for the use of these books. This schedule can be also found in this manual (see “Scope and Sequence”).

C.        Library books

The EGRA Plus: Liberia project will provide Grades 2 and 3 with a sufficient number of books to build small libraries. These books are to be used by students in Grades 2 and 3 for reading at home or during school hours for “independent” reading. Teachers will be provided with ‘reading-at-home’ log of books as well as library logs to keep track of the books’ use.

  1. Phonics Reading Plan Outline Weeks 9-12 Comments Off
  2. Phonics Reading Plan Outline Weeks 5-8 Comments Off
  3. Phonics Reading Plan Outline Weeks 1-4 Comments Off
  4. Phonics Reading Plan Outline Weeks 13-16 Leave a reply
  5. Free Phonics Reading Lesson Plan: Week 10: Day 3 Leave a reply
  6. Free Phonics Reading Lesson Plan: Week 7: Day 3 Leave a reply
  7. Free Phonics Reading Lesson Plan: Week 6: Day 5 Leave a reply
  8. Free Phonics Reading Lesson Plan: Week 6: Day 4 Leave a reply
  9. Free Phonics Reading Lesson Plan: Week 6: Day 3 Leave a reply