by Franco Di Cesare, MD; Giulia Piccinini, MD; Cristiana Di Carlo, MPhil; and Leonardo Di Cesare, MD 

Drs. F. Di Cesare, Di Carlo, and L. Di Cesare are with Leoben Research srl in Rome, Italy. Dr. Piccinini is with Department of Neurosciences, Università Cattolica del Sacro Cuore in Rome, Italy, and UOS Rehabilitation Unit, ASL Rieti in Poggio Mirteto (Rieti), Italy. 

Funding: Leoben Research sponsored and funded the project.

Disclosures: Drs. F. Di Cesare and Di Carlo are co-founders and co-owners of Leoben Research. Dr. L. Di Cesare was a Leoben Research employee at the time of the research. Dr. Piccinini has no conflicts of interest relevant to the content of this article.

Innov Clin Neurosci. 2023;20(7–9):11–17.

Abstract

Objective: The paucity of psychometric instruments is one of the challenges preventing the effective implementation of child cognitive health testing in Sub-Saharan countries. WORDS is a new memory assessment tool originally validated in a Zambian school-aged population.

Design: WORDS involves the standardized administration of a multitrial, verbal, free recall task with acquisition and immediate recall of a structured list of 16 words, followed by a second acquisition and delayed recall after a five-minute interference. A field trial was carried out to evaluate test feasibility of use, reliability, and validity. A total of 448 children, aged 5 to 17 years, were recruited at multiple Zambian clinical sites and schools.

Results: WORDS provides a summary score of recall productivity derived from the sum of items correctly retrieved at immediate and delayed recalls. Recall productivity showed a moderate/good reliability at test-retest with alternate forms (N=53, 95% confidence interval of the intraclass correlation [CI-IC]: 0.73, 0.54–0.85). Recall productivity was found to increase with age. Altered memory functioning, as indicated by lower recall productivity, was associated with poor health status resulting from chronic neurologic or medical disease or prolonged exposure to psychosocial stress and deprivation [recall productivity, range: 0–32, healthy mean (standard deviation [SD]): 15.36 (4.7) vs. poor health, mean (SD): 12.62 (4.8); F (1,446)=27.79, p=0.000, d=0.58].

Conclusion: WORDS addresses the need for new psychometric tools to evaluate memory in a school-aged Zambian population. WORDS has the potential for clinical utility to evaluate cognitive development, as well as the effect of different health conditions on memory. Study results warrant research to further validate its use in Sub-Saharan clinical and epidemiological research settings.

Keywords: Pediatric cognitive assessment, memory, test development, Sub-Saharan countries, clinical trials, epidemiology, cognitive development

The exposure to multiple risk factors of altered cognitive functioning (e.g., postinfective cognitive complications of widespread infective diseases, malnutrition, psychosocial deprivation) makes children in Sub-Saharan countries a particularly vulnerable health population.1,2 Cognitive impairments may deter the child’s psychological development and pose a threat to full emotional and psychosocial functioning in life. Early detection and identification of impairment can inform on children at increased risk for altered psychological development or other health conditions and enable a more efficient evaluation of therapeutic intervention. However, the paucity of valid diagnostic tools is a barrier preventing the efficient assessment of cognitive health in clinical research and healthcare settings.3,4

Zambia is an illustrative example of a Sub-Saharan country meeting the challenge of enabling efficient pediatric cognitive health assessment. Zambia mostly relies on a handful of diagnostic tools initially developed for use in North American and European countries that were imported to Zambia.4–6 However, adopting instruments designed for use and standardized in a distinct socioeconomic and cultural context poses the risk of poor cross-cultural validity and fairness, thus potentially misrepresenting the cognitive abilities of children.4,7–10 Furthermore Zambia presents a rich composition of spoken languages, most of them derived from Bantu, as well as a widespread use of English as lingua franca.

A development strategy for a cognitive test for use in Sub-Saharan countries should be based on the original design of a culturally appropriate diagnostic instrument and its validation in the specific context of use.4,5,11 Cognitive health screening for early detection of impairment and monitoring of therapeutic intervention can benefit from easy-to-use and cost-efficient memory assessment tools. To the best of our knowledge, no psychometric test originally designed to evaluate memory in Zambian children is currently available. We report the initial development of WORDS, a new verbal memory test designed to address the need for new psychometric tools to evaluate cognitive health in the Zambian school-aged population.

Materials and Methods

WORDS consists of a standardized administration of a multitrial, verbal, free recall task. A multiple-item learning and recall paradigm was chosen from among other possible memory tasks as a relevant and familiar behavioral experience to school-attending children. The basic memory task involves an auditory-verbal acquisition of a supraspan item-list, followed by free verbal recall. The sequence of administration includes five steps. First, a word list is presented via verbal slow reading to the child, with a one-second pause between two consecutive words (1st acquisition). Second, immediately after the presentation of the word list, a verbal free recall starts, with a 60-second time limit to complete the recall (immediate recall). Third, the word list is presented for a second time via verbal slow reading, with a one-second pause between the presentation of two consecutive words (2nd acquisition). Fourth, at the end of the reading of the word list, a five-minute interference by a symbol cancellation task starts (retention interval). Fifth, immediately after the interference task, the delayed verbal free recall starts, with a 60-second time limit (delayed recall). The child is not made aware of the time limit for recall.

The memory task is designed to require effortful cognitive processing during acquisition and retrieval of information. Furthermore, the sequence of administration is designed so that acquisition and immediate recall sessions would enhance subsequent performance at delayed recall after a short retention interval with interference.12–17

Words in the list are substantive nouns with concrete meaning and interitem semantic relatedness distributed across four mutually exclusive taxonomic categories: high imagery, high frequency of use, bisyllabic, or trisyllabic; there are four words for each taxonomic group. The order of item presentation is pseudorandomized to disallow the sequential presentation of two words belonging to the same taxonomic category or with similar phonetics. The word list was built to facilitate recall organization by grouping items that belong to the same taxonomic category and recall these items together (i.e., category clustering).18 Table 1 shows word list forms A and B in Bemba, Nyanja, and English, which were used in the study.

The Zambia 2010 census19 recorded that over 30 major languages were spoken in Zambia. The most widely spoken languages are Bemba (35% of the population), Nyanja (20%), Tonga (12%), and Lozi (6%). English is spoken by about two percent of the population as a first language, but it is the most frequently used second language, as it is taught at school. An urban variety of Nyanja, Chewa, is the lingua franca of the capital, Lusaka, and is used for communication between speakers of different languages. Nyanja is also widely spoken as a second language throughout Zambia. Bemba is the language most spoken in the Copperbelt Province. Bemba and Nyanja were chosen for use in this study, as the logistics of the study limited recruitment to children residing in the Lusaka Metropolitan Area and Copperbelt Province.

Test instructions and word lists were prepared in collaboration with Zambian research team members who spoke the three languages. Investigators underwent a training program on WORDS administration and scoring. Testing procedure required children to choose their preferred language of testing, and the choice was documented in the specific section of the trial case report form. For each recall session (i.e., immediate and delayed), the sequential order of correct item recall and the sum of items correctly retrieved were noted.

WORDS was designed to provide measures of verbal recall productivity and organization. Recall productivity is the ability to correctly recall a list item. Immediate recall (sum, range: 0–16) and delayed recall (sum, range: 0–16) scores were derived as indicators of productivity of verbal recall. Two more memory scores were also calculated: item gain, defined as the difference of delayed recall and immediate recall (range: 0–16), and recall productivity, defined as the sum of immediate recall and delayed recall (range: 0–32). Nevertheless, recall productivity measures do not inform on the underlying cognitive processes that are implicated in information retrieval and verbal recall. To obtain more specific and articulated information on a child’s memory functioning, measures of recall productivity should be accompanied by quantitative indicators of recall organization, which can provide an indication of how information is acquired and encoded, how children initiate recall and make transitions among successively recalled items and, in synthesis, how encoding and organization is functional to the efficiency of the information retrieval process. Semantic clustering is one of the cognitive processes involved in the encoding, organization, and retrieval of information from memory. Semantic clustering refers to an organization of recall by semantic item groups,18 which primarily relies on pre-existing semantic associations/relatedness of items in the list to be memorized. When asked to recall a list of randomly distributed words that have an implicit semantic structure, subjects show a tendency to cluster semantically related words together and override their order of presentation.18 Even without any explicit information on the possibility of grouping items, subjects automatically use and benefit from implicit semantic structure. In WORDS, free recall organization is expected to depend on interitem list category semantic clustering. The extent of category semantic clustering in free recall is summarized as the adjusted ratio of clustering (ARC), a measure not dependent on correct recalls.20 ARC scores range from -1.0 to 1.0, with 1.0 indicating perfect organization, 0.0 indicating chance level organization, and negative scores indicating below chance-level organization.20

Study design. A cross-sectional field trial, complemented by an ancillary test-retest reliability study on a selected subgroup of participants of the field trial, was designed and conducted. Aims of the trial were to assess the feasibility of WORDS utilization and the reliability of recall productivity and organization measures and their validity to evaluate cognitive development in a Sub-Saharan school-aged population.

The study protocol obtained approval from the National Health Research Authority and the University of Zambia Biomedical Research Ethics Committee (reference number 00808-165, approval on February 27, 2018) and was conducted in accordance with the International Guidelines for Good Clinical Practice and the Declaration of Helsinki,21 as well as Zambian legal and regulatory requirements. Parents or legal guardians of the participating children signed written informed consent. Assent was obtained from children aged 12 years or older. Informed consent was obtained in the presence of an independent witness for illiterate parents or guardians.

Participants. Children aged 5 to 17 years (including limits) with access to formal education, experience and familiarity with cognitive tasks similar to the one required by WORDS (i.e., list repetition), and the ability to communicate effectively with the examiner were eligible for participation.

Exclusion criteria included presence of a serious health condition currently requiring inpatient hospitalization, significant speech disability, hearing impairments, significant sensory or motor disabilities, current acute medical or mental health condition, or any significant chronic conditions preventing cognitive testing; inability to understand test instructions; or being uncooperative during the assessments.

Participants and sites for test administration represented a broad sample of children and settings for which WORDS use is intended. The study population was recruited from multiple clinical and school sites in two diverse regions of Zambia, the Lusaka Metropolitan Area and Copperbelt Province. Schools were purposely selected to provide a heterogeneous sample representative of the wide socioeconomic, cultural, and linguistic diversity of the pediatric population across regions.

Clinical sites were selected to ensure that a heterogeneous sample representative of the diversity of medical and psychosocial conditions affecting child health and cognitive development was drawn. Clinical sites were the outpatient clinics at the University Teaching Hospital Department of Pediatrics and Child Health in Lusaka and tertiary outpatient clinics in the Copperbelt Province, Ndola area. University Teaching Hospital is the highest referral hospital in Zambia delivering specialized care in pediatrics and receives children from all over the country. Enrollment started in August 2018 and ended in October 2019.

Data collection methods and analysis. Sociodemographic and medical history information were collected, and WORDS was administered to all participants. A subgroup of participants also underwent clinical and neurological examinations, as well as electroencephalographic evaluation.

STATA software version 15.122 was used for statistical analyses. ARC was calculated using a specifically designed software.23 Descriptive statistics and nonparametric and parametric tests were applied. Reliability and validity of WORDS measures were assessed using methods and techniques for analysis of correlation and variance based on data extracted from the field trial clinical database.

Results

Table 2 summarizes sample characteristics. The sample included children in apparent good health condition at the time of assessment, although they may have had a chronic health condition that did not represent an important risk or causative factor for cognitive impairment. The poor health group included children with at least one chronic health condition that is a recognized, important risk or causative factor for cognitive impairment and required therapeutic intervention at the time of study assessment. Diagnostic classification was based on medical review of the children’s available health information, as made available by medical history, clinical and neurological examination, instrumental diagnosis, or reporting from reliable proxies. All participants were regularly attending school. The healthy group showed more favorable indicators of social functioning than the poor health group (Table 2). All participants but one, who was excluded from the analysis, completed the memory task, demonstrating a satisfactory feasibility of use.

Reliability of WORDS measures. Table 3 shows the results of the test-retest study. Reliability as stability of the measure was moderate-to-good for delayed recall and recall productivity.24,25 Immediate recall, ARC scores at immediate and delayed recalls, and the difference between delayed and immediate recalls productivity (item gain) showed unsatisfactory reliability point-estimates. ARC near-zero means and confidence intervals (CIs) indicated a tendency not to use semantic clustering as an information processing mechanism for immediate and delayed recall organization.

Effect of language of testing on recall productivity. Table 4 shows the distribution of WORDS scores by language of testing chosen by the child. No statistically significant differences between test language forms were found.

Effect of repeated list presentation on recall productivity. In healthy participants (n=341), a two-sample t-test showed that delayed recall (mean: 8.42, standard deviation [SD]: 2.9) was higher than immediate recall (mean: 6.94, SD: 2.3) indicating an improvement in recall productivity. The difference of 1.48 was statistically significant [t (682)=7.41, p=0.000, effect size d=0.57]. Correlation of delayed recall and immediate recall was large (n=341, r=0.61, p=0.000).26

Effect of sex, age, and other psychosocial indicators on recall productivity. Effect of sex on recall productivity was evaluated by comparing two groups of 155 male and 186 female participants. Immediate recall [1-way ANOVA, male, mean (SD): 6.92 (2.2) vs. female, mean (SD): 6.97 (2.4), at the p<0.05 level for the two conditions, F (1,339)=0.03, p=0.872], delayed recall [1-way ANOVA, male, mean (SD): 8.30(2.7) vs. female, mean (SD): 8.53 (3.0), F (1,339)=0.52, p=0.472], and recall productivity [1-way ANOVA, male, mean (SD): 15.21 (4.4) vs. female, mean (SD): 15.58 (4.9), F (1,339)=0.27, p=0.601] showed no significant differences.

Immediate recall (n=341, r=0.27, p=0.000), delayed recall (n=341, r=0.38, p=0.000), and recall productivity (n=341, r=0.37, p=0.000) were associated with age in months, with a small strength of positive correlation.26

The ability to speak multiple languages, (e.g., Bemba and Nyanja in Lusaka area) was not associated with immediate recall [1-way ANOVA, F (1, 332)=0.22, p=0.604], delayed recall [1-way ANOVA, F (1, 332)=1.34, p=0.248], or recall productivity [1-way ANOVA, F (1,332)=0.23, p=0.634].

Being an English speaker was not associated with immediate recall [1-way ANOVA, F (1,332)=0.31, p=0.751], delayed recall [1-way ANOVA, F (1,332)=1.61, p=0.205], or recall productivity [1-way ANOVA, F (1,332)=0.38, p=0.537].

Having had delayed access to school was not associated with immediate recall [1-way ANOVA, F (1,324)=0.28, p=0.594], delayed recall [1-way ANOVA, F (1,324)=0.51, p=0.477], or recall productivity [1-way ANOVA, F (1,324)=0.49, p=0.483].

Effect of health status on recall productivity. Health status influenced recall productivity, as better health status was associated with higher scores in immediate recall [healthy, mean (SD): 6.94 (2.3) vs. poor health, mean (SD): 5.80 (2.2), at the p<0.05 level for the two conditions, F (1,446)=20.14, p=0.000, d=0.50], delayed recall, [healthy, mean (SD): 8.42 (2.9) vs. poor health, mean (SD): 6.81 (3.0), F (1,446)=24.90, p=0.000, d=0.55], and recall productivity, [healthy, mean (SD): 15.36 (4.7) vs. poor health, mean (SD): 12.62 (4.8), F (1,446)=27.79, p=0.000, d=0.58].

The combined effect of health status and age on recall productivity scores were evaluated by comparing early childhood (5–7 years of age, n=94), late childhood (8–11 years of age, n=140), and adolescence (12–17 years of age, n=214) age groups, with poor health status and healthy subgroups. Table 5 shows summary results. Both health status and age influenced all recall productivity scores, as higher scores were associated with better health status and older age. The percentile rank of recall productivity scores by age group are reported in Table 6.

Discussion

WORDS is a paper-and-pencil verbal memory test with a user-friendly, simplified, fast (10 minutes), and streamlined procedure for administration and scoring by qualified healthcare professionals. It is intended for use to evaluate memory over time and detect memory impairment.

WORDS provides a measure of recall productivity, which combines immediate recall and delayed recall scores. Recall productivity showed a moderate-to-good reliability24, 25 as stability over time of the measure when the purpose of the measurement was to provide an assessment of cognitive status in a sample of the Zambian school-aged population.

Study results highlighted the potential utility of WORDS recall productivity measures in evaluating the effect of health-related risks or protective factors on children’s cognitive functioning and development. Productivity of verbal recall increased with age, showing a progressive improvement of memory functioning from early to late childhood, which then stabilizes during adolescence. Recall productivity measures were affected by health status, with lower scores associated with poor health status (resulting from chronic neurologic or medical disease or prolonged exposure to psychosocial stress and deprivation).

At this point of instrument development, WORDS does not provide measures of organization of recall. Semantic clustering is an important aspect of information organizational processing in memory functioning. The measurement of semantic clustering entails making assumptions about the existence of an internal semantic similarity model and the ability to use it to encode and retrieve verbal information. We applied ARC as a single semantic clustering metric to measure the extent of recall organization. We observed a near-zero ARC mean score in the test-retest study, and this finding should be interpreted with caution. An observed near-zero clustering score does not necessarily indicate a true lack of semantic clustering in recall organization, but instead might evidence a mismatch between a child’s internal similarity model and their ability to use it and the ARC scoring model. For this reason, the observed clustering score remains difficult to interpret precisely. Furthermore, ARC showed inadequate critical psychometric properties (i.e., unsatisfactory reliability).

An important limitation of our research concerns the characteristics of the study sample, which is not representative of the entire Zambian school-aged population due to its limited size and composition. Constraints on the availability of human and economic resources to conduct the field trial had negative impact on the recruitment process (i.e., number of participants and duration of enrollment period), site selection, and geographical distribution. For instance, nearly all of the participants were from an urban area, whereas about two-thirds of the Zambian pediatric population lives in rural areas.21 Furthermore, the study did not include all languages spoken by Zambian children. It should be noted also that all participants had access to formal education and regular attendance at school. Nevertheless, due to occasionally insufficient educational provision, circumstances such as displacement by traumatic life events following the death of a child’s mother or poverty, severe social deprivation or neglect, or chronic health conditions, many school-aged children in Zambia might attend school irregularly or not at all, and this group of children was not represented in the study sample. In this regard, robust evidence shows how exposure to formal schooling is associated with improved performance on cognitive testing.5

Notwithstanding limitations relating to study design (i.e., concerning initial and partial assessments of psychometric properties) and sample characteristics, results enable and inform on the next steps of the WORDS validation process and its potential applications.

First, a strategy for the validation of measures of free recall organization targeting a combination of semantic clustering and semantic similarity metrics might provide a convenient means of quantifying semantic clustering.27 Semantic clustering and similarities metrics should be complemented with other measures traditionally used in experimental settings to investigate modalities of recall organization beyond semantic clustering. Measures of serial position effects, such as primacy, recency, and temporal clustering, are a viable option. The recency and primacy effects refer to the tendency of subjects to show superior recall of items from the ends, and, to a lesser extent, the beginnings of the word list.28,29 Temporal clustering refers to the tendency to successively recall items that occupy neighboring positions in the word list.30 As there is a wealth of metrics available from experimental psychology research, the specific mode of recall organization metrics to be used should pursue a goal of achieving memory measures with superior reliability, validity, and utility in the specific context of use, a multilingual and culturally diverse Sub-Saharan setting.

Second, WORDS can contribute to the characterization of the effects of widespread disease and psychosocial conditions on child cognitive functioning and development. It could be used to inform on the extent to which postinfective neurocognitive complications prevent full development in children, provide a punctual estimation of the negative impact of malnutrition on cognitive development, assess the consequences of situations of early neglect and severe social deprivation, and be an early marker of learning disability and cognitive impairment. New research should address the clinical utility and diagnostic accuracy of WORDS in relation to the clinical management of specific disease conditions (e.g., malaria, epilepsy, human immunodeficiency virus/acquired immunodeficiency syndrome [HIV/AIDS]-related neurocognitive impairment).

Third, WORDS retains the potential for application in hypothesis-driven or exploratory clinical research and epidemiology. WORDS measures should be validated for use in clinical trials for the purpose of diagnostic classification or as an outcome measure to determine clinical benefits, safety risk, and cost-efficiency of intervention programs. Reliability of WORDS when the purpose of the measurement is prediction or monitoring of medical intervention (e.g., evaluation of a therapeutic treatment) should be assessed.

Fourth, new studies should provide supporting evidence for convergent and discriminant validity of interpretation of WORDS. Future evaluations should be conducted to assess the relationship of WORDS memory measures with other constructs (e.g., measures of intelligence, social cognition, selective and sustained attention, visual memory, language, and executive functioning); the use of diagnostic instruments originally developed in Zambia or imported with cross-cultural validation should be prioritized for use in these studies.

Finally, there is a complex situation of language use, multilingualism, and code-switching in Zambia, in which children employ different languages in different social contexts. For example, a child in Lusaka might speak Bemba as their home language and speak Nyanja and English as languages of wider communication at school. The effects of the ability to speak and switch between multiple languages on WORDS measures should be a target for in-depth investigation.

Conclusion

In conclusion, the initial evaluations of feasibility of use and psychometric properties encourages the development of WORDS as a diagnostic tool to evaluate verbal memory in a Sub-Saharan school-aged population. Study results warrant further validation of WORDS in Sub-Saharan clinical and epidemiological research settings.

Acknowledgements

The authors acknowledge the contribution to the successful conduct of the study of all members of the Zambia Pediatric Cognitive Assessment Working-Group: Virginia Chanda, Ornella Ciccone, Prisca Kalele, Kalima Kalima, Nfwama Kawatu, Sylvia Mwanza-Kabaghe, Lisa Nkole, Somwe Somwe, Aaron Phiri, Mercy Sulu, and Owen Tembo.

Study Materials

WORDS administration guidelines and forms can be requested from Dr. Franco Di Cesare at [email protected].

References

  1. Ford ND, Stein AD. Risk factors affecting child cognitive development: a summary of nutrition, environment, and maternal-child interaction indicators for Sub-Saharan Africa. J Dev Orig Health Dis. 2016;7(2):197–217.
  2. Idro R, Carter JA, Fegan G, et al. Risk factors for persisting neurological and cognitive impairments following cerebral malaria. Arch Dis Child. 2006;91(2):142–148.
  3. Stemler ES, Chukwu F, Chart H, et al. Assessing competencies in reading and mathematics in Zambian children. In: Grigorienko E, ed. Multicultural Psychoeducational Assessment. Springer; 2009:159–160.
  4. Matafwali B, Serpell R. Design and validation of assessment tests for young children in Zambia. In: Serpell R, Marfo K, eds. Child Development in Africa: Views from Inside: New Directions for Child and Adolescent Development. Wiley; 2014:77–96.
  5. Mulenga K, Ahonen T, Aro M. Performance of Zambian children on the NEPSY: a pilot study. Dev Neuropsychol. 2001;20:(1)375–383.
  6. Wicherts JM, Dolan CV, Carlson JS, van der Maas HLN. Raven’s test performance of Sub-Saharan Africans: average performance, psychometric properties, and the Flynn effect. Learn Individ Differ. 2010;20(3):135–151.
  7. Zuilkowski SS, McCoy DC, Serpell R, et al. Dimensionality and the development of cognitive assessments for children in Sub-Saharan Africa. J Cross Cult Psychol. 2016;47(3):341–354.
  8. Ardila, A. Cultural values underlying psychometric cognitive testing. Neuropsychol Rev. 2005;15(4):185–195.
  9. Serpell R. How specific are perceptual skills? A cross of pattern reproduction. Br J Psychol. 1979;70(3):365–380.
  10. Deregowski JB, Serpell R. Performance on a sorting task: a cross cultural experiment. Int J Psychol. 1971;6(4):273–281.
  11. Gladstone M, Lancaster GA, Umar E, et al. The Malawi Developmental Assessment Tool (MDAT): the creation, validation, and reliability of a tool to assess child development in rural African settings. PLoS Med. 2010;7(5):e1000273.
  12. Tulving E. The effects of presentation and recall of material in free-recall learning. J Verbal Learn Verbal Behav. 1967;6(2):175–184.
  13. Donaldson W. Output effects in multitrial free recall. J Verbal Learn Verbal Behav. 1971;10(5):577–585.
  14. Karpicke JD, Roediger HL III. Repeated retrieval during learning is the key to long-term retention. J Mem Lang. 2007;57(2):151–162.
  15. Lachman R, Laughery KR. Is a test trial a training trial in free recall learning? J Exp Psychol. 1968;76(1 Pt 1):40–50.
  16. Roediger HL III, Butler AC. The critical role of retrieval practice in long-term retention. Trends Cogn Sci. 2011;15(1):20–27.
  17. Roediger HL III, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci. 2006;17(3):249–255.
  18. Bousfield AK, Bousfield WA. Measurement of clustering and of sequential constancies in repeated free recall. Psychol Rep. 1966;19(3):935–942.
  19. Central Statistical Office, Government of Zambia. 2010 census of population and housing. Dec 2012. https://web.archive.org/web/20151026014633/http://www.zamstats.gov.zm/report/Census/2010/National/2010%20Census%20of%20Population%20National%20Analytical%20Report.pdfAccessed 6 Mar 2022.
  20. Roenker DL, Thompson CP, Brown SC. Comparison of measures for the estimation of clustering in free recall. Psychol Bull. 1971;76(1):45-48.
  21. World Health Organization. Handbook for Good Clinical Research Practice (GCP): Guidance for Implementation. World Health Organization. 2005.
  22. StataCorp. Stata: Release 15. Statistical Software. StataCorp LLC; 2017.
  23. Senkova O, Otani H. Category clustering calculator for free recall. Adv Cogn Psychol. 2012;8(4):292–295.
  24. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163.
  25. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284–290.
  26. Sawilowsky SS. New effect size rules of thumb. J Mod Appl Stat Methods. 2009;8(2):597–599.
  27. Manning R, Kahana MJ. Interpreting semantic clustering effects in free recall. Memory. 2012;20(5):511–517.
  28. Deese J, Kaufman RA. Serial effects in recall of unorganized and sequentially organized verbal material. J Exp Psychol. 1957;54(3):180–187.
  29. Murdock BB. The serial position effect of free recall. J Exp Psychol. 1962;64(5):482–488.
  30. Kahana MJ. Associative retrieval processes in free recall. Mem Cogn. 1996;24(1):103–109.