by Philip D. Harvey, PhD; Michael Foster Green, PhD; and Keith H Nuechterlein, PhD
Dr. Harvey is Professor of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, Georgia. Drs. Green and Nuechterlein are Professors of Psychiatry, Department of Psychiatry, David Geffen School of Medicine at UCLA, Los Angeles, California.
Psychiatry (Edgemont). 2010;7(6):49–52
Funding: There was no funding for the development and writing of this article.
Financial Disclosures: The authors have no conflicts of interest relevant to the content of this article.
Key words: Schizophrenia, cognition, disability, cognitive enhancement
Abstract: The Measurement and Treatment Research to Improve Cognition in Schizophrenia Research process has led to several developments in the assessment of cognitive functioning for schizophrenia-treatment studies. The first development was the development of a consensus cognitive battery and a United States Food and Drug Administration-endorsed research design. Since the development of the cognitive battery, interest has been spurred in clinical trials in different countries and the development of co-primary functional outcomes measures for these. The MATRICS Consensus Cognitive Battery has been translated into 11 different languages and is being translated into even more. A study has been completed that compared the usefulness of multiple potential co-primary measures, suggesting that the University of California San Diego Performance-Based skills assessment, version II (UPSA-II) is the most suitable for studies conducted in English. These findings suggest that reliable performance-based measures that are easy to administer and highly correlated with cognitive functioning are now available for use in treatment studies.
Introduction
As we have discussed before in this column, the Measurement and Treatment Research to Improve Cognition in Schizophrenia Research (MATRICS)[1] process helped to develop the MATRICS Consensus Cognitive Battery (MCCB)[2] for measuring cognitive treatment outcomes in schizophrenia, helped achieve consensus on an appropriate research design and also spurred interest in the treatment of cognitive impairment in schizophrenia. This assessment battery has already been used in multiple treatment trials and is available from commercial psychological assessment vendors. In addition, at the time of the development of the MCCB, a validation study was performed to examine the relationship between the MCCB and performance-based measures of functional capacity and real-world functional outcomes.[3] This is an important aspect of the MATRICS process, because the United States Food and Drug Administration (FDA) is requiring that studies of cognitive enhancement demonstrate cognitive improvement, indexed by MCCB performance, as well as improvement on functional indices.
Many companies interested in cognitive enhancement studies want to have the option to conduct studies outside the United States. Given that such options require valid versions of the MCCB in other languages, one of the new developments in the MATRICS process was the development of translated versions of the MCCB for use in other countries. An additional need that was addressed by the latest steps in the MATRICS process was for definitive information on co-primary outcomes measures, either interview-based assessments of cognitive impairments or performance-based measures that address functional capacity. Thus, an additional phase of the MATRICS process involved a validation study that examined the relationships between various performance and interview-based co-primary measures for use in cognitive enhancement studies. A final part of that process was a cross-cultural survey, wherein experts on schizophrenia in multiple different countries were presented with the likely candidate co-primary measures and asked to comment on the degree to which they could be easily adapted for use in their respective countries. We will talk about the first two goals and then address the final issue—cross-cultural functional assessment—in a subsequent column.
This process was supported by a consortium of pharmaceutical industry partners. These partners made equal donations to a fund established and managed by the Foundation for the National Institutes of Health (FNIH). This allowed for funding that originated in industry, but was administered impartially by the government, avoiding any suggestions of conflict of interest for the participants or undue influence on the part of the sponsors.
Translation
The MCCB has been successfully translated and implemented in multiple foreign languages. These translations have occurred through two different routes: commercial and academic. For the commercial route, MCCB has been professionally translated into 10 foreign languages. The MATRICS-CT Consortium sponsored the translation of the MCCB into the following: Simplified Chinese, German, Hindi, Russian, Spanish for Spain, and Spanish for Latin America. These translations are currently commercially available through test publishing companies. In addition, companies have contracted to develop professional translations in four additional languages: Croatian, Hebrew, Italian, and Japanese. These four translations are finished and the languages are in final stages of page composition and printing. The commercial translations go through a very extensive process intended to meet FDA requirements for translations, including two forward translations, two back translations, multiple reconciliation stages, and field testing by local language experts. Information on the translations is available at www.matricsinc.org.
Another route for translation has come from academic researchers in other countries who want to translate the MCCB for their research. In this academic model, investigators need to obtain permission for the translations because the tests comprising the MCCB are owned by separate test developers who retain the copyright. The permission and translation process is done by the academics with help from MATRICS Assessment Inc., a nonprofit company formed to facilitate use of the MCCB. The steps involved with an academic translation of the MCCB are much simpler than a commercial version, but still need to include forward and back translations. Academic translations of the MCCB are being developed for a range of languages, including Norwegian, Polish, Brazilian Portuguese, French, Dutch, Turkish, and Armenian.
Interesting issues arise in translation of psychological tests. For example, some Asian languages do not have a sequential alphabet. As a result, letter-number sequencing, which requires organizing strings of numbers and letters into ascending and alphabetical order, respectively, is not directly translatable. Further, certain words are not directly translatable from one language to another and lead to complex, multisyllabic or hyphenated words. Because the stimuli on word list learning tests are selected for comparable complexity and frequency, some words need to be replaced.
Co-Primary
The Validation of Intermediate Measures (VIM) study was sponsored by the MATRICS-CT Consortium and was completed in October of 2009. This study was conducted over an 18-month period, beginning with an expert survey, followed with a RAND Corp. (Santa Monica, California) panel to narrow the field of potential candidate measures, then a four-site validation study. Each of these tasks was performed similarly to the original MATRICS process for selection of the MCCB.
A paper with the results of the VIM study is currently under review, so we will present an overview of the process and main findings that have been presented at conferences. One of the issues that had to be considered is that, in contrast to the neuropsychological tests evaluated in the original MATRICS Psychometric and Standardization Study, many of the potential co-primary measures considered had been used in only one or two previously published studies. Further, many of the functional measures were highly specialized and were seen to have limited potential for international studies. Another issue is the use of informants. As it is widely appreciated that there are some limitations associated with self-reports of functioning on the part of individuals with schizophrenia, there was considerable discussion about whether informants would be used for interview-based assessments of cognitive functioning. In the end, the decision was made to have the interview-based measures studied based on interviews with only the patients, partly because of concerns that they would be used in that manner in most clinical trials.
The basic design of the VIM study was a test-retest study with stable outpatients with schizophrenia. A sample of 163 participants was collected, assessed with the MCCB, and rated with the PANSS and the candidate co-primary measures. These potential co-primary measures were selected by a RAND panel and the final measures included three performance-based measures—the Test of Adaptive Behavior in Schizophrenia (TABS),[4] the UCSD Performance-Based Skills Assessment (UPSA),[5] the Independent Living Scales (ILS)[6]—and two interview-based measures—Cognitive Assessment Interview (CAI)[7] and Clinician Global Impression scale for cognition (CGI-cognition).[8] All three of the performance-based measures were also examined as short forms. All available participants were reassessed four weeks later.
While all of the measures (and the short forms) manifested suitable test-retest reliability and manageable practice effects, there was some variability in a critical factor: correlation with the MCCB. While all three performance-based measures and two of the three short forms all shared at least 26-percent variance with the MCCB, the interview-based measures manifested correlations that were much smaller, sharing as little as five-percent variance. In addition, the time required to perform the assessments varied considerably, with the longest full-length, performance-based measure requiring an average of 46 minutes to complete and the shortest full-length form requiring 27 minutes. All of the short forms required about 15 minutes.
The VIM committee agreed that the full form of the UPSA manifested the best combination of test-retest reliability, concurrent validity, practicality, and tolerability. For the short forms, both the TABS and UPSA appeared suitable. All short forms had lower convergent validity and slightly lower reliability, as would be expected from abbreviated measures. The main implication of this finding is that larger sample sizes would be required for short forms to identify the same treatment-related effect sizes. As the longer form of the UPSA was well-tolerated and had only one case of 163 with missing data, it seems prudent to use the longer form for treatment studies in order to conserve sample size, although for large-scale correlational studies, the short forms of the performance-based measures seem suitable.
References
1. Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS). http://www.matrics.ucla.edu/index.shtml. Accessed on June 16, 2010.
2. Nuechterlein KH, Green MF, Kern RS, et al. The MATRICS Consensus Cognitive Battery: Part 1. test selection, reliability, and validity. Am J Psychiatry. 2008;165:203–213.
3. Green MF, Nuechterlein KH, Kern RS, et al. Functional co-primary measures for clinical trials in schizophrenia: results from the MATRICS Psychometric and Standardization Study. Am J Psychiatry. 2008;165:221–228.
4. Patterson TL, Goldman S, McKibbin CL, et al. UCSD performance-based skills assessment: development of a new measure of everyday functioning for severely mentally ill adults. Schizophr Bull. 2001;27:235–245.
5. Velligan DI, Diamond P, Glahn, DC, et al. The reliability and validity of the test of adaptive behavior in schizophrenia. Psychiatry Res. 2007;151:55–66.
6. Loeb PA. Independent living scales. Psychological Corporation: San Antonio; 1996.
7. Ventura J, Reise SP, Keefe RS, Baade LE, et al. The Cognitive Assessment Interview (CAI): development and validation of an empirically derived, brief interview-based measure of cognition. Schizophr Res. 2010 Jun 11. Epub ahead of print.
8. Ventura J, Cienfuegos A, Boxer O, Bilder R. Clinical global impression of cognition in schizophrenia (CGI-CogS): reliability and validity of a co-primary measure of cognition. Schizophr Res. 2008;106(1):59–69. Epub 2007 Sep 27.