There are several other important features of this design. In order for the procedure to simulate the reality of clinical practice, we did not perform extensive assessment of inter-rater reliability after the initiation of the project. Instead, our goal was to determine if using a semi-structured interview and a trained rater would generate stable diagnoses. To examine this question, we compared the rate of clinician change of rater-generated admission diagnoses across the three raters. Thus, the outcome was diagnostic stability across raters within rating method and not agreement on a specific diagnosis such as bipolar disorder or schizophrenia. Patients were treated at this facility on average over three months, with regular therapist and psychiatric consultations and round-the-clock clinical observation.
Primary axis-I diagnoses were examined during the entire period of treatment for each case during the three year period. Changes in the original admission diagnosis prior to discharge from treatment were recorded as the primary outcome measure. As a secondary outcome in Years 2 and 3, the original clinician diagnosis was compared to the admission diagnosis assigned following the structured diagnostic interview. In the calculation of “change in diagnosis,” we used the following rules: 1) We generated global categories in order to avoid characterizing minor changes in diagnoses as discrepant. For instance, we considered a diagnosis of bipolar II and bipolar I disorder to be consistent, although a change in diagnosis from bipolar depression to major depression was considered a change. 2) We did not consider schizophrenia subtypes as part of the diagnostic agreement, but considered schizophrenia to be different from schizoaffective disorder. 3) Changes in clinical state codes (i.e., severe to remission) during the course of treatment within the same diagnosis were not considered as a difference in diagnosis.
Results
As can be seen in Table 1, the overall pattern of admission diagnoses changed slightly over the study period. Diagnoses of major depressive disorder became more common and diagnoses of psychosis became less so. When the primary outcome, change in diagnosis from admission to discharge, was examined there were clear differences across the methods. In the year prior to the implementation of the SCID, 74 percent of admission diagnoses based on referral diagnosis and unstructured interview were changed over the course of the treatment period, with the most common changes being that an admission diagnosis of major depression was changed to a diagnosis of bipolar disorder or vice versa. In marked contrast, the rates of change of diagnoses generated by the SCID during similarly lengthy stays was four percent and the rates of changes in MINI diagnoses was 11 percent. Chi-square tests were used to compare the differences in rate of change in diagnoses across procedures. The difference in rates of change between the clinical diagnostic assessment procedure and the SCID was significant, chi2(1)=19.09, p<0.001, as was the difference between clinical diagnoses and the MINI, chi2(1)=7.50, p<0.005. However, the difference in rates of changes in diagnoses between SCID and MINI procedures was not significant, chi2(1)= 2.01, p=0.16.
In order to determine whether there were differences across the three raters in the extent to which their diagnoses were changed over the course of treatment for the patients, we performed a 3 (rater) x 2 (changed or not changed) Chi-square test for each of the three years. All three years suggested no differences across the three raters in the extent to which their diagnoses were changed by the clinicians (all chi2(2 df)<1.47, all p>0.48).
In the analyses of data from Years 2 and 3, we compared clinical diagnosis provided by the referring source to the diagnoses generated with the structured procedures. These data are presented in Table 2. See Figure 1 for a graphic depiction of these results. There were substantial discrepancies between these diagnoses. Of the cases interviewed with the SCID, 56 percent of the cases were assigned a diagnosis that was different from that provided by the referral source and for the MINI the number of cases whose diagnosis was different was 42 percent. Diagnostic confirmation rates for bipolar were 40 percent and 50 percent for the two years, and confirmation of major depressive disorder were somewhat higher. Most diagnostic discrepancies were interview-based diagnoses of major depression in cases referred as bipolar and bipolar depression in cases referred as major depression. Diagnostic confirmation of schizoaffective disorder was also very low at 10 percent and 11 percent across the two years.
Discussion
Given the importance of matching appropriate diagnosis with evidence-based pharmacologic, psychotherapeutic, and psychosocial interventions, identifying an accurate working diagnosis quickly and efficiently in community mental health settings is essential. Relying on referring diagnoses and self-report of previous diagnoses may yield a diagnosis that requires modification, even in the context of an initial unstructured psychiatric diagnostic assessment by experts. Implementing structured interviews in such settings may be prudent, as rates of diagnostic changes were significantly reduced in our study using both the SCID and MINI. Length of administration of the MINI is shorter than administering the SCID (20 minutes compared to over 90 minutes), and diagnostic stability was not notably different between these two structured interviews in our cohorts.
Themes of modifications in referral diagnoses in our study mirrored data from other community diagnostic studies. In our Year-2 and Year-3 cohorts, patients who presented initially with diagnoses of bipolar illness, unipolar depression, and schizoaffective disorder had significant rates of re-diagnosis of their primary mental illness using a structured diagnostic tool. Bipolar illness prior to admission seemed to be both over-diagnosed and under-recognized within patients referred to this treatment facility. While the origin of the pattern of such diagnostic changes is unclear, there was both a bias toward over-diagnosis of bipolar disorder in the absence of symptomatic evidence in patients eventually diagnosed with major depression and tendencies to miss euphoric bipolar symptoms. Unsystematic assessment may produce both types of diagnostic errors: inadequate knowledge of the signs of a manic or hypomanic episode may lead to a false positive bipolar diagnosis and failure to assess for manic episodes may lead to false negatives. Schizoaffective disorder seems to be more often found in the diagnostic opinions of clinicians than in the results of structured assessments.
There are some limitations to these data and these analyses. Stability is not the only element of validity; we did not examine treatment response, biomarkers, or course of illness as validity indicators. We could not quantify the reasons why clinicians changed admission diagnoses, and there may be several reasons for these changes. It also is possible that clinicians were less likely to change diagnoses generated by structured interviews, but the clinicians were not aware we planned to examine changes in diagnosis as an outcome variable in a research study. Discrepancies between referring clinician diagnoses and the results of structured assessments cannot be attributed to bias on the part of clinicians toward not modifying a diagnosis. The facility at which these analyses occurred is a private, non-profit treatment program that does not bill Medicaid or Medicare. A selection bias in referred patients may limit generalizability of these outcomes to other treatment settings in which patients from more varied economic groups are assessed and the applicability to inpatient settings cannot be determined. The very short stays typical in current inpatient treatment make the use of structured diagnostic assessments less useful. A relatively younger age of the patients in this study may limit generalizability of the findings to older patients who have had longer experiences with serious, persistent mental illnesses. Neither the SCID nor the MINI was designed for the purpose of diagnosing axis-II pathology and as a result, these diagnoses could not be systematically assessed in this study. The required investment in training clinicians and administering the standardized assessments may similarly limit the practicality of assessment of these outcomes in busy community mental health treatment centers, where resources tend to be limited.
Although the study participants were on average quite young, patients were typically not recovering from their first episode of illness, during which time actual changes in symptomatology and presentation might account for diagnostic uncertainty. As treatment options for mental illnesses continue to improve, diagnostic stability and reliability become even more important in community mental health settings. Pharmacologic, psychotherapeutic, and social interventions used to treat patients with bipolar illness are significantly different than similar classes of interventions for patients with personality disorders or even unipolar depression. This study suggests that the up-front investment of effort and time to use a structured diagnostic assessment at the time of admission to residential, partial hospitalization, and intensive out-patient programs may be a wise course of action for patients and payers alike. Matching specific and timely treatment to the appropriate diagnosis makes sense for all stakeholders, despite the requisite time involved in administering the assessments.
Future directions for efficiently diagnosing mental illness in community psychiatric facilities should include assessing variables related to patients’ socioeconomic factors, referral sources, age of patients, and stigma. Although currently limited in application, including biomarkers and imaging data to make clinical diagnoses will also help determine cost-effective and practical structured diagnostic tools that busy community clinicians may implement in their treatment planning for patients.
References
1. Beck AT, Ward CH, Mendelson M, et al. Reliability of psychiatric diagnosis 2: a study of consistency of clinical judgments and ratings. Am J Psychiatry. 1962;119:351–357.
2. Wing JK, Birley JL, Cooper JE, et al. Reliability of a procedure for measuring and classifying “present psychiatric state. Br J Psychiatry. 1967;113:499–515.
3. Spitzer RL, Endicott J. Current and Past Psychopathology Scales. New York: New York State Psychiatric Institute, Biometrics Research Division; 1968.
4. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 3rd edition. Washington, DC: American Psychiatric Press, Inc.; 1980.
5. Amador XF, Flaum M, Andreasen NC, et al. Awareness of illness in schizophrenia and schizoaffective and mood disorders. Arch Gen Psychiatry. 1994;51:826–836.
6. Bowie CR, Twamley EW, Anderson H, et al. Self-assessment of functional status in schizophrenia. J Psychiatr Res. 2007; 41:1012–1018.
7. Burdick KE, Endick CJ, Goldberg JF. Assessing cognitive deficits in bipolar disorder: are self-reports valid? Psychiatry Res. 2005;136:43–50.
8. Harvey PD, Heaton RK, Carpenter, WT, Jr., et al. Diagnosis of schizophrenia: consistency across information sources and stability of the condition. Schizophr Res. 2012;140:9–14.
9. Schwartz JE, Fennig S, Tanenberg-Karant M, et al. Congruence of diagnoses 2 years after a first-admission diagnosis of psychosis. Arch Gen Psychiatry. 2000;57:593–600.
10. Bromet EJ, Naz B, Fochtmann LJ, et al. Long-term diagnostic stability and outcome in recent first-episode cohort studies of schizophrenia. Schizophr Bull. 2005;31:639–664.
11. Yutzy SH, Woofter CR, Abbott CC, et al. The increasing frequency of mania and bipolar disorder: causes and potential negative impacts. J Nerv Ment Dis. 2012;200:380–387.
12. Keck PE Jr, Kessler RC, Ross R. Clinical and economic effects of unrecognized or inadequately treated bipolar disorder J Psychiatr Pract. 2008;14(Suppl 2):31–38.
13. Hoch P, Pollatin P. Pseudoneurotic forms of schizophrenia. Psychiatric Q. 1949;23:248–276.
14. Pogge DL, Wayland-Smith D, Zaccario M, et al. Diagnosis of manic episodes in adolescent inpatients: structured diagnostic procedures compared to clinical chart diagnoses. Psychiatry Res. 2001;101:47–54.
15. Zimmerman M, Ruggero CJ, Chelminski I, Young D. Is bipolar disorder overdiagnosed? J Clin Psychiatry. 2008;69:935–940.
16. Hirschfeld RM, Vornik LA. Recognition and diagnosis of bipolar disorder. J Clin Psychiatry. 2004;65(Suppl 15):5–9.
17. Leboyer M, Kupfer DJ. Bipolar disorder: new perspectives in health care and prevention. J Clin Psychiatry. 2010;71:1689–1695.
18. Harvey PD, Yehuda R. Strategies to study risk for Post-traumatic stress disorder. In Yehuda R (ed). Risk Factors for Post-traumatic Stress Disorder. Washington, DC: American Psychiatric Press, Inc.;1999:1–22.
19. Jäger M, Haack S, Becker T, Frasch K. Schizoaffective disorder: an ongoing challenge for psychiatric nosology. Eur Psychiatry. 2011;26:159–165.
20. Malhi GS, Green M, Fagiolini A, et al. Schizoaffective disorder: diagnostic issues and future recommendations. Bipolar Disord. 2008;10:215–230.
21. Lake CR, Hurwitz N. Schizoaffective disorder merges schizophrenia and bipolar disorders as one disease: there is no schizoaffective disorder. Curr Opin Psychiatry. 2007;20:365–379.
22. First MB, Spitzer RL, Gibbon M, et al. User’s guide for the Structured Clinical Interview for DSM-IV Axis I (SCID-I). Washington, DC: American Psychiatric Press, Inc.;1995.
23. Sheehan DV, Lecrubier Y, Harnett?Sheehan K, et al. The Mini International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview. J Clin Psychiatry. 1998;59(Suppl 20):22–33.
