by Leslie Citrome MD, MPH
Dr. Citrome is Clinical Professor of Psychiatry and Behavioral Sciences at New York Medical College in Valhalla, New York, and is the Editor-in-Chief for the International Journal of Clinical Practice, published by Wiley-Blackwell. He is also a member of the World Association of Medical Editors, the International Society for Medical Publication Professionals, and a participant in the Medical Publishing Insights and Practices initiative.
Innov Clin Neurosci. 2014;11(7–8):42–46
Funding: No funding or external editorial assistance was provided for the production of this article.
Financial Disclosures: In the past 36 months Dr. Citrome has engaged in collaborative research with, or received consulting or speaking fees, from: Alexza, Alkermes, AstraZeneca, Avanir, Bristol-Myers Squibb, Eli Lilly, Forest, Forum (Envivo), Genentech, Janssen, Jazz, Lundbeck, Medivation, Merck, Mylan, Novartis, Noven, Otsuka, Pfizer, Reckitt Benckiser, Reviva, Shire, Sunovion, Takeda, Teva, and Valeant.
Key words: Clinical trial registries, digital archives, grey literature, PubMed, publishers, searching
Abstract: Clinical trial results have been traditionally communicated through the publication of scholarly reports and reviews in biomedical journals. However, this dissemination of information can be delayed or incomplete, making it difficult to appraise new treatments, or in the case of missing data, evaluate older interventions. Going beyond the routine search of PubMed, it is possible to discover additional information in the “grey literature.” Examples of the grey literature include clinical trial registries, patent databases, company and industry-wide repositories, regulatory agency digital archives, abstracts of paper and poster presentations on meeting/congress websites, industry investor reports and press releases, and institutional and personal websites.
Introduction
The United States National Center for Biotechnology Information (NCBI) operates an array of online resources intended to facilitate access to scientific and health information, with a focus on biomedicine and genomics (http://www.ncbi.nlm.nih.gov/). Among those resources is PubMed (http://www.ncbi.nlm.nih.gov/pubmed/), a database of citations and abstracts for biomedical literature from MEDLINE (the United States National Library of Medicine bibliographic database that contains over 21 million references to journal articles in life sciences with a concentration on biomedicine) and additional life science journals. Links are provided when full text versions of the articles are available via PubMed Central (a digital archive of full-text journal literature) or other websites. With regard to clinical trial results, if they are published in a medical journal indexed on PubMed, they are relatively easy to locate. However, accessing the full-text may provide some challenges.[1]
PubMed is not exhaustive. Additional resources such as Google Scholar (http://scholar.google.com/) and subscription-based bibliographic databases such as Elsevier’s EMBASE (http://www.elsevier.com/ online-tools/embase) and Thomson Reuters’ Web of Science (http://wokinfo.com/citationconnection/) include periodicals that PubMed does not currently index, and should also be searched when conducting a systematic review.
Unfortunately, the dissemination of information through journal articles can be delayed or incomplete, making it difficult to appraise new treatments, or in the case of missing data, evaluate older interventions. However, published journal articles are not the only source for finding clinical trial results. The “grey literature,” materials not published commercially or indexed by major databases such as PubMed, can also be searched.
What is the grey literature?
The Fourth International Conference on Grey Literature in Washington, DC, in October 1999 defined grey literature as follows: “That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers.”[2] The University of Adelaide Library uses the term grey literature to denote “research that is either unpublished or has been published in non-commercial form. As the growth of the web has increased opportunities, grey literature is now freely available on many sites. However it is only selectively indexed by database vendors. Many organizations and individuals also provide access to their works online, and most of this material remains unindexed.” (http://libguides.adelaide.edu.au/content.php?pid=266172&sid=5155372).
The New York Academy of Medicine operates a website “Grey Literature Report” (http://www.greylit.org/) that provides links to new grey literature publications in health services research and selected public health topics. The Canadian Agency for Drugs and Technologies in Health has made available a hypertext guide to searching the medical grey literature (http://www.cadth.ca/en/ resources/finding-evidence-is/grey-matters). A specialized search engine geared toward medicine can be found at http://mednar.com/. GreySource is another web portal that provides links to a comprehensive array of information sources across many scientific disciplines (http://www.greynet.org/greysourceindex.html).
For clinical trials for which results have not been published, as well as for more complete information for clinical trials that have been published, the grey literature can be a treasure drove of data. Examples of the grey literature relevant to the topic of communication of clinical trial results include clinical trial registries, patent databases, company and industry-wide repositories, regulatory agency digital archives, abstracts of paper and poster presentations on meeting/congress websites, industry investor reports and press releases, and institutional and personal websites.
ClinicalTrials.gov
Operated by the National Library of Medicine, ClinicalTrials.gov (http://clinicaltrials.gov/) is a registry and results database of publicly and privately supported clinical studies of human participants. ClinicalTrials.gov was created as a result of the United States Food and Drug Administration (FDA) Modernization Act of 1997 (https://clinicaltrials.gov/ct2/about-site/background). It is now expected that clinical trials are registered prior to their commencement and that proof of this is provided when submitting results for publication. Although the studies listed in the database are conducted in all 50 States and in 187 countries, additional registries exist elsewhere in the world, such as the European Clinical Trials Database (https://eudract.ema.europa.eu/).
Study information available at ClinicalTrials.gov includes the purpose, disease/condition to be examined, interventions to be tested, study phase (i.e., 1, 2, 3, and 4), study type, study design, primary outcome measures, secondary outcome measures, enrollment target, start and stop dates, eligibility requirements, study site locations, and optional links for further information. It is becoming more common for journal reviewers to cross-check the ClinicalTrial.gov registry entry with the submitted study manuscript to check for concordance.
The ClinicalTrials.gov registration requirements were expanded in 2007 after passage of the FDA Amendments Act, including the requirement for the submission of results for certain trials. This led to the development of the ClinicalTrials.gov results database, which contains information on study participants and a summary of study outcomes, including adverse events. Upon completion of a study, top-line results are generally required to be posted within one year (with exceptions made for clinical trials of drugs, biological products, or devices that have not been approved, licensed, or cleared by the FDA). Because dissemination of the clinical trial results can thus occur quickly, often before a published manuscript can be made available, the ClinicalTrials.gov results database is an excellent example of the grey literature providing information in a more timely way than traditional means.
Patent databases
Patent databases such as Google Patents (https://www.google.com/? tbm=pts) can provide information regarding new products in development or new formulations of older medications. This can be the only publically available source extant for many compounds.
Company and industry-wide repositories
Some pharmaceutical companies have established their own clinical trial registries that supplement the information that can be found elsewhere. For example, the Eli Lilly and Company Clinical Trial Registry (http://www.lillytrials.com/) and the AstraZeneca Clinical Trials Website (http://www.astrazenecaclinicaltrials.com/) provide lists of studies conducted together with links to multipage clinical study summaries. Other companies have subscribed to services such as TrialScope’s PharmaCM (http://www.trialscopeinc .com/) that host study synopses. Links to these synopses are often available in the ClinicalTrials.gov registry record and take the user to the hosting website. Initiatives by several companies include pathways to request clinical study reports or participant level data (http://www.clinicaltrialstudy transparency.com/, https://clinical studydatarequest.com/).
Regulatory agency digital archives
For medications that have received US FDA approval, Drug Approval Packages are made available at http://www.accessdata.fda.gov/scripts/cder/drugsatfda/. Each Drug Approval Package consists of results and analyses from preclinical and all clinical phases, including integrated tolerability/safety analyses.[3–6] After a product receives initial approval for an indication, information may not be as readily available for subsequent indications.[4] The documents that are downloaded are in PDF format and not necessarily searchable. It is useful to concatenate the documents into one large file and then use an optical character recognition program (such as Adobe Acrobat) to render the file searchable. The resultant document is often about 1000 pages in length.
The Drug Approval Packages are made available for approved medications only. Information about the clinical trials for medicines that have not received FDA approval may be available in briefing documents prepared by both the FDA and the pharmaceutical manufacturer in preparation for Advisory Committee meetings.[5,6] These documents, together with slide presentations and transcripts from the Psychopharmacologic Drugs Advisory Committee meetings can be accessed through the website http://www.fda.gov/AdvisoryCommittees/CommitteesMeetingMaterials/Drugs/PsychopharmacologicDrugsAdvisoryCommittee/default.htm.
The FDA digital archive also includes product package inserts (product labels) that can contain information not otherwise available in published reports and represent authoritative sources for further research and analyses.[7,8] Examples of their use include analyses of adverse event data from pooled data of all available clinical trials, including those that have not been published in a medical journal.[8] The FDA digital archive also includes older versions of the package inserts as well, making these accessible without needing to resort to internet archive portals such as https://archive.org/web/.
Similar efforts at making clinical trial information more available are in progress at the European Medicines Agency. Currently available are detailed public assessment reports at http://www.ema.europa.eu/, although it is more expedient to search on Google “EMEA EPAR” together with the name of the drug of interest. Product labeling in the EU (referred to as summaries of product characteristics and abbreviated as SmPC or SPC) are more detailed than in the United States and can contain additional information about the clinical trials for the drug of interest.
In the US and Europe, debate is ongoing regarding making available to the public the full study protocols and clinical study reports, particularly for new applications. This would represent a quantum leap from the clinical study synopses that are currently available today. Clinical study reports are lengthy documents that contain a complete set of all results from a particular clinical trial, including safety narratives that will require some redaction in order to preserve patient subject anonymity. Potentially useful would also be access to the New Drug Application file as submitted by the company seeking approval of their product; however, concerns over disclosure of proprietary information remains a significant obstacle.
Meeting/congress websites
Although the abstracts of papers and posters presented at some meetings are published as supplements to indexed journals and can be found using search engines such as EMBASE and Web of Science (but not PubMed), for other congresses the abstracts are unavailable unless one has attended the meeting. However, it is now becoming more common for the professional organization that is hosting the meeting to archive the program book and syllabi on the organization’s website. An example is the American Psychiatric Association’s website, http://www.psychiatry.org/learn/library–archives, where these documents are freely available and go back to as early as 1973, although not all years are represented. The authors of the poster presentations can often be contacted for PDF copies, or in the case of industry-sponsored poster presentations, the medical affairs department of that company can also be contacted to request a copy.
Industry investor reports and press releases
For business reasons, some clinical trials of new agents are never presented at a scientific congress, much less published in a medical journal. Registration of the trial may have occurred, but no study results disclosed. Nevertheless, some mention of the investigational agent can usually be found in press releases and investor-related reports for publically traded companies. This requires knowledge of the name of the manufacturer and the name of the medication under study and entering these into an internet search engine such as Google. Companies usually maintain archives of their press releases on their websites. It is also possible to also find audio files of “investor conference calls,” where clinical trial results can be mentioned. Searching for specific file types in Google, such as “filetype:ppt” and “filetype:pdf” can yield investor and analyst slide presentations.
On occasion, disclosures about medications and adverse events are made in filings with the US Securities and Exchange Commission. These get picked up by the lay press and disseminated through newspapers such as the Wall Street Journal, the New York Times, and internet blogs such as http://blogs.wsj.com/ pharmalot/.
Institutional and personal websites
Government institutions such as the National Institute of Mental Health have websites that can provide information from clinical trials that go beyond what is found in the published literature. An example is the creation of specific websites that focus on mega-projects, such as the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) Study (http://www.nimh.nih.gov/ funding/clinical-trials-for-researchers/practical/catie/index.shtml), the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Study (http://www.nimh .nih.gov/funding/clinical-trials-for-researchers/practical/stard/), and the Systematic Treatment Enhancement Program for Bipolar Disorder (STEP-BD) (http://www.nimh.nih.gov/ funding/clinical-trials-for-researchers/ practical/ step-bd/).
Systematic reviews of clinical trials can be found on websites such as those operated by the US Agency for Healthcare Research and Quality (AHRQ) (http://www.ahrq.gov/) and UK’s National Institute for Health and Care Excellence (NICE) (http://www.nice.org.uk/).
Universities also often host archives of materials produced by their faculty and graduate students, in some cases including dissertations. Individual members of the university may also have their own webpage as part of the university site and may have the ability to upload to it. Faculty may sometimes also create their own website outside the direct purview of their institution and thus independent of any oversight. A google search may lead a reader to one of these websites and sometimes PDF copies of papers, posters and even books can be found, even though in some instances this is in violation of copyright. Problems can occur when citing online resources that turn out to be ephemeral.
Lastly, scholarly networking sites such as https://www.researchgate.net/can allow access to the full-text of papers or other work products, unavailable elsewhere.
Summary
The availability of the internet has transformed how we can search for information, and transparency initiatives have increased the amount of data available to way beyond what was accessible a short one or two decades ago. Persons interested in clinical trial results are no longer restricted to what is contained in scholarly publications in indexed journals. The so-called grey literature includes clinical trial registries (together with newly created result databases), patent databases, company and industry-wide repositories, digital archives maintained by regulatory authorities such as the FDA, websites maintained by professional organizations that archive abstracts from national and international congresses, financially driven investor service websites, and institutional, scholarly-networking, and personal websites.
References
1. Citrome L, Moss SV, Graf C. How to search and harvest the medical literature: let the citations come to you, and how to proceed when they do. Int J Clin Pract. 2009;63(11):1565–70.
2. The New York Academy of Medicine. Fourth International Conference on Grey Literature: new frontiers in grey literature. Grey Literature Network Service. Washington DC, USA, 4–5 October 1999.
3. Citrome L. Iloperidone redux: a dissection of the Drug Approval Package for this newly commercialized second-generation antipsychotic. Int J Clin Pract. 2010;64(6):707–18.
4. Turner EH. How to access and process FDA drug approval packages for use in research. BMJ. 2013;347:f5992.
5. Ohmer J, Miller JW, Sasich LD. Accessing FDA approval packages and briefing documents. Ann Pharmacother. 2007;41(12):2071–2.
6. Sasich LD, Sukkari SR, Cook GE, Tuttle DA. The importance of FDA approval packages and briefing documents in pharmacy education. Am J Pharm Educ. 2009;73(7):126.
7. Citrome L, Nasrallah HA. On-label on the table: what the package insert informs us about the tolerability profile of oral atypical antipsychotics, and what it does not. Expert Opin Pharmacother. 2012;13(11):1599–613.
8. Citrome L. Quantifying risk: the role of absolute and relative measures in interpreting risk of adverse reactions from product labels of antipsychotic medications. Curr Drug Saf. 2009;4(3):229–37.
