-- EPA's ability to assess risks and establish risk-based priorities has been hampered by data quality problems, including critical data gaps, databases that do not operate compatibly with one another, and persistent concerns about the accuracy of the data in many of EPA's data systems. While EPA's priorities should reflect an understanding of relative risk to the environment and public health, good data often do not exist to fully characterize risk. In the absence of reliable data, public perceptions of risk can influence how EPA determines its priorities and allocates resources. EPA has taken major steps during the past few years to improve its data and to better inform the scientific community and general public of environmental and public health risks. To finish this job, the agency will need to expand its data improvement initiatives to fill key gaps in its data, take advantage of opportunities to develop and implement data standards to achieve compatibility among environmental databases, and ensure the accuracy of its data.
-- Measuring the results (outcomes) of its programs is critical to determining EPA's effectiveness. Nevertheless, the agency historically has relied on activity-based output measures, such as the number of inspections performed, because of inherent technical difficulties in establishing sound linkages among program activities, environmental improvements, and public health. Spurred by the requirements of the Government Performance and Results Act of 1993 (Results Act), EPA has made progress in recent years in measuring the outcomes of its programs. To ensure future success in developing outcome measures, however, EPA will need to make a long-term management commitment to overcome major challenges to obtaining the data needed to show the results of environmental programs.
Background
Since EPA's establishment in 1970, the federal government has developed a complex system of laws and regulations to address the nation's environmental problems. Over the years, as environmental threats were identified, the Congress responded by enacting laws to address each problem, incrementally adding to the statutory framework that sets EPA's agenda. However, these laws were not coordinated or integrated to provide EPA with an overall system for prioritizing problems so that the most serious problems can be addressed first.
Impelled by budgetary constraints and a growing list of environmental problems, EPA, in the late 1980s, began to consider whether its resources were being spent on the problems that pose the greatest risks to public health and the environment. The agency concluded that the nation actually was devoting more resources to problems that had captured public attention than to problems that were less well known but potentially more serious. Subsequently, EPA began incorporating the concept of relative health and environmental risk into decisions on environmental priorities and emphasizing the need to identify the most serious risks and to keep the public informed about the relative seriousness of various environmental problems. To assess risks and deal with those likely to do the most harm, EPA has recognized that it needs to have adequate environmental and scientific data to conduct risk assessments, set standards, and develop regulations. It also needs such data to identify and develop measures of environmental quality and to assess the effectiveness of its programs by linking program activities to changes in environmental conditions.
EPA Needs Better Data to Establish Risk-based Program Priorities
Establishing risk-based priorities for EPA's program activities requires good data on the use and disposal of thousands of chemicals. To assess human exposure to a chemical, EPA needs to know how many workers, consumers, and others are exposed; how the exposure occurs; and the amount and duration of the exposure. For environmental exposure, EPA needs to know whether the chemical is being released to the air, water, or land; how much is being released; and how wide an area is being affected. EPA's ability to make such assessments is limited by (1) gaps in environmental and health data, (2) databases that do not operate compatibly with one another, and (3) the lack of an effective system for ensuring the accuracy of the agency's data. Although EPA has implemented several agencywide initiatives to address these problems, each of the initiatives has encountered obstacles that must be overcome to substantially improve the agency's data.
Extensive Gaps Exist in EPA's Information About the Environment and Health Risks
Our work over the past few years has shown that very little is known about the risks of potential exposure to chemicals and environmental conditions for workers, the general public, and plant and animal life. For example, we reported the following:
-- EPA's Integrated Risk Information System, which is a database of the agency's consensus on the potential health effects of chronic exposure to various substances found in the environment, lacks basic data on the toxicity of about two-thirds of the known hazardous air pollutants.2
-- EPA's National Water Quality Inventory does not accurately describe water quality conditions nationwide. Only 19 percent of the nation's rivers and streams were assessed for the 1996 Inventory (the latest report available at the time of our review), as were 6 percent of ocean and other shoreline waters. Pollution of the latter has resulted in an increasing number of beach advisories and closures in recent years.3
-- Of 1,456 toxic chemicals we recently reviewed, data on human exposure were being collected for only about 6 percent. For example, of the 476 chemicals that EPA identified as most in need of testing under the Toxic Substances Control Act, only 10, or 2 percent, were being measured for human exposure. (See table 1.)
Table 1: Extent to Which Human Exposure Data Are Collected for Potentially Harmful Chemicals Through Surveys of EPA and the Department of Health and Human Services
[omitted]
Note: Our analysis was based on human exposure data collected through the Department of Health and Human Services' National Health and Nutrition Examination Survey or EPA's National Human Exposure Assessment Pilot Surveys through 2000.
EPA has recognized that it has numerous and significant gaps in its data and has initiated several efforts to fill at least some of the gaps. For example, under its Environmental Monitoring and Assessment Program, EPA is working with other federal agencies to develop information that the public, scientists, and the Congress can use to evaluate the overall health of the nation's ecological resources. EPA also recently launched its High Production Volume Challenge Program, which asked chemical companies to voluntarily generate data on the effects of the chemicals they manufacture or import. As of December 1999, over 400 participants had agreed to make public, before the end of 2005, basic hazard data on over 2,000 of 2,800 high-production-volume chemicals, which are chemicals manufactured or imported into the United States in amounts equal to or greater than one million pounds per year. Furthermore, EPA's new information office will be responsible for encouraging the agency's program offices to reach out to other federal agencies as well as to universities, research institutes, and other sources of environmental information for data that EPA does not collect but that may exist elsewhere. To date, however, such efforts have been hampered by technological limitations imposed by the myriad of incompatible information systems in use across the government.
Moreover, much of the information needed, such as environmental monitoring data, will be expensive to obtain. Thus, it will be important for EPA to work with the states and industry to reduce the reporting burden and to encourage efforts to use data that may already have been collected by other federal agencies or other entities. Likewise, as we recommended to EPA in our September 1999 report on its information management activities, it will be essential for the agency to develop a strategy that prioritizes its requirements for additional data and identifies milestones and needed resources. EPA can then use this information to support its budget requests.
Incompatible Data Systems Limit the Usefulness of Environmental Data
Over the years, EPA has developed and maintained "stovepipe" data systems that are not capable of sharing the enormous amounts of data gathered. EPA now recognizes that common data definitions and formats, known as data standards, are essential to its efforts to integrate data from various databases, including those of its state partners. EPA also considers data standards as key to reducing the reporting burden on industry and the states because such standards would permit integrated, and thus more efficient, reporting of information to the agency. In recent years, EPA has undertaken several efforts to develop standards for some of the data items in its information systems. According to the Office of Environmental Information, EPA recently approved six data standards and expects that all of these standards will be implemented in the relevant data systems by fiscal year 2003.
EPA recognizes that its current data improvement efforts are only first steps toward its goal of full data integration. For example, EPA has focused primarily on the compatibility of its data with those of state environmental agencies, rather than of other federal agencies and nongovernmental sources. In a May 2000 report, we stated that improved collaboration among federal agencies in meeting the needs for human exposure data is essential because individual agencies have different capacities and skills and separate attempts have fallen short of supporting the large efforts that are needed.4 EPA's Science Advisory Board5 has also recommended that EPA do more to link the agency's databases with external databases. The Board noted that "answering many health-related questions frequently requires linking environmental data with census, cancer or birth registry data, or other data systems (such as water distribution maps) to determine whether there is a relationship between the environmental measures and health."6 EPA officials acknowledge the importance of linking EPA's databases with those of other agencies at all levels of government. However, they told us that their actions to do so have been limited by resource constraints and by the fact that EPA's statutes do not give the agency the authority to require that other agencies collect or report data using formats compatible with those used by EPA.
Concerns Persist About the Accuracy of EPA's Data
In various reviews, we and others have identified persistent concerns about the accuracy of the data in many of EPA's information systems. EPA acknowledges that data errors exist but believes that, in the aggregate, its data are of sufficient quality to support its programmatic and regulatory decisions. However, EPA has not assessed the accuracy of its information systems agencywide, and preventing errors and correcting them once they have been identified has proved daunting for the agency. For example, in January 1998, an EPA advisory council on information management issues described the difficulty of correcting errors in EPA's databases: "Once an error is stored in one or more of the agency's systems, making corrections to all those systems is an exercise in frustration and futility. There is no simple way to ensure corrections are made to all possible systems."
To address such problems, EPA revised its agencywide quality system in 1998 to expand and clarify requirements for how environmental data are collected and managed. Although the Science Advisory Board recently commended the agency for its development of this system, the Board also found that its implementation has been uneven within the agency. Moreover, the Board reported that more than 75 percent of the states authorized to implement EPA's environmental programs lack approved quality management plans for all or some of these programs and thus are likely to be generating data of unknown quality. We recently reported that EPA's National Water Quality Inventory, which EPA uses as a basis for measuring progress under the Clean Water Act, does not accurately describe water conditions nationwide. While EPA prepares the Inventory on the basis of data submitted by the states, the states do not use a statistical sampling design that provides a comprehensive picture of water quality. The Science Advisory Board has pointed out that EPA programs that rely on data of unknown quality are exposing themselves, the reliability of their decisions, and their credibility to criticisms.
Correcting errors in the agency's data is an important responsibility for the new information office. This office recently developed an Internet-based system to identify, track, and resolve errors found in national environmental databases. The system currently allows individuals to notify EPA of suspected errors in some of the agency's major databases, and EPA intends to implement the data correction system in additional databases during the next two years. Efforts to Develop Outcome-Oriented Performance Measures Are Constrained by Data Limitations
Well-chosen environmental measures inform policymakers, the public, and EPA managers about the condition of the environment and provide for assessing the potential danger posed by pollution and contamination. They also serve to monitor the extent to which EPA's programs contribute to environmental improvement and can be used in future priority-setting, planning, and budgeting decisions. EPA has been aware of the need for environmental measures since the mid-1970s. Nevertheless, the agency made little progress in developing such measures until the Results Act mandated their use by requiring federal agencies to report annually on their progress in meeting performance goals. Under the Results Act, EPA has begun to set goals and measures that are intended to help the agency, as well as the Congress and the public, assess the environmental results of the agency's activities. While EPA has made progress in adopting more measures that reflect the environmental or health outcomes of programs, the overwhelming number of EPA's measures reflect outputs, such as the number of inspections performed or regulations issued, and additional progress is needed. EPA considers getting the data needed to measure results its biggest challenge in developing outcome-oriented performance measures. To date, EPA and the states have made limited progress in developing such measures, as these examples indicate:
-- Of the 364 measures of performance that EPA has developed for use during fiscal year 2000, only 69 (19 percent) are environmental outcomes; the other measures reflect program activities, such as the number of actions taken to enforce environmental laws. (See table 2.)
-- Given inherent uncertainties about the results of research and development activities, the problem of developing outcome-oriented measures is particularly difficult for EPA's science activities. Of 36 measures related to EPA's strategic goal of "sound science," only 2 reflect outcomes.
Table 2: EPA's Analysis of the Number and Type of Annual Performance Measures for Its Strategic Goals for Fiscal Year 2000
[omitted]
Source: GAO's analysis of EPA data.
In addition to establishing output-and outcome-oriented performance measures, EPA has adopted a framework for categorizing its performance measures according to the type of outputs or outcomes to be achieved. As shown in figure 1, most of the performance measures are outputs involving either research and development efforts or actions by EPA, states, tribes, or other governmental bodies, such as establishing standards for hazardous levels of lead in paint, dust, and soil. The other categories represent outcomes, including measures that focus on risks to ecology, health, or welfare; pollutants absorbed by the body; and concentrations of pollutants in the environment. Over time, EPA plans to increase the number of such measures, as it is able to obtain better data linking its program activities with changes in environmental and health conditions.
Figure 1: Number and Percentage of Performance Measures for Each Type of Activity
[omitted]
Even with better data, it will be a major challenge for EPA to link its environmental programs and activities to outcomes. Environmental conditions may change because of a number of factors, including variables such as the weather or economic activity, many of which are beyond the control of EPA and its state partners. Likewise, it may be difficult to show the relationship between EPA's annual program activities and some outcomes that may not be apparent until many years later. For example, current EPA activities to reduce the amount of polluting nutrients from fertilizers in the ground may not result in improved water quality for a decade or more. EPA program officials recognize that they need additional measures that show the outcomes of programs, and they have recently taken actions that should strengthen the agency's ability to develop them. For example, EPA is developing processes and long-term strategies to improve the quality of performance measures and link the activities of program offices with environmental results. However, substantial resources are required to identify and test the potential measures. Once the measures are established, gathering and analyzing the data can be resource-intensive, and it can take years to show environmental improvement.
Observations
Our prior work has identified numerous problems in the quality of EPA's data and the way that the agency manages its data systems. These problems cut across the various programs regulated by EPA and have limited the agency's ability to assess risks and measure environmental results. To its credit, EPA has initiated actions to improve its information management activities. While EPA has made progress, its initiatives do not provide a long-term strategy to ensure the completeness, compatibility, and accuracy of its data. Furthermore, the initiatives have encountered obstacles that highlight the difficulties facing EPA as it attempts to improve its information management activities.
As we recommended in our September 1999 report, to substantially improve the quality of the data used to set risk-based priorities and report on progress toward improving environmental conditions and human health, EPA needs to develop a strategy that reflects a long-term commitment to resolving data problems. Such a strategy should include establishing milestones and identifying the resources necessary to fill major data gaps, identify and develop all needed data standards and implement them in key databases, and coordinate the agency's data standardization efforts with those of the states, federal agencies, and other organizations. This effort would provide both senior agency managers and the Congress with what is now missing--the information they need to make the best decisions possible on the costs, benefits, and trade-offs involved in providing scarce resources to meet critical data requirements. Although EPA concurred with our recommendation, the agency has made little progress toward developing and implementing a comprehensive strategy. For example, EPA recently informed us that it has not yet completed the first stage of a multi-phase effort to develop an information plan for the agency. EPA plans to complete the first stage by December 2000, which will identify broad options for information management over the next several years. - - - - -
Mr. Chairman, I would be happy to respond to any questions that you or other Members of the Committee may have.
Contact and Acknowledgments
For further information, please contact Peter F. Guerrero at (202) 512-6111. Individuals making key contributions to this testimony included Ed Kratzer and Cecilia Lee.
\1\ Environmental Protection Agency: Protecting Human Health and the Environment Through Improved Management (GAO/RCED-88-101, Aug. 16, 1988). 2Major Management Challenges and Program Risks: Environmental Protection Agency (GAO/OCG-99-17, Jan. 1999). 3Water Quality: Key EPA and State Decisions Limited by Inconsistent and Incomplete Data (GAO/RCED-00-54, Mar. 15, 2000). 4 Toxic Chemicals: Long-Term Coordinated Strategy Needed to Measure Exposures in Humans (GAO/HEHS-00-80, May 2, 2000). 5The EPA Science Advisory Board was created by the Congress to provide advice to EPA from scientists outside the agency. 6 Science Advisory Board, Review of the Agency-Wide Quality Management Program, EPA-SAB-EEC-LTR-98-003 (Washington, D.C.: EPA, July 24, 1998).