Skip Navigation
| Skip Navigation |
|
|
|
|
|
Since publishing its first macroeconomic forecast in 1976, the Congressional Budget Office (CBO) has compiled a forecasting track record comparable in quality with those of a sizable sample of private-sector forecasters
as well as five Administrations. CBO's errors for two-year forecasts made
between 1982 and 2000 did not differ markedly either from those of the
Administration or from the average of the 50 or so forecasts that have
made up the Blue Chip survey over the years. Comparing CBO's forecasts
with those of the Blue Chip consensus suggests that when CBO's economic
predictions missed the mark by a margin wide enough to contribute to sizable
errors of the surplus or deficit, those inaccuracies probably reflected
limitations that confronted all forecasters. That result is not surprising
because all forecasters, when making their predictions, have the same basic
information available about the state of the economy, which they may then
interpret differently. Moreover, CBO examines other forecasts when constructing
its own; in turn, CBO's forecast may affect others in a similar way.
OverviewFor forecasts looking two years ahead--those that are most important for
the budget year being considered by the Congress--CBO's accuracy is essentially
the same as that of the various Administrations over the past two and a
half decades. Even though CBO and the Administration usually make different
policy assumptions--CBO assumes a continuation of current policy and the
Administration normally incorporates the projected economic effects of
its policy proposals--the differences between the two forecasters are small,
especially when compared with the size of the forecast errors. Also, CBO's
forecasts have been about as accurate as the Blue Chip's average
forecasts, in general (see Table 1).
For longer-run budget planning, the accuracy of five-year forecasting is important. CBO's inaccuracies in five-year projections of growth rates for real (inflation-adjusted) and nominal (current-dollar) output were similar to those of the various Administrations and the Blue Chip consensus. CBO's forecasts of real growth over the long run have alternated between periods of optimism and pessimism. The five-year forecasts produced during the late 1970s turned out to be too optimistic, averaging about 3 percentage points a year too high. Forecasts from the early 1980s, in contrast, were too pessimistic by a little less than half a percentage point. Forecasts from the late 1980s were overly optimistic again, but the projections made between 1992 and 1997 have been too pessimistic by more than a percentage point, on average. (The five-year forecast from January 1997 is the most recent one that can be compared with actual results.) As the track record shows, forecasters collectively tend to err during periods that include either turning points in the business cycle or significant shifts in major economic trends. For example, most forecasters overestimated the economy's growth rate and underestimated inflation in forecasts they made just before and during the back-to-back recessions of the early 1980s. That pattern was repeated, albeit to a lesser degree, in the forecasts they made just prior to the more moderate recession of the early 1990s. In addition, during the middle to late 1970s, forecasters continued to assume that the productivity trend of the previous two decades would prevail. In retrospect, however, the productivity trend of the 1970s and 1980s was significantly lower than that of the 1950s and 1960s. Because forecasters in the 1970s felt that the previous trend would recur, their forecasts of real output in the middle to late 1970s turned out to be too optimistic. The years from 1995 to 2000 were a mirror image of the forecasting experience of the late 1970s. Partly because forecasters underestimated the trend rate of productivity growth since 1996, they underpredicted the economy's growth rate and overpredicted inflation. To be sure, as the economy continued to outperform expectations, analysts put more effort into investigating the possible causes of the increase in productivity growth. Those investigations focused on the possible contribution of the so-called new economy--especially the better flow of information among producers and between producers and consumers, which has permitted improved productivity, lower inventories, and greater customer satisfaction. However, it is too soon to draw unambiguous conclusions about the role of the new economy from the historical evidence. Because the reasons for the phenomenal economic performance of those five years are not yet fully understood, the uncertainty about the next few years may be larger than indicated by the average errors of the past two decades. CBO also underestimated taxable income in the forecasts it made in 1995
through 1999, which contributed to revenue projections far below what actually
occurred. However, underestimates of real growth account for only part
of those underpredictions. Projections of taxable income were pessimistic
not only because projections of nominal output were too low, but also because
the projected relationship of taxable income to output--the taxable-income
share--was too small. Income categories in the national income and product
accounts that are used to project revenues--primarily book profits, and
wages and salaries--grew much more rapidly between 1995 and 2000 than did
nominal output, but CBO's forecasts failed to anticipate that increase
in the taxable-income share of output. In contrast, the most recent CBO
forecast presented in this analysis--the January 2000 forecast--was overly
optimistic about the growth of income. That contributed to CBO's overestimate
of revenues for 2000 and 2001.
Sources of Data for the EvaluationEvaluating CBO's forecasting record requires compiling the basic historical and forecast data for growth in nominal and real output, inflation in the consumer price index (CPI), interest rates, and taxable income. Although each of those data series has an important influence on budget projections, an accurate forecast of the two-year average growth in real output is the critical economic factor in accurately estimating the surplus or deficit for the upcoming budget year. The data were compiled using forecasts published early in the years from 1976 through 2000. (Two-year average forecasts published in early 2001 and 2002 could not be included in this evaluation because actual values for 2002 were not available in time to be included in this document.) Selection of Historical Data
Growth in Real and Nominal Output. Historical two-year averages of growth in real output were developed from calendar year averages of the quarterly chain-type annual-weighted indexes of real gross national product (GNP) and real gross domestic product (GDP) published by the Bureau of Economic Analysis (BEA). The fact that several real GNP and GDP series were discontinued because of periodic benchmark revisions meant that they were unsuitable historical series. For example, during the 1976-1985 period, the three forecasters published estimates for a measure of growth in real GNP that was based on 1972 prices, which was the measure published by BEA at that time. In late 1985, however, BEA discontinued the 1972-dollar series and began to publish GNP on a 1982-dollar basis. As a result, an official series of values for GNP growth in 1972 dollars is not available for the years after 1984, and actual two-year average growth rates are not available to compare with the forecasts made in early 1984 and 1985. From 1986 to 1991, forecasters published estimates of growth in real GNP based on 1982 prices. BEA revised the benchmark again in the second half of 1991: it discontinued the 1982-dollar GNP and began to publish GNP on a 1987-dollar basis.(2) Consequently, the historical annual series for 1982-dollar GNP is available only through 1990, and actual two-year average growth rates are not available for the forecasts made in early 1990 and 1991. The forecasters then published estimates of growth in real GDP on a 1987-dollar basis until 1995, when BEA made another switch, late in the year, to a chain-weighted measure of GDP. Therefore, the historical annual series for 1987-dollar GDP ends with the 1994 annual value, and actual two-year average growth rates are not available for the forecasts made in early 1994 and 1995. By periodically updating the series to reflect more recent prices, BEA's benchmark revisions yield a measure of real output that is more relevant for analyzing contemporary movements in real growth. But the process makes it difficult to evaluate forecasts of real growth produced over a period of years in series that are later discontinued. This comparison avoids the difficulties presented by periodic revisions of the data by using BEA's chain-type annual-weighted index of real GNP or GDP throughout the data series.(3) Historical two-year averages for growth of nominal GNP and GDP were developed from calendar year averages of the quarterly values published by BEA. CPI Inflation. CBO calculated two-year averages of inflation in the consumer price index from calendar year averages of monthly data published by the Bureau of Labor Statistics. Before 1978, the bureau published only one consumer price index series, now known as the CPI-W (the price index for urban wage earners and clerical workers). In January 1978, however, the bureau began to publish a second, broader consumer price index series, the CPI-U (the price index for all urban consumers). CBO's comparison of forecasts uses both series. Until 1992, the Administration published its forecasts for the CPI-W, the measure used to index most of the federal government's spending for entitlement programs. In contrast, for all but four of its forecasts since 1979 (1986 through 1989), CBO based its inflation forecast on the CPI-U, a more widely cited measure of inflation and the one now used to index federal income tax brackets. The Blue Chip consensus has always published its forecast for the CPI-U. Although annual fluctuations in the CPI-U and CPI-W are virtually indistinguishable, the indexes differ in some years. For that reason, CBO used historical data for both series to evaluate the alternative forecasting records. Interest Rates. CBO used monthly data published by the Board of Governors of the Federal Reserve System to calculate two-year averages of nominal short- and long-term interest rates. The forecasts of short-term interest rates were compared using historical values for two measures of the interest rate on three-month Treasury bills: the new-issue rate and the secondary-market rate. The Administration forecasts the new-issue rate, which corresponds to the price of three-month bills auctioned by the Treasury Department--that is, it reflects the interest actually paid on that debt. CBO forecasts the secondary-market rate, which corresponds to the price of the three-month bills traded outside the Treasury auctions. Such transactions occur continually in markets that involve many more traders than do Treasury auctions. As a result, the secondary-market rate provides an updated evaluation of short-term federal debt by the wider financial community. Blue Chip has alternated between those two rates; it published the new-issue rate from 1982 to 1985, switched to the secondary-market rate during the 1986-1991 period, and then returned to the new-issue rate beginning in 1992. Clearly, there is no reason to expect the two rates to differ persistently; indeed, the differences between their calendar year averages are minuscule. CBO likewise compared the various forecasts of long-term interest rates using historical values for two measures of long-term rates: the 10-year Treasury note rate and Moody's Aaa corporate bond rate. A comparison of forecasts is not possible before 1984 because not all of the forecasters published projections of long-term interest rates before then. For forecasts made in early 1984 and 1985, CBO projected the Aaa corporate bond rate. Beginning with its early 1986 forecast, however, CBO switched to the 10-year Treasury note rate. The Administration has always published its projection for the 10-year Treasury note rate, but Blue Chip published the Aaa corporate bond rate until January 1996, when it switched to the 10-year Treasury rate. CBO calculated separate historical values for real short-term interest rates using the nominal short-term interest rate and the inflation rate appropriate for each forecaster. In each case, the two-year average nominal short-term interest rate was discounted by the two-year average rate of inflation. The resulting real short-term interest rates were similar. Taxable Income. Through its direct influence on projections of federal revenues, the forecast for taxable income plays a critical role in determining the accuracy of budget projections. The income measure examined here--wage and salary distributions plus the book value of corporate profits--combines the two sources of income to which tax receipts are most sensitive. Considering wages and profits together is appropriate because the effective rates of taxation on wages (including payroll and income taxes) and on corporate profits are nearly the same. In addition, those tax rates exceed the rate at which other income sources (such as interest income) are taxed. Although the level of taxable income is the factor that most directly affects federal revenues, historical estimates of that level are subject to substantial statistical revision. As a result, using the level of taxable income would distort the comparison of forecasts. Instead, the forecasts are presented here as changes in taxable income as a share of total output; the historical revisions, carried forward consistently to projections, should not affect projections of revenues. Moreover, the change in taxable income as a share of total output is closer to the concept that macroeconomists consider when they construct their forecasts. Sources of Forecast Data
The average two-year forecasts in the Blue Chip consensus survey, which are published monthly, were taken from those published in the same month as CBO's forecasts. Because the Blue Chip consensus did not begin publishing its two-year forecasts until the middle of 1981, the first such forecast available for this comparison was published in early 1982. Average five-year projections, however, are published by Blue Chip only two or three times a year. All but one of its five-year projections used in this evaluation were published in March; the 1980-1984 projection of real output was published in May. Some of the CBO forecasts for wages, salaries, and corporate profits
that are used here were not published in CBO's annual reports. Instead,
they were taken from CBO's files of unpublished forecasts. CBO has published
forecasts for wages and salaries regularly since 1985 but has published
forecasts for book profits only in recent reports.
Measuring the Quality of ForecastsLike earlier studies of economic forecasts, this evaluation focused on two aspects of the quality of CBO's forecasts: statistical bias and accuracy. Other desirable characteristics--such as the efficiency of a forecast, which is discussed later--are harder to assess definitively and would require a larger sample than is available for CBO's forecasts. Bias
Accuracy
Other Measures of Forecast Quality
A number of other methods have been developed to evaluate a forecast's "efficiency." Efficiency indicates the extent to which a particular forecast could have been improved by using additional information that was available to the forecaster when the forecast was made.(6) The Blue Chip consensus forecasts represent a wide variety of economic forecasters and thus reflect a broader blend of sources and methods than can be expected from any single forecaster. In this evaluation, the Blue Chip predictions can therefore serve as a proxy for an efficient forecast. The fact that CBO's forecasts are about as accurate as the Blue Chip's is a rough indication of their efficiency. Such elaborate measures and methods, however, are not necessarily reliable indicators of a forecast's quality when the sample of observations is small, as is the case with CBO's sample of only 25 two-year observations. Small samples present three main problems in evaluating forecasts. First, they reduce the reliability of statistical tests that are based on the assumption that the underlying population of errors in the forecast follows a normal distribution. The more elaborate measures of forecast quality all make such an assumption about the hypothetical ideal forecast with which the actual forecasts are being compared. Second, in small samples, individual errors in a forecast can have an unduly large influence on the measures. The mean error, for example, can fluctuate in its arithmetic sign when a single observation is added to a small sample. Third, the small sample means that CBO's track record cannot be used in a statistically reliable way to indicate either the direction or the size of forecasting inaccuracies in the future. Apart from the general caveat that should attend any statistical conclusions,
there are several reasons for viewing any evaluation of CBO's forecasts
with particular caution. First, the procedures and purposes of CBO's and
the Administration's economic forecasts have changed over the past two
decades and may change again. For example, in the late 1970s, CBO characterized
its long-term projections as a goal for the economy; it now considers them
to be a projection of what will prevail, on average, if the economy continues
to reflect historical trends. Unlike CBO's projections, the Administration's
normally included the projected economic effects of its own policy proposals.
Second, an institution's track record in forecasting may not indicate its
future abilities because of changes in personnel or methods. Third, inaccuracies
in a forecast increase when the economy is more volatile and when economic
trends change. All three groups of forecasters--CBO, the Administration,
and the Blue Chip survey--made exceptionally large errors when forecasting
for periods that included turning points in the business cycle and for
the late 1990s, when the sustainable growth of the economy apparently increased.
CBO's Forecasting RecordThis analysis evaluates the Congressional Budget Office's macroeconomic forecasts over two-year and five-year periods. Because the budget reports that the Administration and CBO publish each winter focus on budget projections for the fiscal year that begins in the following October, an economic forecast that is accurate not only for the months leading up to that budget year but also for the budget year itself will provide the basis for a more accurate forecast of the budget's bottom line--hence the interest in the two-year period. The five-year period is used to examine the accuracy of longer-term projections of growth in real and nominal output. This analysis does not consider how assumptions about key economic factors can affect federal budget projections. "Rules of thumb" for estimating the effects of changes in various macroeconomic variables appear in Appendix A of CBO's The Budget and Economic Outlook: Fiscal Years 2003-2012 (January 2002). Two-Year Forecasts
Growth in Real Output. There is
little difference in forecast accuracy between CBO and the Administration
for the two-year forecasts made between 1976 and 2000 (see Table
2). CBO was closer to the actual value in 11 of the 25 forecasts made
between 1976 and 2000, the Administration was closer in 10 periods, and
the two had identical errors in four periods. CBO's predictions of real
growth made between 1982 and 2000 were, on average, as accurate as those
of the Blue Chip consensus.
As noted earlier, forecast errors tend to be larger during periods of economic turmoil or upheaval. That tendency can be clearly seen in the forecasts of real output growth by comparing the large errors for 1979 through 1983--when the economy went through its most turbulent recessionary period of the postwar era--with the smaller errors recorded for later years. Similarly, the business cycle accounts for the large errors in the predictions made in the 1989-1991 period. During that time, the Congressional Budget Office's errors were only slightly larger than those of the Blue Chip consensus. All three forecasters underpredicted two-year real GDP growth in every year between 1992 and 1999. Much of that apparent pessimism, however, resulted from revisions in the national income and product accounts; the BEA benchmark revisions published in November 2000 increased the two-year growth rates for real GDP over most of the historical period, especially the past two decades. The upward revision in growth rates stemmed largely from including software spending as investment in the accounts as well as adopting new price series for various categories of consumption. In addition to making the mean forecast error less informative, those revisions distort the reliability of the statistical measures of accuracy. Nevertheless, even after accounting for the latest revisions, the underpredictions of economic growth since 1996 are significant. Growth in Nominal Output. The records
of CBO and the Administration in forecasting two-year growth in nominal
output are also quite similar (see Table 3). Of the 25 forecasts made between 1976 and 2000, the Administration recorded the smaller error 13 times, CBO had the smaller error 10 times, and the two forecasters recorded identical errors twice. CBO's mean error and root mean square error for that period
are similar to the Administration's.
Over the shorter interval between 1982 and 2000, the bias and accuracy of CBO's forecasts of two-year growth in nominal output are nearly identical to those of the Blue Chip consensus. CPI Inflation. CBO and the Administration
had very similar records in forecasting the two-year average growth in
the consumer price index as well (see Table 4). Both underestimated future inflation in their forecasts for 1977 through 1980 and overestimated it for 1981 through 1986. Their average measures of bias and accuracy were virtually identical. CBO was closer to the actual value in eight of the 25 periods, the Administration was closer in 11 periods, and the two forecasters had matching errors in six periods. For the 1982-2000 period, CBO's forecasts of inflation were as accurate as those of the Administration and the Blue Chip consensus.
Nominal Interest Rates. For the
1976-2000 forecasts, CBO's record was about as accurate as the Administration's
for nominal short-term interest rates over a two-year period (see Table 5). On average, the Administration tended to underestimate those rates, whereas CBO's mean error was zero over that period.
The Administration was closer to the actual value in 12 of the 25 periods, and the two forecasters had identical errors twice. Between 1982 and 2000, however, the root mean square error of CBO's forecasts was slightly above those of the Administration and the Blue Chip consensus, meaning that CBO made a few relatively large errors (such as those in 1982, 1983, and 1991). For the 1984-2000 forecasts of long-term interest rates, CBO did somewhat
better than the Administration (see Table 6). The
Administration tended to underestimate rates, and its mean error was slightly
larger than CBO's. In addition, the Administration's forecasts were less
accurate, on average, than CBO's. CBO was closer to the true value in 10
of the 17 periods, the Administration was closer in six periods, and the
two forecasters had identical errors in one period.
The Congressional Budget Office's forecasts of long-term interest rates were about as accurate as those of the Blue Chip consensus. Both CBO and Blue Chip tended to overestimate long-term rates for 1982-2000. CBO had a mean error of 0.2 percentage points compared with 0.3 percentage points for Blue Chip. Real Short-Term Interest Rates.
For the forecasts made in 1976 through 2000, CBO had an edge over the Administration
in estimating short-term interest rates adjusted for inflation (see Table 7). Again, the Administration was more likely than CBO to underestimate interest rates. Both forecasters recorded similar mean absolute and root mean square errors. CBO's forecasts were closer to the actual value in 13 of the 25 periods, the Administration's were closer in 10, and the two registered identical errors in two periods. For forecasts made between 1982 and 2000, CBO's errors were generally similar in both direction and magnitude to those of the Blue Chip consensus.
Taxable Income. One of the greatest
sources of error in budget projections is in forecasting taxable income.
On average, both CBO and the Administration have been too optimistic in
their projections of the two major components of taxable income--wages
and salaries, and corporate profits (see Table 8).
(The Blue Chip consensus does not forecast wages and salaries.)
Since 1994, however, both CBO and the Administration have significantly
underestimated the growth of wages and profits relative to output. Apart
from the usual difficulties associated with forecasting GDP, two other
factors contribute to the errors.
The first is the degree to which total income has exceeded total product
in the national income and product accounts (NIPAs). In principle, those
two aggregate measures of economic activity should be equal, but in practice
they are not, largely because the Bureau of Economic Analysis, which publishes
the NIPAs, must use different primary sources to estimate total income,
on the one hand, and total product, on the other. The statistical discrepancy
in the NIPAs measures the difference between total product and total income;
between 1997 and 2000, the excess of total income over total product grew
(see Figure 1).
The change in the discrepancy presents a problem for forecasters. If they have assumed, in line with historical experience, that the discrepancy will revert toward zero and that it mainly results from mismeasurements on the income side, they will have been more apt to understate income in recent years. At this point, it is impossible to tell exactly how much the discrepancy has caused forecasters to err in their forecasts of income, but the sheer size of the imbalance in recent years compounds the importance of each forecaster's assumptions about how to predict the discrepancy. Forecasters' use of alternative assumptions to resolve that imbalance could broaden the dispersion of forecasts of total income in coming years. A second source of difficulty in forecasting taxable income as a share
of output was the reversal in the last half of the 1990s of another long-standing
trend. Throughout the post-World War II period, nonwage labor income rose
as a share of total labor compensation, a trend that many analysts believe
reflects an increased tendency by employers to substitute fringe benefits
(such as employer-paid insurance premiums and pension contributions) for
wages and salaries as a means of compensating their workforce. But, between
1994 and 2000, that trend reversed (see Figure 2). Because nonwage labor income is not taxed, the decline in its share of total labor compensation has had the effect of increasing the share of compensation that is taxed (namely, wages and salaries). That turnaround was relatively unpredictable
and as yet is not fully understood. However, nonwage labor income is no longer growing more slowly than wages. Thus, the trend in nonwage labor income is not likely to decline in the future.
Five-Year Projections
Neither the Administration nor CBO, however, considers its projections to be its best guess about the year-to-year course of the economy. The Administration's annual projections are based on the adoption of the President's budget as submitted, and for most years, CBO has considered its projections an indication of the average future performance of the economy if major historical trends continue. Neither institution attempts to anticipate cyclical fluctuations within the projection period. CBO's projections of longer-term growth in real output were closer to the actual values than the Administration's were in 13 of the 22 forecasts. The average errors for 1976-1997 indicate that the Administration's projections had an upward bias of 0.4 percentage points compared with an upward bias of 0.1 percentage points for CBO. Those biases occurred largely because of the projections made in early 1976 through 1979. Those projections were too optimistic, in part because CBO and the Administration presented their projections at that time as target rates of growth (not forecasts) and because real growth was extraordinarily low during the back-to-back recessions of 1980 and 1982. Through the subsequent years of expansion until the 1990-1991 recession, the upward bias was much smaller in CBO's projections. In the five-year-ahead projections made between 1992 and 1997, both CBO and the Administration underpredicted long-term growth. The reasons were the surprisingly strong economy of the late 1990s and, to a lesser extent, the upward revisions to BEA's estimates of the rate of growth. Those underpredictions of real growth also resulted in underestimates
of long-run growth in nominal output, but the errors for nominal growth
in the 1990s were smaller than those for real growth (see Table
10). That is because most forecasters have overestimated inflation
in recent years. Between 1976 and 1997, CBO and the Administration appear
to have done equally well in forecasting five-year growth in nominal output.
Moreover, as the record since 1982 shows, both forecasters compared well
with the Blue Chip consensus.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
