Review of NEMS Forecast Evaluation Methodology
By Julian Silk and Robert P. Trost
We are pleased to see that EIA is making a serious
attempt to track, tabulate, and analyze the energy price and quantity forecast
errors from NEMS. We are even more pleased to see that EIA has commissioned the
well-known and well-respected energy economist George Lady to begin the initial
steps to complete this task.
Theoretically, the task of computing the inherent
energy price and quantity forecast errors from the NEMS model is
straightforward. When a forecast period actually occurs and the actual values of the
exogenous variables[1]
are known, the model can be re-run using the actual values for exogenous
variables and the resulting energy price and quantity projections compared to
the original projections based on assumed values for the exogenous variables.
The collective impact of using the correct exogenous variables can be
determined by comparing the projections from the original projections to the
projections made using the realized values of the exogenous variables.
Presumably, using the actual values of the exogenous variables will produce a
more accurate forecast than the forecast based on values of the exogenous
variables assumed to hold at the time of the original forecast. The remaining
errors forecast errors in energy prices and quantities when the actual values
of the exogenous variables are used can be collectively attributed to errors in
the structure of the NEMS model, errors in the choice of NEMS model parameters,
and the stochastic nature of a real world model as opposed to the deterministic
nature of the NEMS model. Routines within the model might also be amended to
account for changes in government policies, technologies, or consumer behavior.
Although the size of the model and need to maintain and then run non-current
versions of the model over many years may make this approach impractical, we
would encourage EIA to pursue this approach anyway. In any event, that approach
has not been pursued in the current methodology proposed by Lady.
The approach proposed by Lady is to estimate the market
sensitivities implicit to NEMS based on the NEMS solution output prepared each
year for the AEO. This method, which replicates forecast results from large
computer process models without re-running the actual model, is described and
applied in Buck and Lady (2005). The approach is not new, and was pioneered by
The pseudo data approach to approximating output from large computer based process models is not without its critics, the most noted one coming from the late G. S. Maddala in Maddala and Roberts (1980), quoted below:
“Given
that the generation of pseudo data involves solving a linear programming
problem that is the minimization (or maximization) of an objective function
subject to some constraints, the data generated will depend on how the
optimization problem is set up and what constraints are imposed. It is
therefore, fair to say, that the estimated function ( cost function, revenue
function, profit function, or whatever), from these data cannot be taken to be
giving a summary description of the complex technology of the process model. It
is just a summary description of the particular set of data generated. If we
change the structure of the problem, or the constraints, or the set of price
vectors chosen, the same process model will give a different summary description.”
In fact, Lady seems aware of this pitfall when he writes: “Many issues, such as the choice of estimation method or kernel function, the specification of the included variables in the regression equations, the experimental design (if any) of the solution sets used as data, remain open and provide an interesting agenda for further efforts in developing approximation approaches.”
While we agree in principle with Maddala’s overall assessment of this approach, we also appreciate the difficulty of re-running prior NEMS forecasts and computing forecast errors with actual values of the exogenous variables and policies plugged into the prior NEMS runs. But we recommend that EIA pursue this ideal, even if it only means re-running the current NEMS model for the prior period, using now known values of the exogenous and policy variables. In the meantime, EIA should purse the pseudo data approach for computing and analyzing NEMS forecast errors as proposed by Lady.
Below
we offer some specific suggestions for improving this general method. It will
be assumed that there are no changes possible to the substantive content
discussed in the document before presentation.
All comments below will relate to the description of what is being
done. Should it be possible to redo
work, some of these comments might be addressed substantively, instead of just
changing description.
The
nature of the writing in the document is not being evaluated in this review; it
is assumed that the writing is suitable for an informal presentation. For publishable work, the writing should be
substantially revised and rigorously reviewed.
Some
specific comments about the method follow.
1.
Partitioning of Forecast Errors into its Components.
In the methodology
section, Lady states: “Based on this, “differences”
between NEMS projections and the actual values of the variables projected can
be partitioned among general uncertainty, “errors” in projecting explanatory
variables, structural changes in market behavior, and transitory influences
such as the weather.”
He further states:
“It is proposed to configure the means of
performing the differential analyses, as based upon specialized NEMS runs or
the regression analysis of NEMS solutions,
in a fashion that can be routinely conducted and maintained by EIA
staff. The goal of the statistical analysis is to enable the errors in EIA
forecasts to be explicitly decomposed with respect to influences such as the
following:
Transitory Influences, e.g., weather, strikes, accidents, embargoes
not accounted for in the projections.(* for weather)
Institutional
Influences, e.g., changes in laws and regulations and changes in data
series definitions compared to model assumptions.
Structural
Influences, e.g., changes in resource availability or energy use technology
compared to model assumptions.*
Errors
in Projecting Conditional Variables, e.g., differences in the eventual
values of activity drivers and other exogenous factors such as GDP and
population.*
Errors
in Behavioral Parameters, e.g., changes in consumer price sensitivities
compared to those assumed by the forecasting methodology.*
Uncertainty,
e.g., the residual error of the projection method.*
The
methodology for partitioning forecast differences among (such as) the
influences outlined above is as follows for the items indicated by “*”, given
the availability of actual data for previously forecast values. “
While this partitioning of forecast errors may be possible in the “pseudo data model”, because of the intricate interactions between exogenous and policy variables in the actual NEMS model, we find it very ambitious to try to partition forecast errors into various sources, other than simultaneously netting out the impact of assumed versus known exogenous variables and policies.
In addition, the pseudo data model only approximates what the NEMS model would forecast, so an additional “uncertainty” is added to the forecast when one uses the pseudo data approach versus actually re-running the NEMS model with the now known values of exogenous variables inputted into the run. That is, the pseudo data model may forecast an error that would not occur in an actual re-running of the NEMS model.
2. Computing
Price Elasticities
Computing implied energy price elasticities in NEMS is a very useful exercise itself., and we encourage any attempt to do so. However, since price is endogenous to NEMS, using these price elasticities to approximate and analyze NEMS forecast errors is not meaningful. For example, Lady states:
“The
approach noted above would be to solve NEMS components, relative to a base or
reference case, with each important assumption, e.g., the residential sector
price of energy, changed, individually (with all other assumptions held
constant), and compare results.”
Since
price is endogenous within NEMS., there are no explicit assumptions about “the
residential price of energy” Indeed, the forecasted energy prices are not known
until the NEMS model is run. We recommend that for purposes for
approximating the NEMS forecast with elasticities, only the elasticities of the
eight exogenous variables cited in Buck and Lady (2005) be computed.
But
we reiterate our earlier point that knowing the implicit energy prices in NEMS
is valuable information. When computing energy price elasticities we make the
following recommendations:
- To the extent possible,
compute these elasticities analytically from the NEMS code. These can be
used as check against the elasticities estimated based on data from NEMS
runs.
- Use some kind of
simultaneous equations estimation method to estimate these elasticities
with data from NEMS runs.
- Note that estimated
elasticities from data on NEMS runs may include some impacts not
traditionally associated with end-user elasticities. For example, a high
WOP will result in high energy prices, which will impact the technology
choices NEMS brings online and made available to end-users. Since these
new technologies will be more energy efficient, there will be a decline in
energy use as result of the new
technologies brought online. In traditional models of end-user price
elasticities, the only impact price has on consumer energy use is the
short-run impact of “turning down the thermostat” and the longer run
aspect of “buying more energy efficient capital equipment” from the
choices available. The impact energy price has on actually changing the
choice of this capital is not usually considered a part of end user price
elasticity.
- Some other problems
with the use of price elasticities are implicit assumptions of linearity
and parallel shifts. Consider the
following graph, which shows two supply-demand equilibria.
If
we calculate price elasticities, for either demand or supply or both, from
“supply_q1” and “demand_1”, our elasticities will be relatively low. Both supply and demand are relatively
inelastic for this case.
But
supply and demand can both move, in the real world and inside the model. If there are parallel shifts of both supply
and demand, the elasticities obtained from evaluation of the first case are
useful in discussing what happens in the second. On the other hand, in the case above, both
demand and supply become more elastic, and also become more nonlinear. So any elasticity estimated from the first
case would quickly become misleading in evaluating the second.
Some
idea of what assumptions are necessary for the use of price elasticities to
make sense in the evaluation would therefore be highly valuable.
3. Point out Past Work
The
document does not give any history of the evaluation of EIA forecasts. Admittedly, there has not been much work in
this area in the immediate past.
However, one example of work from the more distant past that might be
quoted is “DOE/EIA-0292(85), “An Assessment of the Quality of Selected EIA Data
Series – Energy Consumption Data” (Washington, DC: Energy Information
Administration, 1986).
4. Market Forecasts vs. Forecasts of Policy Changes
The
summary statement should make clear which type of accuracy for NEMS is being
evaluated. There are at least two
possible choices. NEMS could be used to
forecast market results (such as the market price of petroleum, or how many
trillion cubic feet of natural gas are produced in
5. Relatively Simple Models of NEMS Behavior
The
models used as base cases to derive the elasticities, e.g.,
Qt
= A + B(Sector Price)t + C(Driver)t + DQt-1,
as
noted on p. 12, are awfully simple. This
may be necessary for ease of explanation in the setting to which the document
applies. But if the intent is to use
modeling techniques to capture what NEMS is doing, the document ought to
explain that a more serious effort will be made to model the NEMS equations
than just these sample equations.
Specify whether the changes
in the elasticities come from shifts in equations or measurement of one
equation at different locations
Presumably
the arc and constant elasticities reported in the appendices are price
elasticities. This should be made
absolutely clear.
But
in addition, it should be specified in each case whether we are looking at
different locations for one equation, or whether the equations themselves have
shifted. Suppose we consider the graph
above, and hold demand constant at “demand_1”.
Supply shifts over the course of time from “supply_q1” to
“supply_q2”. We will get one elasticity
measure near the first intersection of supply and demand, and another near the
second intersection of supply and demand.
This is very different from what we would get comparing the first
equilibrium with the second equilibrium when both supply and demand shift.
The documentation stresses that “Driver” for the residential
and commercial sectors are millions of households and billions of square feet,
respectively (p. 13). So we are
evaluating a demand equation in essence, each time. But it may be the case that there are shifts
in supply to the residential and demand sectors, and forecast errors (and
miss-specification) of these supply equations could be part of the cause of our
errors. Some evaluation of these supply
functions (or at least
a note stating that this work will be done and it is currently assumed that
these functions have not shift) is warranted
(or
at least a note stating that this work will be done and it is currently assumed
that these functions have not shifted) is warranted.
6. Point out that metric
tons are 1000s of kilograms, not 1000s of pounds
On
p. 61, it is pointed out that the metric tons match history. There have been a number of problems with the
forecasts of carbon dioxide at NETL, and part of these problems had to do with
the units in which the carbon dioxide was reported. People tend to assume that metric tons and
standard tons are similar, but this is not the case. 1000 kilograms = 2200 pounds, which is not
close to 1000 pounds. The document
should make this clear as well.
7. Preliminary versus
revised data.
The
methodology needs to compute forecast errors with preliminary data and also
with revised data. If you have
preliminary data for 2007 in January 2008, and then revise and the final data
in July 2008, probably you should do 2 NEMS runs (or elasticity estimates), one
each time, to get the effect of the revision on the elasticity and sort that
impact out from misspecification.
Summary:
This
document discusses a valuable project that is decades overdue: the evaluation
of the forecast accuracy of NEMS models.
The procedure of using regression analysis to approximate NEMS results
is a reasonable basis for future work.
Such work is essential to more thorough use of NEMS, and to increased
credibility and confidence in NEMS results.
We recommend that eventually EIA track the forecast errors of NEMS in the usual way by re-running the NEMS model with known values of exogenous variables and policies inputted; even if it is just re-running the current model for the prior period, using now known values of the exogenous and policy variables.
When implementing the Lady method for approximating NEMS forecast errors, just use the elasticities for the eight truly exogenous variables.
Recognize that some of the forecast error from the Lady model may be inherent to the approximation and not to the actual NEMS model.
Computing the implied energy price elasticities is valuable information, but it should not be used to approximate NEMS forecast errors since price is endogenous in NEMS. Also, estimate more realistic models of energy supply and demand, and use some kind of simultaneous equations estimation method to estimate these elasticities.
To the extent possible, analytically compute all elasticities from the NEMS model documentation and computer code, and compare the analytical computations to those estimated with NEMS output data.
References
Buck, Andrew, and George Lady, “Approximations of
Large Computer-Based Economic Models”, Working paper, Department of Economics,
School of Business and Management, Temple University, 2005.
Maddala,
G. S. and Blaine Roberts, “Alternative Functional Forms and Errors of Pseudo
Data Estimation” The Review of Economics and Statistics, 62, No. 2, May
1980, pp. 323-327.