Sensitivity Analysis of EIA Forecasting Systems

Disclaimer

This is a working document prepared under Contract 01-08EI38172.A003 in order to solicit advice and comment on statistical matters from the American Statistical Association Committee on Energy Statistics. This topic will be discussed at EIA's fall 2008 meeting with the Committee to be held October 24, 2008.

Abstract

This paper reports upon sensitivity analyses conducted with the National Energy Modeling System (NEMS) and the Regional Short Term Energy Model (RSTEM). Sensitivities were developed for NEMS consuming sector modules (residential, commercial, industrial, and transportation) using a regression analysis for data extracted from NEMS solutions for which energy prices and activity drivers were systematically varied. These results were reported to the ASA Committee in October 2007. For this report, sensitivities to weather are also investigated. Based upon the regressions and weather variable analyses, error attribution to important influences was conducted for the AEO1998-AEO2003 versions of NEMS with respect to projections for the year 2006 for residential and commercial sector electricity and natural gas consumption. A sensitivity analysis was conducted at the census region level of detail for price, driver, and weather variables for RSTEM forecasting equations for residential and commercial sector electricity and natural gas consumption. Similar sensitivities for were found for NEMS at the census region level of detail. These results are reported here for comparative purposes.

I. Background. This paper is one in a series of reports presented to the ASA Committee on a project to develop methods for evaluating EIA forecasts of energy production, consumption, and other projected attributes of the U.S energy system. Prior reports concentrated on projections made for the Annual Energy Outlook(AEO) based upon the National Energy Modeling System (NEMS). Sensitivity analysis results for NEMS were reported to the ASA Committee in October 2007. For the AEO2007 version of NEMS, scenarios were formulated and run with fuel prices and activity drivers systematically changed for selected fuels for the residential, commercial, industrial, and transportation sectors. Subsequently, as reported on here, weather scenarios were formulated for increases in heating and cooling fuel demands. These results tabulated here for residential and commercial sector electricity and natural gas consumption. Then, for demonstration purposes, the regression results and weather analyses were used to assess the differences between the NEMS reference case forecasts and actual outcomes for the AEO1998-AEO2003 versions of NEMS with respect to projections for the year 2006 for residential and commercial sector electricity and natural gas consumption. The differences between actual and projected consumption are partitioned with respect to the degree to which they can be explained by differences between actual and assumed values for fuel prices, driver(s), and weather variables. The results are derived at the national level of regional detail. These results are presented in Section II.

A similar sensitivity analysis was performed for the RSTEM forecasting equations for residential and commercial sector electricity and natural gas consumption. For RSTEM, fuel demands are expressed by explicit, regression-based functions derived at the census level of regional detail. Depending upon the fuel and sector, these functions embody price(s), activity driver(s), and weather variables. Sensitivities were derived by changing the explanatory variables individually, running the forecasting equation with the edited data as input, and comparing results with the reference case projections. These results are presented in Section III.

The NEMS scenarios run in support of the sensitivity analysis conducted at the national level of geographic detail also report the price and quantity data at the census region level of detail. Weather variables input separately to the model are also reported at the census region level of detail. For the activity drivers “total residential households” and “total commercial floorspace,” regional detail is not provided in the NEMS solution file typically accessed by the project from which NEMS solution data are selectively extracted for further analysis. These data are reported in other solution files generated at run-time as part of NEMS solution reporting. Accordingly, NEMS scenarios were re-run with households and floorspace selectively varied. The data were assembled and combined with the regional price and quantity solution data. Census level data sets were then created and the regression analysis run for each census region. This provided regional detail for NEMS that enables comparison with the corresponding regional detail for RSTEM. These results are presented in Section IV. In Section V proposed next steps for the project are presented.

II. Analysis of Differences Between Actual and Forecast Values For Selected NEMS Demand Projections. A goal of the forecast evaluation project is to measure the differences between projections and the eventual outcomes of the projected series. Once done, the analysis is additionally to identify those portions of the difference found that can be attributed to differences between the assumed and actual values of the explanatory variables on which the projected series depends. One method for assessing sensitivities in this way, used for NEMS, is to approximate major components of the model’s representation of the energy economy, e.g., major fuel supply and demand relationships using regression analysis, and use the results of the regression analyses to estimate the portions of the differences between projected and actual outcomes as related to differences in assumed versus actual explanatory variables. A group of regression results was developed for each of the NEMS demand modules for selected fuels. For residential and commercial sector demand for electricity and natural gas, reported on here, only two explanatory variables were investigated, price and an activity driver, plus the lagged endogenous variable. The data supporting theses regressions were generated by multiple NEMS solutions with the roster of explanatory variables varied subject to a full factorial experimental design. These results were presented to the ASA Committee on Energy Statistics on October 19, 2007 (http://optima-com.com/ASA_Oct_2007/Oct_Main_Page.htm). The results reported here apply the regression results to an assessment of differences in NEMS projections presented by the AEO1998-AEO2003 for residential and commercial sector electricity and natural gas consumption as projected for the year 2006. Since NEMS is benchmarked to short term forecasts derived using RSTEM, it is about three years into the forecast period that the projects become entirely due to NEMS. Since the sensitivities are based on the AEO2007 version of NEMS, the results are not diagnostic with respect to the prior versions of NEMS. The results were developed for demonstration purposes.

II.1 Method. A differential assessment of the impact of assumed versus actual values for the explanatory variables is based upon a linear function of the form: ΔQuantity(t) = A*ΔPrice(t) + B*ΔDriver(t)+C*ΔQuantity(t-1)+D*ΔHDD(t) + E*ΔCDD(t).[1] The portions of this relationship corresponding to the coefficients A, B, and C were estimated via the regression analysis cited above (for residential sector natural gas demand a secular trend variable was also included). The explanatory driver for residential sector demand was total residential households and the corresponding driver for commercial sector demand was total commercial floorspace. For the weather effects corresponding to the coefficients D and E, two separate weather scenarios were constructed, one with HDD assumptions incremented by +10% and the other with CDD assumptions incremented by +10%. The differences on the right-hand-side of the relationship are between the assumed and actual values for the various explanatory variables. The difference on the left-hand-side of the relationship is the “expected” difference between actual and projected fuel consumption as related to the actual versus assumed outcomes for the values of the explanatory variables. The values used for the coefficients are given in the table below. The elasticity is also given for interpretive purposes. The percentage base for the elasticity is the average for the exogenous and endogenous variable over the projection time frame 2010-2015.

Table II.1: AEO2007 NEMS Demand Sensitivities

Price

Driver

Lag Q

HDD

CDD

Res. Gas Coefficient

-.043361

.02955

.601247

D
.0006661

N/A

Res. Gas

Elasticity

-.085704

.693619

.59695

.563279

N/A

Res. Elec. Coefficient

-.015162

.020493

.550345

.000978

.0004841

Res. Elec.

Elasticity

-.075772

.483561

.542066

.08496

.12623

Comm. Gas Coefficient

-074526

.032519

.288889

.0003459

.000248

Comm. Gas

Elasticity

-.184832

.776192

.283375

.4452072

.0098

Comm. Elec. Coefficient

-.02482

.041423

.332754

.000413

.0004313

Comm. Elec.

Elasticity

-.115481

.68628

.325934

.03758

.117668

II.2 Results. Data for the projected values of consumption and the corresponding explanatory variables were taken from the associated AEO reference case *.ran files. Data for the actual values of the variables in 2006 were taken from the NEMS historical data set as reported in aeo2008.0202a.ran. The regressions run for data from the AEO2007 version of NEMS reported prices in the metric of $2005. Accordingly, all prices were set to this metric using the CPI: Energy Commodities and Services as reported in ftab Table 18 of aeo2008.0202a.ran.

Below is an example of the sensitivity analysis performed for the AEO1998 projection of commercial sector natural gas consumption in the year 2006.

Table II.2 AEO 1998 Projections for Commercial Sector Natural Gas Consumption in 2006

Actual Projected Difference %Difference Impact %Impact

Quantity 2.92 3.646 .726 24.86 N/A N/A

Price 10.34825 7.542997 -2.8052 -27.11 .2090603 7.16

Floorspace 74.8 79.8 5 6.68 .162595 5.57

Lag Quantity 3.089 3.618 .529 17.13 .1528223 5.23

Weather:HDD 3996 4524 528 13.21 .1826 6.25

Weather:CDD 1368 1242 -126 -9.21 -.0031 -.11

Total % 24.11

Explicit% 18.88

Lag% 5.23

Unexplained% .022 .75

The projected rate of consumption was 3.646 quads compared to 2.92 quads actual, a difference of .726 quads which is high by 24.86% of actual. The projected price was 27.11% low. The regression coefficient for price is A = -.074526. Multiplying this by the difference between projected and actual price results in the finding that the projection would be .209 quads (or 7.16%) high for this reason. Projected floorspace was 6.68% high compared to actual. Using the regression coefficient shows that the corresponding projection of natural gas consumption would be 5.57% high. The lagged quantity projected was 17.13% higher than actual which would result in the projection being 5.23% high for this reason. As measured by HDDs, the winter was 13.21% warmer than “normal” and this in turn would result in a 6.25% over projection. Alternatively, the summer was 9.21% warmer than usual as measured by CDDs, resulting in an under prediction of consumption of .11% for this reason. Altogether, differences in assumed versus actual values for the explanatory variables “explained” 24.11% of the over-projection, leaving .75% “unexplained.” The lagged endogenous variable represents a number of implicit influences on demand, such as the roster and efficiency of energy consuming devices that satisfy commercial sector service demands, e.g., heating and cooling. The other influences: price, driver, and weather variables are explicit influences. Taken together, these are termed the “explicit%,” which in this case constitutes 18.88% of the over-projection. Displays such of the above for each of the four fuels and six versions of NEMS are presented in:

http://optima-com.com/Project_Archive/Detailed%20Error%20Attribution_AEO98_03.htm

It should be immediately acknowledged that, in addition to improvements in the particular methods reported on, that there are numerous influences as might explain differences between projected and actual energy consumption. These include unanticipated changes in laws and regulations, differences between projected and actual efficiencies of the underlying energy consuming equipments, and behavioral parameters such as the sensitivity of energy consumers to changes in energy prices. Ultimately, it is the goal of the project to assemble estimates of the importance of other influences such as these. As currently formulated, a study of these influences is beyond the scope of the current project.

II.3 Aggregated Results. The results can be aggregated for each version of NEMS for all four fuels; or, aggregated across the six versions of NEMS with respect to each fuel. The summary table below provides aggregated results for each version of NEMS.

Table II.3: Summary Across All Fuels

AEO Version % Diff % Explicit % Lag % Total % Unexplained

1998 5.606 5.551 2.279 7.83 -2.225

1999 5.794 -1.782 2.118 .335 5.459

2000 4.865 -1.795 1.768 -.027 4.892

2001 11.06 .752 4.128 4.881 6.179

2002 14.364 2.602 5.107 7.709 6.655

2003 11.704 4.84 4.345 9.185 2.519

The results in the table are mixed; but perhaps, indicative. Specifically, the differences between forecast and actual values across the different versions of NEMS tends to increase as the forecast horizon shortens (and this may be due to factors not accounted for here). Never-the-less, the proportion of the difference that is “unexplained” does not increase to the same degree. In general, the volatility of forecast differences is greater than that of the “unexplained” proportion of the differences.

In the tables below, the error attribution for each fuel is averaged over the six versions of NEMS studied.

Table II.4: Average for all versions of NEMS: Residential Sector Electricity

Actual Projected Difference %Difference Impact %Impact

Quantity 4.611 4.5347 -.0763 -1.6567 N/A N/A

Price 27.4633 35.9832 8.5199 31.0217 -.1292 -2.8017

Households 112.6 112.1833 -.4167 -.3717 -.0085 -.185

Lag Quantity 4.638 4.4362 -.2018 -4.3533 -.1111 -2.4083

Weather:HDD 3996 4524 528 13.21 .0516 1.12

Weather:CDD 1368 1242 -126 -9.21 -.061 -1.32

Total % -5.598333

Explicit% -3.19

Lag% -2.4083

Unexplained% .1819 3.9433

Table II.5: Average for all versions of NEMS: Residential Sector Natural Gas

Actual Projected Difference %Difference Impact %Impact

Quantity 4.499 5.4457 .9467 21.0417 N/A N/A

Price 12.058 10.1201 -1.9379 -16.0717 .084 1.865

Households 112.6 112.1833 -.4167 -.3717 -.0123 -.2733

Lag Quantity 4.497 5.406 .909 20.2133 .5465 12.1483

Weather:HDD 3996 4524 528 13.21 .3517 7.82

Total % 21.56

Explicit% 9.413333

Lag% 12.1483

Unexplained% -.0233 -.5183

Table II.6: Average for all versions of NEMS: Commercial Sector Electricity

Actual Projected Difference %Difference Impact %Impact

Quantity 4.435 4.2738 -.1612 -3.635 N/A N/A

Price 24.9708 30.5798 5.609 22.4617 -.1392 -3.1417

Floorspace 74.8 72.7583 -2.0417 -2.7283 -.0846 -1.9067

Lag Quantity 4.351 4.1918 -.1592 -3.6583 -.053 -1.195

Weather:HDD 3996 4524 528 13.21 .0218 .49

Weather:CDD 1368 1242 -126 -9.21 -.0543 -1.22

Total % -6.971666

Explicit% -5.78

Lag% -1.195

Unexplained% .1481 3.3383

Table II.7: Average for all versions of NEMS: Commercial Sector Natural Gas

Actual Projected Difference %Difference Impact %Impact

Quantity 2.92 3.676 .756 25.89 N/A N/A

Price 10.3482 8.3936 -1.9546 -18.8883 .1457 4.9883

Floorspace 74.8 72.7667 -2.0333 -2.7183 -.0661 -2.265

Lag Quantity 3.089 3.6405 .5515 17.855 .1593 5.455

Weather:HDD 3996 4524 528 13.21 .1826 6.25

Weather:CDD 1368 1242 -126 -9.21 -.0031 -.11

Total % 14.32667

Explicit% 8.873333

Lag% 5.455

Unexplained% .3376 11.5633

Inspection of these results reveals that for both sectors the projections of natural gas consumption were the least accurate. Nevertheless, the proportion of the error that could be attributed to errors in projecting the explanatory variables were considerable, particularly for residential sector natural gas consumption. In the case of commercial sector natural gas consumption, the “unexplained” portion of the difference between forecast and actual may simply be due to important influence(s) not yet accounted for by the analysis.

III. RSTEM Sensitivity Analysis. The Regional Short Term Energy Model (RSTEM) is made up of hundreds of equations of which over a hundred are estimated (in contrast to accounting relationships or equations that convert physical units). The model is estimated and run in the econometric modeling platform Eviews 6. Many of the estimated equations are structural demand equations. The equations are estimated by month by census region. The exogenous variables are not always the same for a given fuel across census regions. In general, the exogenous variables include own-price, rarely a cross-price, one (sometimes more) macroeconomic or activity driver, and weather as measured by heating degree days (HDD) and/or cooling degree days (CDD). An assessment of how RSTEM represents energy markets can be accomplished by measuring the sensitivity of fuel demands to changes in each exogenous variable. Since the equations typically have substantial lag structures, it is not generally possible to measure demand sensitivity in terms of a single regression coefficient. Instead, each exogenous variable is changed, one at a time, and the demand simulation is run. The results are then compared to a base case and the corresponding sensitivity is calculated as an arc elasticity. For (say) PB = base case price, PS = scenario price (say PS=.75PB), and QB and QS the corresponding solution quantities, the arc elasticity for each month is given by:

The purpose of this memo is to go through the steps required to edit the data and calculate the corresponding elasticities using the Eviews (version 6) development platform under which RSTEM is run. The results presented here are based upon the October 2007 version of RSTEM.

III.1 Example: Commercial Sector Natural Gas Demand in the Middle Atlantic Region (MAC). The variable naming convention utilized for RSTEM variables is given in:

http://www.eia.doe.gov/emeu/steo/pub/document/partb.html.

The notation for the commercial sector natural gas demand equation for the Middle Atlantic region is:

eq_ngccp_mac,

where “eq” designates an Eviews equation, “ngccp” is the RSTEM syntax for commercial sector natural gas demand, and “mac” denotes the Middle Atlantic census region. When eq_ngccp_mac is opened, the equation is listed in both generic and estimated form. The generic form for this equation is given below (the notation key follows the specification of the equation):

NGCCP_MAC = C(1)

+ C(2)*EESPP_MAC

+ C(3)*(((ZWHD_MAC-ZWHN_MAC)/ZSAJQUS)*(OCT+NOV+DEC+JAN+FEB+MAR+APR))

+ C(4)*(((ZWHD_MAC(-1)-ZWHN_MAC(-1))/ZSAJQUS(-1))*(OCT(-1)+NOV(-1)+DEC(-1)+JAN(-1)+FEB(-1)+MAR(-1)+APR(-1)))

+ C(5)*FEB + C(6)*MAR + C(7)*APR + C(8)*MAY + C(9)*JUN + C(10)*JUL + C(11)*AUG + C(12)*SEP + C(13)*OCT + C(14)*NOV + C(15)*DEC

+ C(16)*TIME*D2001ON

+ C(19)*NGCCU_MAC(-1)*NGCCP_MAC(-1)/CPI2000_MAC(-1)

+ C(20)*NGCCU_MAC(-2)*NGCCP_MAC(-2)/CPI2000_MAC(-2)

+ C(21)*NGCCU_MAC(-3)*NGCCP_MAC(-3)/CPI2000_MAC(-3)

+ [AR(1)=C(18)].

For this specification,

NGCCP_MAC = Middle Atlantic commercial sector natural gas demand;

C(*) = the estimated regression coefficients;

EESPP_MAC = Middle Atlantic private service employment;

ZWHD_MAC = Middle Atlantic forecast HDD;

ZWHN_MAC = Middle Atlantic normal HDD;

ZSAJQUS = days in the month;

JAN, FEB, …, DEC = monthly dummy variables;

TIME = secular trend variable;

D2001ON = 0 before Jan 2001 and = 1 for Jan 2001 and later months;

NGCCU_MAC = natural gas price, commercial sector, Middle Atlantic region; (note that “(-t)” designates a lag of t-many months);

CPI2000_MAC = consumer price index, 2000=1.0, Middle Atlantic region; and,

[AR(1)=C(18)] = adjustment for autocorrelation.

For this specification, sensitivities will be computed for three exogenous variables:

EESPP_MAC = Middle Atlantic private service employment (scenario = +10%)t;

ZWHD_MAC = Middle Atlantic forecast HDD (scenario = +10%); and,

NGCCU_MAC = natural gas price, commercial sector, Middle Atlantic region (scenario = -25%).

The forecast horizon is December 2008. The edits to each of the three exogenous variables are for the twenty four months, January 2007 through December 2008. The procedure is given below for EESPP_MAC. The same procedure was followed for the other exogenous variables. The edit steps, given in Eviews syntax, are:

(1) Save the reference case values: Series EESPP_MAC_Base = EESPP_MAC.

(2) Create an edited file: Series EESPP_MAC_dy = (1.1)*EESPP_MAC.

(3) Now, open EESPP_MAC and EESPP_MAC_dy, delete the values in EESPP_MAC for January 2007 through December 2008 and copy/paste the corresponding values from EESPP_MAC_dy.

(4) Open eq_ngccp_mac and select the “Forecast” option. At the prompt, name the solution to be simulated NGCCP_MAC_dy and execute the simulation.

(5) Set EESPP_MAC back to the base values: Series EESPP_MAC = EESPP_MAC_Base.

Repeat the same steps for ZWHD_MAC and NGCCU_MAC. Finally, open eq_ngccp_mac and select the “Forecast” option. At the prompt, name the solution to be simulated NGCCP_MAC_Sim. This solution will be the reference case against which the other cases will be compared. The price scenario file was identified by a “_dp” extension and the HDD scenario file was identified by a “_dw” extension.

The tables below give the arc elasticities, by month, for 2007 and 2008. The average given is the simple average for 2008. The signs of the sensitivities are correct for ESSPP_MAC, i.e., all positive, and NGCCU_MAC, i.e., all negative. For ZWHD_MAC, the elasticities are generally positive, as expected, i.e., heating demand increases as the winter is colder. For June, July, and August the elasticities (although quite small) are negative, i.e., cooler summer days lead to a (small) reduction in natural gas demand.

Scenaro: Macro-Driver: EESPP_MAC(+10%)

Month ngccp_mac ngccp_mac_dy Scenario/Base Elasticity

2007M01 2.719104 3.080652 1.133 1.3091

2007M02 3.316941 3.673067 1.107 1.0699

2007M03 2.755485 3.1007 1.125 1.2379

2007M04 2.016546 2.346213 1.163 1.5868

2007M05 1.343054 1.67279 1.246 2.296

2007M06 1.103042 1.433589 1.3 2.7365

2007M07 1.074686 1.408466 1.311 2.8228

2007M08 1.12469 1.458851 1.297 2.7162

2007M09 1.140985 1.475732 1.293 2.6865

2007M10 1.334839 1.671187 1.252 2.3497

2007M11 1.909397 2.245976 1.176 1.701

2007M12 2.57941 2.915615 1.13 1.2849

2008M01 3.013029 3.348985 1.112 1.1089

2008M02 3.177304 3.511564 1.105 1.0494

2008M03 2.648252 2.981567 1.126 1.2433

2008M04 1.963052 2.296727 1.17 1.645

2008M05 1.352261 1.686358 1.247 2.309

2008M06 1.145764 1.481168 1.293 2.6813

2008M07 1.109907 1.447743 1.304 2.7739

2008M08 1.154503 1.492561 1.293 2.6819

2008M09 1.166829 1.504764 1.29 2.6563

2008M10 1.365254 1.703642 1.248 2.3155

2008M11 1.944705 2.282203 1.174 1.6767

2008M12 2.624128 2.96069 1.128 1.2655

2008 Average 1.9506

Scenaro: Price: NGCCU_MAC (-25%)

Month ngccp_mac ngccp_mac_dp Scenario/Base Elasticity

2007M01 2.719104 2.719104 1 0

2007M02 3.316941 3.327135 1.003 -.0107

2007M03 2.755485 2.788537 1.012 -.0417

2007M04 2.016546 2.083438 1.033 -.1142

2007M05 1.343054 1.409892 1.05 -.17

2007M06 1.103042 1.154045 1.046 -.1582

2007M07 1.074686 1.107498 1.031 -.1053

2007M08 1.12469 1.148145 1.021 -.0722

2007M09 1.140985 1.162207 1.019 -.0645

2007M10 1.334839 1.356033 1.016 -.0551

2007M11 1.909397 1.932003 1.012 -.0412

2007M12 2.57941 2.605981 1.01 -.0359

2008M01 3.013029 3.049218 1.012 -.0418

2008M02 3.177304 3.228369 1.016 -.0558

2008M03 2.648252 2.712188 1.024 -.0835

2008M04 1.963052 2.031548 1.035 -.12

2008M05 1.352261 1.414605 1.046 -.1577

2008M06 1.145764 1.191668 1.04 -.1375

2008M07 1.109907 1.141155 1.028 -.0972

2008M08 1.154503 1.177893 1.02 -.0702

2008M09 1.166829 1.188824 1.019 -.0654

2008M10 1.365254 1.388278 1.017 -.0585

2008M11 1.944705 1.970047 1.013 -.0453

2008M12 2.624128 2.654266 1.011 -.04

2008 Average -.0811

Scenaro: HDD's: ZWHD_MAC (+10%)

Month ngccp_mac ngccp_mac_dw Scenario/Base Elasticity

2007M01 2.719104 2.824678 1.039 .3999

2007M02 3.316941 3.502538 1.056 .5715

2007M03 2.755485 2.902691 1.053 .5463

2007M04 2.016546 2.10557 1.044 .4535

2007M05 1.343054 1.355896 1.01 .0999

2007M06 1.103042 1.092847 .991 -.0975

2007M07 1.074686 1.070405 .996 -.0419

2007M08 1.12469 1.12448 1 -.002

2007M09 1.140985 1.14157 1.001 .0054

2007M10 1.334839 1.377325 1.032 .329

2007M11 1.909397 2.001058 1.048 .4922

2007M12 2.57941 2.715749 1.053 .5407

2008M01 3.013029 3.177243 1.055 .5571

2008M02 3.177304 3.338495 1.051 .5195

2008M03 2.648252 2.775878 1.048 .4941

2008M04 1.963052 2.04365 1.041 .4224

2008M05 1.352261 1.364278 1.009 .0929

2008M06 1.145764 1.137237 .993 -.0784

2008M07 1.109907 1.106148 .997 -.0356

2008M08 1.154503 1.15428 1 -.002

2008M09 1.166829 1.167304 1 .0043

2008M10 1.365254 1.408156 1.031 .3248

2008M11 1.944705 2.037595 1.048 .4898

2008M12 2.624128 2.762716 1.053 .5403

2008 Average .2774

III.2 Summary of Results. The same procedures were followed for each census region for each of residential and commercial sector electricity and natural gas demand. The elasticities given for each census region are the average of the arc elasticities for each month in 2008. The national elasticity, for comparison to NEMS, is the quantity-weighted average of the regional elasticities. The census region abbreviations are defined here: http://www.eia.doe.gov/emeu/reps/maps/us_census.html. The NEMS results above at the national level are provided for comparative purposes. As given, the NEMS drivers are total residential households and total commercial floorspace. These elasticities are included in the tables for comparison to the RSTEM driver elasticities.

Table III.1: RSTEM Structural Sensitivities For Residential Sector Electricity Demand

Census Region	Macro-Driver qhallc_[reg]	Macro-Driver yrpic_[reg]	Electricty Price esrcu_[reg]	Weather: HDD zwhd_[reg]	Weather: CDD zwcd_[reg]
1 NEC	.907	.086	-.196	.200	.042
2 MAC	.930	.065	-.055	.153	.131
3 ENC	.994	.005	-.230	.316	.191
4 WNC	.976	.022	-.460	.138	.124
5 SAC	.978	.020	-.222	.201	.222
6 ESC	.984	.015	-.237	.174	.126
7 WSC	.981	.017	-.172	.153	.309
8 MTN	.841	-.111(?)	.011(?)	.044	nil(?)
9 PAC	.895	-.016(?)	.001(?)	.116	nil(?)
RSTEM National Elasticity	.956	.012	-.190	.183	.175
NEMS 2007	.484	N/A*	-.075	.085	.126

*For NEMS, personal income is not reported (at least not in ftab Tables #4 and #18). Disposable income is reported in ftab Table #18. The comparative statics NEMS scenario with the single change of +10% for disposable income did not show any variation in residential electricity demand.

Driver #1: qhallc_{reg} = regional family and non-family number of households

Driver #2: yrpic_{reg} = regional nominal personal income

Price: esrcu_{reg} = regional residential sector electricity price

Weather #1: zwhd_{reg} = regional heating degree days

Weather #2: zwcd_{reg} = regional cooling degree days

Table III.2: RSTEM Structural Sensitivities For Residential Sector Natural Gas Demand

Census Region	Macro_Driver cwd_[reg]	Price ngrcu_[reg]	Weather: HDD zwhd_[reg]
1 NEC	.108	-.098	1.00
2 MAC	.753	-.228	.794
3 ENC	1.90	-.705	1.78
4 WNC	.079	-.072	.856
5 SAC	.057	-.052	.717
6 ESC	.112	-.101	.877
7 WSC	.114	-.103	.558
8 MTN	.076	-.088	.815
9 PAC	.012	-.011	.559
RSTEM National Elasticity	.871	-.323	1.27
NEMS 2007	.694	-.085	.563

Driver: cwd_{reg} = regional total, nonfarm, real wage disbursements

Price: ngrcu_{reg} = regional residential sector natural gas price

Weather: zwhd_{reg} = regional heating degree days

Table III.3: RSTEM Structural Sensitivities For Commercial Sector Electricity Demand

Census Region	Macro-Driver eespp_[reg]	Macro-Driver cwdspp_[reg]	Electricty Price escmu_[reg]	Gas Price ngcuu_[reg]	Weather: HDD zwhd_[reg]	Weather: CDD zwcd_[reg]
1 NEC	N/A	.059	-.094	N/A	.026	.011
2 MAC	N/A	N/A	N/A	N/A	.061	.039
3 ENC	1.11	N/A	N/A	N/A	.091	.031
4 WNC	-.048	.597	-.600	-.024	.029	.114
5 SAC	.768	N/A	N/A	N/A	.021	.075
6 ESC	.225	.070	-.070	N/A	.137	.595
7 WSC	N/A	N/A	N/A	-.032	.047	-.120
8 MTN	N/A	.223	-.225	N/A	.011	.087
9 PAC	-.017	.241	-.243	N/A	N/A	.794
RSTEM National Elasticity	.353	.094	-.097	N/A	.045	.228
NEMS 2007	.686		-.115	N/A	.0376	.118

Driver #1: eespp_{reg} = regional private sector employment

Driver #2: cwdspp_{reg} = regional private sector real wage disbursments

Price #1: escmu_{reg} = regional commercial sector electricity price

Price #2: ngcuu_{reg} = regional commercial sector natural gas price

Weather #1: zwhd_{reg} = regional heating degree days

Weather #2: zwcd_{reg} = regional cooling degree days

Table III.4 RSTEM Structural Sensitivities For Commercial Sector Natural Gas Demand

Census Region	Macro_Driver eespp_[reg]	Price ngcuu_[reg]	Weather: HDD zwhd_[reg]
1 NEC	N/A	N/A	.531
2 MAC	1.951	-..081	.277
3 ENC	.316	-.071	.488
4 WNC	N/A	-.142	.436
5 SAC	.057	-.034	.279
6 ESC	.636	-.066	.432
7 WSC	-.129	-.240	.171
8 MTN	.559	-.144	.353
9 PAC	N/A	N/A	-.172
RSTEM National Elasticity	.481	-.068	.492
NEMS 2007	.776	-.185	.445

Driver: eespp_{reg} = regional private sector employment

Price: ngcuu{reg} = regional commercial sector natural gas price

Weather: zwhd_{reg} = regional heating degree days

Inspection of the national level comparisons between NEMS and RSTEM for the directly comparable price and weather sensitivities, shows RSTEM residential sector sensitivities larger than the corresponding values for NEMS. For the commercial sector, the RSTEM weather sensitivities are larger, but the own-price sensitivities are smaller. Some of the values are quite similar for RSTEM compared to NEMS, even though the methodologies utilized by the two forecasting systems are quite different. Comparison of results at the census region level of detail is presented in the next section.

IV. Project status and results were presented to SMG and OIAF in April 2008. The NEMS modelers proposed that since the NEMS modules represented their corresponding portion of the U.S. energy system at a regional level of detail, that the sensitivity analyses would be potentially more useful if conducted at a regional level of detail. For the NEMS demand modules regional detail is that of census regions, the same regional detail as used for the RSTEM demand equations. Accordingly, work was undertaken to redo the NEMS sensitivities at a census region level of detail. As discussed below, data problems delayed the construction of data sets to use for the regression analyses; however, data for the weather cases were readily available. As a result, an assessment of NEMS weather sensitivities for comparison to RSTEM could be undertaken immediately. These results are presented in the next subsection.

IV.1 NEMS Regional Weather Sensitivities. Weather sensitivities were computed at the regional level for each of residential and commercial sector electricity and natural gas demand. The sensitivities were based upon two NEMS scenarios for which HDD and CDD values were incremented individually by +10%. The arc elasticities were then computed for each of the years 2010-2015. For QB and QS the base and weather scenario consumption quantities; and, WB and WS the corresponding weather variable values, i.e., WS = (1.1)WB, the arc elasticities calculated as:

In the tables below, for NEMS, for each region and nationally, the elasticity reported is the average for the period 2010-2015. For RSTEM, the value given is the average of the monthly elasticities for the year 2008.

Table IV.1: Residential Sector Electricity Consumption

Region	NEMS HDD Elasticity	RSTEM HDD Elasticity	NEMS CDD Elasticity	RSTEM CDD Elasticity
NEC	.0625	.1995	.0389	.0424
MAC	.0929	.1526	.067	.131
ENC	.0807	.316	.0863	.1906
WNC	.0878	.138	.0998	.1242
SAC	.0822	.201	.1782	.2216
ESC	.118	.174	.1495	.1261
WSC	.0703	.1526	.2032	.3085
MTN	.0701	.0437	.1001	N/A
PAC	.1009	.1163	.044	N/A
National Average	.085	.183	.1267	.1754

Table IV.2: Residential Sector Natural Gas Consumption

Region	NEMS HDD Elasticity	RSTEM HDD Elasticity	NEMS CDD Elasticity	RSTEM CDD Elasticity
NEC	.646	1.002	nil	N/A
MAC	.6137	.7936	nil	N/A
ENC	.6242	1.7758	nil	N/A
WNC	.5934	.8561	nil	N/A
SAC	.4974	.7169	nil	N/A
ESC	.5761	.8772	nil	N/A
WSC	.3797	.5581	nil	N/A
MTN	.532	.8147	nil	N/A
PAC	.4768	.5589	nil	N/A
National Average	.5632	1.268373	nil	N/A

Table IV.3: Commercial Sector Electricity Consumption

Region	NEMS HDD Elasticity	RSTEM HDD Elasticity	NEMS CDD Elasticity	RSTEM CDD Elasticity
NEC	.0337	.0261	.0792	.0108
MAC	.0522	.0607	.0977	.0388
ENC	.0456	.0907	.0962	.0307
WNC	.0388	.0294	.0872	.1135
SAC	.0376	.0206	.1571	.0748
ESC	.0343	.1373	.1179	.5946
WSC	.0257	.0468	.1637	-.1204(!)
MTN	.022	.0114	.1028	.0871
PAC	.039	N/A	.0819	.7944
National Average	.0376	.0448	.1181	.2276

Table IV.4: Commercial Sector Natural Gas Consumption

Region	NEMS HDD Elasticity	RSTEM HDD Elasticity	NEMS CDD Elasticity	RSTEM CDD Elasticity
NEC	.4809	.5305	.017	N/A
MAC	.5217	.2774	.0079	N/A
ENC	.5423	.4878	.0026	N/A
WNC	.4903	.4359	.0008	N/A
SAC	.3111	.2787	.0299	N/A
ESC	.4288	.4318	.0138	N/A
WSC	.3077	.1709	.0079	N/A
MTN	.5806	.3527	.0135	N/A
PAC	.2639	.1726	.0065	N/A
National Average	.4455	.4915	.0098	N/A

Summary of results. At the national level, of the eight weather sensitivities considered, residential and commercial natural gas consumption do not have CDD’s as explanatory variables in the RSTEM forecasting equations. For NEMS, residential natural gas consumption is not significantly sensitive to CDD variation. Commercial sector natural gas consumption has an extremely small sensitivity to CDD variation. For the remaining six sensitivities, in general, RSTEM’s projections of energy consumption are more sensitive to weather than NEMS. Of these, residential electricity sensitivity to CDD’s and commercial sector natural gas and electricity demand sensitivity to HDD’s are (nevertheless) quite similar. For residential sector natural gas and electricity demand sensitivities to HDD’s and commercial sector demand electricity demand sensitivity to CDD’s, the RSTEM elasticities are on the order of twice as large as NEMS’. Accordingly, in adjusting forecast “errors” to account for differences between assumed and actual weather, the RSTEM adjustments would be on the order of double those for NEMS in percentage terms.

Table IV.5: National Level Elasticity Ratio (RSTEM/NEMS)

Fuel Demand	HDD Elasticity	CDD Elasticity
Residential Sector Natural Gas	2.25	N/A
Residential Sector Electricity	2.15	1.38
Commercial Sector Natural Gas	1.10	N/A
Commercial Sector Electricity	1.19	1.93

IV.2 NEMS Price and Driver Sensitivities at the Census Region Level of Detail. Regional values for total residential households and total commercial floorspace are reported out in separate files compared to the ftab format solution files that had been used for the national level NEMS sensitivity analyses. As a result, scenarios for which these drivers were incremented and decremented had to be re-run and the data extracted for combination with the price and quantity data that are reported in regional detail by the ftab format. The data for the regressions were assembled from NEMS scenarios specified specifically to support the project. Price variations were chosen as base (AEO2007) or (+/- 25%) for the high/low cases. Total residential households were not edited directly. Instead, each of new housing starts for single and multiple family homes, delivered mobile homes, and population over 16 years old were changed +/- 10%. This caused total households to vary by +/- 1/2 % in the high low cases. All possible combinations of these changes were included in the database used for the regressions. Altogether, for each fuel demand, this required nine NEMS solutions to use to generate the data. This particular strategy is termed a “full factorial” experimental design. For RSTEM, the model configuration includes individual demand functions for each region and fuel. These could be run directly with each explanatory variable changed individually as reported in Section III. The RSTEM elasticities given in the tables below are based upon this approach.

For NEMS, the elasticities were calculated from regression results derived from data extracted from NEMS solutions. The regression specification utilized was for each fuel was:

Fuel Demand (Q) = A(Price) + B(Driver) + C(Q_LAG) + D. (1)

For residential sector natural gas demand a secular trend variable was added to the specification. For each fuel, the price was the own-price. For residential sector fuel demand the driver is total residential households and for commercial sector demand the driver is total commercial floorspace. This specification was employed with great success for data pooled for the forecast years 2010-2015.

The specification given in (1) above is very austere. There are two issues to resolve in using this particular formula. As an example, residential sector electricity demand is considered. The first issue is the inclusion of only the own-price of elasticity. This choice would be defended if it could be determined that the cross-price elasticities are small. Second, is the linear form of the relationship. This was tested by comparing the impacts of changing price and total households individually, and summing the results, to the impact of changing price and households at the same time. At the national level, both of these tests supported the specification given in (1).

These tests were run for the regional data. The outcomes were generally the same as those found for the national data. Links to reports of the test results are provided at the end of the memo. An example for cross-price sensitivity is given in the table below for New England (NEC) residential sector electricity demand.

Region = NEC

Fuel #1 = Sector and Source Residential: Distillate Fuel Oil

Fuel #2 = Sector and Source Residential: Natural Gas

Fuel #3 = Sector and Source Residential: Electricity

Price = Sector and Source Residential: Electricity

year Fuel #1 Fuel #2 Fuel #3

2010 .0049 0 -.079

2011 .0055 .0002 -.1313

2012 .0063 .0005 -.1543

2013 .0093 0 -.1585

2014 .0097 -.0001 -.1605

2015 .0104 .0001 -.1621

2016 .0122 .0002 -.1637

2017 .0135 .0004 -.1655

2018 .0153 .0006 -.1664

2019 .0152 .001 -.1687

2020 .0155 .0017 -.1694

2021 .0161 .0025 -.1715

2022 .016 .0029 -.1736

2023 .0174 .0027 -.1741

2024 .0192 .0031 -.1761

2025 .021 .0034 -.178

2026 .022 .0035 -.179

2027 .0234 .0037 -.1815

2028 .0244 .004 -.186

2029 .0275 .0044 -.1901

2030 .0299 .0078 -.194

The values in each column corresponding to each of the three fuels are the arc elasticities computed from the results of the AEO2007 reference case and the scenario prepared for the regression data base for which the electricity price was incremented by +25%, starting in 2010. For Q_B and Q_S the fuel demands in the base and scenario case; and, P_B and P_S the corresponding values of the price of residential sector electricity, the elasticity is computed by:

Inspection of the table reveals that the cross-elasticities are quite small.

The table below provides the results of testing for the linearity of (1) for the New England data.

Region = NEC

Fuel = Sector and Source Residential: Electricity

Year dPrice dDriver Sum dBoth Sum/dBoth

2010 -.0032 -.0003 -.0035 -.0035 1.0041

2011 -.0052 -.0005 -.0057 -.0057 1.0002

2012 -.0063 -.0007 -.007 -.0069 1.0067

2013 -.0065 -.0009 -.0073 -.0073 1.0078

2014 -.0066 -.001 -.0076 -.0076 1.008

2015 -.0067 -.0012 -.0079 -.0078 1.0063

2016 -.0068 -.0014 -.0082 -.0081 1.0066

2017 -.0069 -.0015 -.0084 -.0084 1.0061

2018 -.0071 -.0017 -.0087 -.0087 1.0076

2019 -.0071 -.0018 -.009 -.0089 1.0063

2020 -.0073 -.002 -.0093 -.0092 1.0063

2021 -.0074 -.0021 -.0095 -.0094 1.0059

2022 -.0075 -.0023 -.0097 -.0097 1.0053

2023 -.0076 -.0024 -.01 -.0099 1.0058

2024 -.0077 -.0025 -.0103 -.0102 1.0061

2025 -.0078 -.0027 -.0105 -.0104 1.0074

2026 -.008 -.0028 -.0108 -.0107 1.0095

2027 -.0082 -.003 -.0111 -.011 1.012

2028 -.0083 -.0031 -.0114 -.0113 1.0121

2029 -.0084 -.0032 -.0116 -.0115 1.0096

2030 -.0085 -.0032 -.0117 -.0117 1.0046

The values in the “dPrice” column are the differences for each year Q_S – Q_B for the AEO2007 reference case and the scenario where only the electricity price was increased by +25%. In the “dDriver” column, the values are the differences from the reference case and the scenario where only total households was changed by about -1/2%. The value in the “Sum” column is the sum of the values in the “dPrice” and “dDriver” column for each year. The values in the “dBoth” column are the differences for each year between the reference case and the scenario where both price (+25%) and total households (~ -1/2%) were changed. The last column shows the ratio “Sum/dBoth.” As the ratio is near “1,” the relationship being modeled appears linear. The results support the linear specification of (1) since the ratio values are consistently close to “1.” Similar results were found for each fuel, sector, and region.

The regression results for residential sector electricity and natural gas demand are given in the tables below, with the NEMS results for the price and driver based upon the regression specification given above. The elasticities are point elasticities, measured at the means of the variables from the pooled data. For example, given (1) above, the price elasticity of demand is computed as:

with the percentage bases for Price and Q computed as the average for the time span chosen across all observations. For RSTEM, as discussed, the elasticities are arc elasticities computed from a reference case and a case with price (+25%) or the driver (+10%) varied individually.

Table IV.6: NEMS and RSTEM Elasticities For Residential Sector Electricity Demand

Census Region	RSTEM Driver #1 Households	RSTEM Driver #2 Personal Income	RSTEM Electricity Price	NEMS Driver Households	NEMS Electricity Price
1 NEC	.907	.086	-.196	.581	-.08
2 MAC	.930	.065	-.055	.529	-.08
3 ENC	.994	.005	-.230	.474	-.076
4 WNC	.976	.022	-.460	.414	-.074
5 SAC	.978	.020	-.222	.435	-.074
6 ESC	.984	.015	-.237	.557	-.075
7 WSC	.981	.017	-.172	.493	-.077
8 MTN	.841	-.111(?)	.011(?)	.512	-.076
9 PAC	.895	-.016(?)	.001(?)	.386	-.074
National Elasticity	.956	.012	-.190	.484	-.076

Table IV.7: NEMS and RSTEM Elasticities For Residential Sector Natural Gas Demand

Census Region	RSTEM Nonfarm Wages	RSTEM Natural Gas Price	NEMS Driver Households	NEMS Natural Gas Price
1 NEC	.108	-.098	.848	-.098
2 MAC	.753	-.228	.922	-.088
3 ENC	1.90	-.705	.693	-.087
4 WNC	.079	-.072	.634	-.088
5 SAC	.057	-.052	.640	-.085
6 ESC	.112	-.101	.683	-.088
7 WSC	.114	-.103	.845	-.078
8 MTN	.076	-.088	.745	-.083
9 PAC	.012	-.011	1.05	-.079
National Elasticity	.871	-.323	.694	-.086

For the residential sector, the NEMS sensitivities at the census region level of detail are quite uniform and not greatly different than the sensitivity calculated at the national level. For residential electricity demand, both RSTEM and NEMS use total households as explanatory variables. The RSTEM elasticities are on the order of double those of NEMS. The RSTEM price sensitivities are generally larger and more volatile across census regions compared to NEMS. For residential natural gas demand, RSTEM uses nonfarm wage disbursals, rather than households, as the activity driver. On average, the RSTEM sensitivity to this explanatory variable is larger than NEMS, although this is due to a few very large, regional elasticities. The RSTEM price elasticities are also larger in general and more volatile across census regions, compared to NEMS.

Table IV.8: NEMS and RSTEM Elasticities For Commercial Sector Electricity Demand

Census Region	RSTEM Driver 1: Private Sector Employment	RSTEM Driver 2: Private Wages Paid	NEMS Driver: Total Floorspace	RSTEM Electricity Price	RSTEM Gas Price	NEMS Electricity Price
1 NEC	N/A	.059	.642	-.094	N/A	-.093
2 MAC	N/A	N/A	.636	N/A	N/A	-.106
3 ENC	1.11	N/A	.660	N/A	N/A	-.123
4 WNC	-.048	.597	.784	-.600	-.024	-.115
5 SAC	.768	N/A	.685	N/A	N/A	-.127
6 ESC	.225	.070	.774	-.070	N/A	-.114
7 WSC	N/A	N/A	.690	N/A	-.032	-.125
8 MTN	N/A	.223	.681	-.225	N/A	-.106
9 PAC	-.017	.241	.743	-.243	N/A	-.112
National	.353	.094	.686	-.097	nil	-.115

Table IV.9: NEMS and RSTEM Elasticities For Commercial Sector Natural Gas Demand

Census Region	RSTEM Driver: Private Sector Employment	NEMS Driver: Total Floorspace	RSTEM Natural Gas Price	NEMS Natural Gas Price
1 NEC*	N/A	.723	N/A	-.183
2 MAC	1.951	.778	-..081	-.178
3 ENC	.316	.749	-.071	-.184
4 WNC	N/A	.792	-.142	-.184
5 SAC	.057	.781	-.034	-.192
6 ESC	.636	.770	-.066	-.181
7 WSC	-.129(!)	.799	-.240	-.175
8 MTN	.559	.754	-.144	-.187
9 PAC	N/A	.781	N/A	-.177
National	.481	.776	-.068	-.185

*Weather was the only RSTEM explanatory variable.

As with the residential sector, the commercial sector sensitivities at the census region level of detail are quite uniform and not greatly different than the sensitivity calculated at the national level; and, the NEMS own price elasticities are generally larger, with some notable exceptions.

The cross-price effects and linearity tests for both residential and commercial sector electricity and natural gas demand at the regional level are available on the Project Archive website:

Residential Sector Electricity: http://optima-com.com/Project_Archive/RS_EL_TEST.htm

Residential Sector Natural Gas: http://optima-com.com/Project_Archive/RS_NG_TEST.htm

Residential Sector Total Households Scenarios: http://optima-com.com/Project_Archive/RS_HH_07.htm

Commercial Sector Electricity: http://optima-com.com/Project_Archive/CM_EL_TEST.htm

Commercial Sector Natural Gas: http://optima-com.com/Project_Archive/CM_NG_TEST.htm

Commercial Sector Total Floorspace Scenarios: http://optima-com.com/Project_Archive/CM_FS_07.htm

V. Proposed Next Steps. Project activities have to this point been largely demonstrations of techniques and approaches that serve project goals. There has been, and will continue to be, a considerable ingredient of “learning by doing” associated with project activities. The roster of tasks to undertake next include the following:

(1) Bring a compiled version of the AEO2008 version of NEMS and ultimately the AEO2009 version of NEMS in-hand and conduct similar sensitivity analyses for comparative, and possibly diagnostic, purposes as compared to the AEO2007 version of NEMS upon which the NEMS results presented here are based.

(2) Expand the NEMS modules considered to include a supply module and a module using the NEMS optimizing platform (the Electricity Market Module has been proposed).

(3) Increase the influences considered to include, such as, changes in laws, regulations, and other factors against the eventual assessment of NEMS error components to begin in 2010 with the AEO2007 version of NEMS.

(4) Automate and begin to conduct structural sensitivity analyses for each version of RSTEM for diagnostic purposes.

(5) Increase the number of RSTEM forecasting equations considered to include energy supply.

(6) Conduct a reduced form analysis of each published version of RSTEM projections as actual data become available (this portion of the project is described in a separate paper).

A very great deal of work remains to develop and apply all of the necessary components of a program to routinely evaluate EIA forecasts and forecasting methods as related to sensitivity analyses of alternative versions of the forecasting models. Even for the limited results obtained so far, useful insights concerning the way in which the forecasting systems account for important influences have been obtained. These results assist in model development; and, enable an assessment of the accuracy, and uncertainty, embodied in the projections.

[1] “HDD” refer to population weighted heating degree days and “CDD” refers to population weighted cooling degree days.