VA Construction
VA Is Working to Improve Initial Project Cost Estimates, but Should Analyze Cost and Schedule Risks
Gao ID: GAO-10-189 December 14, 2009
The Department of Veterans Affairs (VA) operates one of the largest health care systems in the country. As of August 2009, VA's Veterans Health Administration (VHA) had 32 major ongoing construction projects with an estimated total cost of about $6.1 billion and average cost per project of about $191 million. Some of these projects were initiated as part of VA's Capital Asset Realignment for Enhanced Services (CARES) process, which was a comprehensive assessment of VHA's capital asset requirements. In response to a congressional request, GAO (1) described how costs and schedules of current VHA major construction projects have changed, (2) determined the reasons for changes in costs and schedules, and (3) described the actions VA has taken to address cost increases and schedule delays. To do its work, GAO reviewed construction documents, visited three construction sites, and interviewed VA officials.
While about half of the 32 major ongoing construction projects are within their budget, 18 projects have experienced cost increases and 11 have experienced schedule delays since they were first submitted to Congress. Five projects have experienced a cost increase of over 100 percent. For example, the cost of a new medical center in Las Vegas rose from an initial estimate of $286 million to over $600 million, an increase of about 110 percent. Thirteen projects have experienced cost increases of between 1 and 100 percent. In addition, 11 projects have experienced schedule delays, 4 of which are more than 24 months. There are several reasons for construction project cost increases and schedule delays, including VA preparing initial cost estimates that were not thorough, significant changes to project scope after the initial estimate was submitted, and unforeseen events such as an increase in the cost of construction materials. According to VA officials, VA prepared numerous estimates during the CARES process, and some of these estimates used rudimentary estimating techniques such as average cost-per-square-foot and were completed by VA staff that did not have cost estimating expertise. The scope of some projects changed after VA submitted an estimate to Congress, which increased the projects' costs. For example, the scope for the original design for a new medical center in Las Vegas did not fully account for the amount of medical services the center would need to provide. As a result, the original estimate of $286 million rose to over $600 million. VA has taken steps to improve initial construction project cost estimates, but could better assess the risks to costs and schedules. VA plans to prepare more comprehensive estimates after approving projects and before submitting them to Congress. It is not clear how effective this new process will be, but it could improve VA's estimates. While VA contractors follow construction scheduling procedures that generally meet best practices, VA does not conduct cost or schedule risk analyses, which use statistical techniques to predict risks that can lead to cost increases and schedule delays. Thus, VA cannot quantify the largest risks to a project or mitigate those risks. For example, GAO conducted a schedule risk analysis for a medical center in Las Vegas and found that there is a 50 percent chance that the project won't be finished until more than 6 months after its estimated completion date. VA also does not require an integrated master schedule that includes VA and contractor efforts for all project phases, which can be critical to a project's success.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director:
Team:
Phone:
GAO-10-189, VA Construction: VA Is Working to Improve Initial Project Cost Estimates, but Should Analyze Cost and Schedule Risks
This is the accessible text file for GAO report number GAO-10-189
entitled 'VA Construction: VA Is Working to Improve Initial Project
Cost Estimates, but Should Analyze Cost and Schedule Risks' which was
released on December 14, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Ranking Member, Committee on Veterans' Affairs, House of
Representatives:
United States Government Accountability Office:
GAO:
December 2009:
VA Construction:
VA Is Working to Improve Initial Project Cost Estimates, but Should
Analyze Cost and Schedule Risks:
GAO-10-189:
GAO Highlights:
Highlights of GAO-10-189, a report to the Ranking Member, Committee on
Veterans‘ Affairs, House of Representatives.
Why GAO Did This Study:
The Department of Veterans Affairs (VA) operates one of the largest
health care systems in the country. As of August 2009, VA‘s Veterans
Health Administration (VHA) had 32 major ongoing construction projects
with an estimated total cost of about $6.1 billion and average cost per
project of about $191 million. Some of these projects were initiated as
part of VA‘s Capital Asset Realignment for Enhanced Services (CARES)
process, which was a comprehensive assessment of VHA‘s capital asset
requirements.
In response to a congressional request, GAO (1) described how costs and
schedules of current VHA major construction projects have changed, (2)
determined the reasons for changes in costs and schedules, and (3)
described the actions VA has taken to address cost increases and
schedule delays. To do its work, GAO reviewed construction documents,
visited three construction sites, and interviewed VA officials.
What GAO Found:
While about half of the 32 major ongoing construction projects are
within their budget, 18 projects have experienced cost increases and 11
have experienced schedule delays since they were first submitted to
Congress. Five projects have experienced a cost increase of over 100
percent. For example, the cost of a new medical center in Las Vegas
rose from an initial estimate of $286 million to over $600 million, an
increase of about 110 percent. Thirteen projects have experienced cost
increases of between 1 and 100 percent. In addition, 11 projects have
experienced schedule delays, 4 of which are more than 24 months.
There are several reasons for construction project cost increases and
schedule delays, including VA preparing initial cost estimates that
were not thorough, significant changes to project scope after the
initial estimate was submitted, and unforeseen events such as an
increase in the cost of construction materials. According to VA
officials, VA prepared numerous estimates during the CARES process, and
some of these estimates used rudimentary estimating techniques such as
average cost-per-square-foot and were completed by VA staff that did
not have cost estimating expertise. The scope of some projects changed
after VA submitted an estimate to Congress, which increased the
projects‘ costs. For example, the scope for the original design for a
new medical center in Las Vegas did not fully account for the amount of
medical services the center would need to provide. As a result, the
original estimate of $286 million rose to over $600 million.
VA has taken steps to improve initial construction project cost
estimates, but could better assess the risks to costs and schedules. VA
plans to prepare more comprehensive estimates after approving projects
and before submitting them to Congress. It is not clear how effective
this new process will be, but it could improve VA‘s estimates. While VA
contractors follow construction scheduling procedures that generally
meet best practices, VA does not conduct cost or schedule risk
analyses, which use statistical techniques to predict risks that can
lead to cost increases and schedule delays. Thus, VA cannot quantify
the largest risks to a project or mitigate those risks. For example,
GAO conducted a schedule risk analysis for a medical center in Las
Vegas and found that there is a 50 percent chance that the project
won‘t be finished until more than 6 months after its estimated
completion date. VA also does not require an integrated master schedule
that includes VA and contractor efforts for all project phases, which
can be critical to a project‘s success.
Figure: Range of Cost Changes in Ongoing Projects:
[Refer to PDF for image: horizontal bar graph]
Percent change is cost estimates: -25% to 0;
Number of projects: 1.
Percent change is cost estimates: No change;
Number of projects: 13.
Percent change is cost estimates: 0 to 50%;
Number of projects: 6.
Percent change is cost estimates: 51 to 100%;
Number of projects: 7.
Percent change is cost estimates: 100% plus:
Number of projects: 5.
Source: GAO analysis of VA data.
[End of figure]
What GAO Recommends:
To provide a better estimate of the cost and completion date of a
construction project, GAO recommends that the VA Secretary, for all
major projects, conduct a cost risk analysis, a schedule risk analysis
when appropriate, and require the use of an integrated master schedule.
VA concurred with our recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-10-189] or key
components. For more information, contact Terrell G. Dorn at (202) 512-
6923 or dornt@gao.gov.
[End of section]
Contents:
Letter:
Background:
Costs Have Increased for 18 of the 32 Construction Projects and
Schedules for 11 Construction Projects Have Been Delayed:
Cost Increases and Schedule Delays Result from a Number of Factors:
VA Is Working to Improve Estimates but Could Better Assess Risks to
Costs and Schedules:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Veterans Affairs:
Appendix III: Consolidation and Expansion of Medical Centers in
Cleveland, Ohio:
Appendix IV: Construction of Spinal Cord Injury/Disease Center in
Syracuse, New York:
Appendix V: Construction of New Medical Center Complex in Las Vegas,
Nevada:
Appendix VI: Cost Increases and Schedule Delays:
Appendix VII: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Ongoing Projects That Experienced a Cost Increase as of August
2009:
Table 2: Projects That Have Experienced Schedule Delays:
Table 3: Extent Construction Schedules Met Best Practices:
Table 4: Extent That Bed Tower Cost Estimate Met Best Practices:
Table 5: Extent That Bed Tower Construction Schedule Met Best
Practices:
Table 6: Extent That SCI/D Center Cost Estimate Met Best Practices:
Table 7: Extent That Parking Garage Schedule Met Best Practices:
Table 8: Extent That Cost Estimate for Las Vegas Medical Center Met
Best Practices:
Table 9: Extent That Construction Schedule for Las Vegas Hospital Met
Best Practices:
Table 10: Probability of Project Completion:
Table 11: Risks at the 80th Percentile:
Table 12: Projects That Experienced a Cost Increase and/or a Schedule
Delay:
Figure:
Figure 1: Range of Cost Changes in Ongoing Projects:
Abbreviations:
A/E: architect/engineering:
CARES: Capital Asset Realignment for Enhanced Services:
CFM: Office of Construction and Facilities Management:
NCA: National Cemetery Administration:
OMB: Office of Management and Budget:
PVA: Paralyzed Veterans of America:
SCI/D: Spinal Cord Injury/Disease:
SRA: Schedule Risk Analysis:
VA: The Department of Veterans Affairs:
VBA: Veterans Benefits Administration:
VHA: Veterans Health Administration:
VISN: Veterans' Integrated Service Network:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
December 14, 2009:
The Honorable Steve Buyer:
Ranking Member:
Committee on Veterans' Affairs:
House of Representatives:
Dear Mr. Buyer:
The Department of Veterans Affairs (VA) operates one of the largest
health care systems in the country. VA, through its Veterans Health
Administration (VHA), provided health care to almost 5.5 million
veterans in 2008. VA constructs new medical facilities and also
maintains and renovates existing medical facilities. Any major medical
facility construction project over $10 million must be specifically
authorized by law.[Footnote 1] As part of that approval process, VA
sends a prospectus to Congress[Footnote 2] that contains information
about each planned major project. This information includes an initial
estimate of the overall cost of the project and, in some cases, a
completion date of the project.[Footnote 3] As of August 2009, VHA had
32 ongoing major construction projects with an estimated total cost of
about $6.1 billion and average project cost of about $191 million.
While VA has undertaken a number of major construction projects in
recent years, you have expressed concern that some of these projects
have increased in cost, are behind schedule, or both. To provide you
with information on the costs and schedules of VA's major construction
projects, this report (1) describes how costs and schedules of current
VHA major construction projects have changed since they were first
submitted to Congress, (2) determines the reasons for changes in costs
and schedules, and (3) describes the actions VA has taken to address
cost increases and schedule delays as well as the challenges VA faces
in managing its major construction program.
To do our work, we reviewed VA data on current major construction
projects, including the original cost estimates and completion dates
submitted to Congress and the project's current status. We reviewed and
analyzed construction documents and interviewed VA officials. To obtain
detailed information on specific projects, we selected three major
construction sites to visit based on their phase of construction and
overall estimated cost. We visited construction sites in Cleveland,
Ohio, Las Vegas, Nevada, and Syracuse, New York to determine the
reasons for changes in costs and schedules. In addition, we researched
and reviewed relevant laws relating to the amounts that were authorized
and appropriated for these projects. We also performed a risk analysis
of the construction schedule for a new medical center in Las Vegas--one
of VA's largest ongoing projects--to determine, among other things, the
likelihood of its being completed on time. We selected each site based
on their relatively high construction costs and the fact that
construction was in progress. The information from our site visits is
illustrative and cannot be generalized to sites agencywide.
We conducted this performance audit from October 2008 through December
2009 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We determined the data were sufficiently reliable for the purposes of
this report. Appendix I contains a detailed description of our scope
and methodology.
Background:
Most VA major construction projects are for VHA medical facilities.
[Footnote 4] To determine potential new major construction projects,
VHA officials identify gaps in health service during their strategic
planning process, and VHA officials in field offices develop capital
needs plans to fill these service gaps. These capital plans are then
reviewed by a Capital Investment Panel that gives each proposed project
a score based on a number of factors, including, among other things,
the plan's effect on health care, safety, and energy use. The Capital
Investment Panel then produces a priority list of projects, and the
Secretary of VA determines how many projects to request for funding
each year and works with the Office of Management and Budget (OMB) to
produce VA's part of the President's budget. Some large projects, such
as the construction of a new medical center, can be divided into
distinct phases and funded over several years. When the President
submits VA's budget to Congress, the budget includes a prospectus for
each proposed major construction project. This prospectus includes,
among other things, a cost estimate for the project that VA staff has
assembled. In addition, some prospectuses include an estimated month
and year that the project will be completed, although this is not
required by law. This prospectus is the initial estimate that VA sends
to Congress. Congress uses this information to authorize and
appropriate funds for the project.
In 1999, we reported that with better management of its large, aged
capital assets, VA could significantly reduce the funding used to
operate and maintain underused, unneeded or inefficient properties.
[Footnote 5] We further noted that the savings could be used to enhance
health care services for veterans. Thus, we recommended that VA develop
market-based plans for realigning its capital assets. In response, VA
initiated a process known as the Capital Asset Realignment for Enhanced
Services (CARES)--a comprehensive, long-range assessment of its health
care system's capital asset requirements. As a result of CARES, VA
requested funding for about 30 new major construction projects in
fiscal years 2004 and 2005. While 8 of these projects have been
completed, many are among the 32 ongoing projects. This effort required
VA to prepare initial estimates for each project over the course of a
few months. In the 2 years prior to CARES, VA proposed fewer than five
major construction projects each fiscal year. According to VA, the
CARES process was a onetime major initiative. However, its lasting
result was to provide a set of tools and processes that allow VA to
continually determine the future resources needed to provide health
care to our nation's veterans.
VA's Office of Construction and Facilities Management (CFM) is
responsible for administering major construction projects. Once a
project has been authorized by law and Congress appropriates funds for
it, CFM staff contracts with an architect/engineering (A/E) firm to
design the project. The A/E firm develops an architectural design for
the project and also produces a cost estimate for the entire project.
This cost estimate is generally more detailed and accurate than the
initial cost estimate. After the project has been designed, CFM then
solicits bids for project construction and awards a construction
contract. The construction contractor is responsible for developing a
detailed construction schedule. CFM reviews the construction schedule
and also assigns CFM engineers to work on-site as project managers to
monitor the construction process until the facility is ready to be
turned over to local VA staff. Once construction begins, the
construction company is generally responsible for cost increases and
schedule overruns under the terms of the fixed-price contract, unless
VA and the contractor agree to a change order to the construction
contract to modify scope, account for unforeseen conditions, or remedy
a design error.
We have reported that cost estimates that are completed when a project
is in the conceptual stage have a high degree of uncertainty.[Footnote
6] As a project progresses, this degree of uncertainty decreases
because risks are mitigated or realized. However, we have also found
that cost estimates tend to be lower than the final project costs
because program managers and decision-makers do not always consider all
of the potential risks to a project and tend to be optimistic when
planning a project.
Cost estimating requires both science and judgment. Since answers are
seldom--if ever--precise, the goal is to find a reasonable "answer."
[Footnote 7] Cost estimates are based on many assumptions, including
the rate of inflation and when construction will begin. Generally, the
more information that is known about a project and is used in the
development of the estimate, the more accurate the estimate is expected
to be.[Footnote 8] OMB's guidance for preparing budget documents
identifies many types and methods of estimating project costs. The
expected accuracy of the resulting project cost estimates varies,
depending on the estimating method used.
Costs Have Increased for 18 of the 32 Construction Projects and
Schedules for 11 Construction Projects Have Been Delayed:
While about half of VHA's ongoing major construction projects are
within budget, 18 projects have experienced cost increases and 11 have
experienced schedule delays. The cost for one project has decreased
since the original estimate for it was submitted to Congress.
Several Projects Have Experienced Cost Increases:
Eighteen of the 32 ongoing VHA major construction projects have
experienced cost increases.[Footnote 9] When a project's cost
increases, VA can receive a new authorization and an additional
appropriation from Congress. Without additional funds from Congress, VA
must alter the scope of the project to ensure that the project does not
exceed the amount Congress has appropriated for the project by more
than 10 percent.[Footnote 10] The cost increases that these 18 projects
have experienced since the estimates were initially submitted to
Congress range from 2 to 285 percent. In addition to those 18 projects,
the costs of 13 projects have not changed, and 1 project has
experienced a cost decrease. Figure 1 shows the range of cost changes
in ongoing VHA major construction projects.
Figure 1: Range of Cost Changes in Ongoing Projects:
[Refer to PDF for image: horizontal bar graph]
Percent change is cost estimates: -25% to 0;
Number of projects: 1.
Percent change is cost estimates: No change;
Number of projects: 13.
Percent change is cost estimates: 0 to 50%;
Number of projects: 6.
Percent change is cost estimates: 51 to 100%;
Number of projects: 7.
Percent change is cost estimates: 100% plus:
Number of projects: 5.
Source: GAO analysis of VA data.
[End of figure]
Five projects have experienced a cost increase of more than 100
percent. These projects include new construction and seismic
corrections (which are improvements to a structure to make it less
susceptible to earthquakes). For example, in its fiscal year 2006
budget submission, VA submitted a $286 million estimate to Congress for
a new medical center in Las Vegas, Nevada. However, VA estimated in
2007 that the project would cost just over $600 million (an increase of
110 percent) and in 2008 the project's authorization was modified and
the project received an additional appropriation from Congress.
However, VA now estimates that the project will cost about $100 million
less than it anticipated.[Footnote 11] More information about the new
medical facility in Las Vegas is in appendix V.
Seven projects experienced a cost increase between 51 and 100 percent
and six projects experienced a cost increase between 0 and 50 percent.
These projects vary in size and type, from a modernization of patient
wards in Georgia that is estimated to cost about $24.5 million to a new
medical center in Louisiana that is estimated to cost $925 million. All
projects that experienced a cost increase are listed in table 1.
Table 1: Ongoing Projects That Experienced a Cost Increase as of August
2009:
Location: Las Vegas, NV;
Description: New medical facility;
Initial estimate: $286,000,000;
Estimated cost as of August 2009: $600,400,000;
Cost increase: $314,400,000;
Percent increase: 110%.
Location: Orlando, FL;
Description: New medical facility;
Initial estimate: $347,700,000;
Estimated cost as of August 2009: $656,800,000;
Cost increase: $309,100,000;
Percent increase: 89%.
Location: New Orleans, LA;
Description: New medical facility;
Initial estimate: $636,000,000;
Estimated cost as of August 2009: $925,000,000;
Cost increase: $289,000,000;
Percent increase: 45%.
Location: Denver, CO;
Description: New medical facility;
Initial estimate: $621,000,000;
Estimated cost as of August 2009: $800,000,000;
Cost increase: $179,000,000;
Percent increase: 29%.
Location: San Juan, PR;
Description: Seismic corrections;
Initial estimate: $145,200,000;
Estimated cost as of August 2009: $299,200,000;
Cost increase: $154,000,000;
Percent increase: 106%.
Location: St. Louis, MO;
Description: Medical facility and cemetery improvement;
Initial estimate: $69,053,000;
Estimated cost as of August 2009: $211,300,000;
Cost increase: $142,247,000;
Percent increase: 206%.
Location: Biloxi, MS;
Description: Hospital restoration/consolidation;
Initial estimate: $174,600,000;
Estimated cost as of August 2009: $310,000,000;
Cost increase: $135,400,000;
Percent increase: 78%.
Location: Pittsburgh, PA;
Description: Medical center consolidation;
Initial estimate: $185,076,000;
Estimated cost as of August 2009: $295,600,000;
Cost increase: $110,524,000;
Percent increase: 60%.
Location: Bay Pines, FL;
Description: New outpatient clinic;
Initial estimate: $65,100,000;
Estimated cost as of August 2009: $131,800,000;
Cost increase: $66,700,000;
Percent increase: 102%.
Location: Gainesville, FL;
Description: Renovate patient rooms;
Initial estimate: $85,200,000;
Estimated cost as of August 2009: $136,700,000;
Cost increase: $51,500,000;
Percent increase: 60%.
Location: San Juan, PR;
Description: Seismic corrections;
Initial estimate: $50,000,000;
Estimated cost as of August 2009: $89,473,968;
Cost increase: $49,473,965;
Percent increase: 79%.
Location: Palo Alto, CA;
Description: Seismic corrections;
Initial estimate: $14,013,000;
Estimated cost as of August 2009: $54,000,000;
Cost increase: $39,987,000;
Percent increase: 285%.
Location: Fayetteville, AR;
Description: Clinical addition;
Initial estimate: $56,163,000;
Estimated cost as of August 2009: $93,000,000;
Cost increase: $36,837,000;
Percent increase: 66%.
Location: Syracuse, NY;
Description: Spinal cord injury/disease center;
Initial estimate: $53,900,000;
Estimated cost as of August 2009: $84,969,000;
Cost increase: $31,069,000;
Percent increase: 58%.
Location: Tampa, FL;
Description: Polytrauma expansion;
Initial estimate: $223,800,000;
Estimated cost as of August 2009: $231,500,000;
Cost increase: $7,700,000;
Percent increase: 3%.
Location: Long Beach, CA;
Description: Seismic corrections;
Initial estimate: $107,845,000;
Estimated cost as of August 2009: $112,845,000;
Cost increase: $5,000,000;
Percent increase: 5%.
Location: Atlanta, GA;
Description: Modernize patient wards;
Initial estimate: $20,700,000;
Estimated cost as of August 2009: $24,534,000;
Cost increase: $3,834,000;
Percent increase: 18%.
Location: Des Moines, IA;
Description: Extended care building;
Initial estimate: $25,000,000;
Estimated cost as of August 2009: $25,550,000;
Cost increase: $550,000;
Percent increase: 2%.
Total:
Initial estimate: $3,166,350,000;
Estimated cost as of August 2009: $5,082,671,968;
Cost increase: $1,916,321,968;
Percent increase: 61%.
Source: GAO analysis of VA data.
[End of table]
As of August 2009, the costs of 13 projects have not changed from their
initial estimated cost. We found that VA reduced the scope of some
projects so that the projects would not exceed their budget. For
example, one project we visited in Cleveland, Ohio, is designed to
consolidate two medical centers and construct a new facility at one of
the medical centers. According to VA officials in Cleveland, VA reduced
the original scope of the project by excluding room for 30 new patient
beds in the new facility so that the project could stay within its
budget. However, VA will make space for the 30 beds by expanding part
of its existing facility through separate facility funds. VA staff made
other changes to the original plan for the new facility, such as
deleting balconies from patient's rooms and using more concrete and
less steel in the structure, so that the facility could be completed
within budget. More information about the medical center consolidation
in Cleveland is in appendix III. In addition to those projects that did
not experience a cost increase, one project experienced a cost
decrease. Specifically, the cost to construct a data center in West
Virginia decreased from $35 million to $33.7 million, or about 4
percent.
Schedule Delays Have Occurred in 11 Projects:
Eleven of the 32 ongoing projects are projected to be completed later
than originally estimated. Even if the cost of a project has not
increased, a schedule delay can lead to an increased cost to VA because
CFM project managers must stay on to monitor the project as it is being
built. A schedule delay can also affect veterans' access to medical
care, since VA constructs facilities where they are needed to serve the
local veteran population and a schedule delay results in veterans
waiting longer for the services to be available. Of the 11 projects
that have experienced a schedule delay, 2 are scheduled to be completed
within 2 months of their originally scheduled end date, 5 are scheduled
to be completed between 12 and 24 months of their originally scheduled
end date, and 4 are scheduled to be completed more than 24 months after
their originally scheduled end date. These projects range from an
electrical upgrade in Florida that is estimated to end less than a
month after its initial estimated completion date to seismic
corrections at a facility in Puerto Rico that are estimated to end
about 7 years after their initial estimated completion date. The
original estimated completion dates, the latest estimated completion
dates, and the change in dates for those projects are in table 2.
Information on the number of projects that experienced both a schedule
delay and a cost increase is in appendix VI.
Table 2: Projects That Have Experienced Schedule Delays:
Location: Tampa, FL;
Description: Electrical upgrade;
Initial estimated completion date: 7/2/10;
Estimated completion date as of August 2009: 7/14/2010;
Change in months[A]: 0.5.
Location: Anchorage, AK;
Description: Outpatient clinic;
Initial estimated completion date: 1/10/2010;
Estimated completion date as of August 2009: 2/15/2010;
Change in months[A]: 1.
Location: Martinsburg, WV;
Description: Capital region data center;
Initial estimated completion date: 5/30/09;
Estimated completion date as of August 2009: 7/7/2010;
Change in months[A]: 13.
Location: Milwaukee, WI;
Description: Spinal cord injury center;
Initial estimated completion date: 12/31/2009;
Estimated completion date as of August 2009: 2/11/2011;
Change in months[A]: 13.5.
Location: American Lake, WA;
Description: Seismic corrections;
Initial estimated completion date: 3/31/2009;
Estimated completion date as of August 2009: 7/10/2010;
Change in months[A]: 15.5.
Location: Las Vegas, NV;
Description: New medical center;
Initial estimated completion date: 9/6/2009;
Estimated completion date as of August 2009: 8/22/2011;
Change in months[A]: 23.5.
Location: Columbia, MO;
Description: Operating suite replacement;
Initial estimated completion date: 5/31/2010;
Estimated completion date as of August 2009: 5/30/2012;
Change in months[A]: 24.
Location: Cleveland, OH;
Description: Medical center consolidation;
Initial estimated completion date: 9/1/2008;
Estimated completion date as of August 2009: 2/21/2011;
Change in months[A]: 29.5.
Location: Syracuse, NY;
Description: Spinal cord injury center;
Initial estimated completion date: 12/6/2009;
Estimated completion date as of August 2009: 5/19/2012;
Change in months[A]: 29.5.
Location: Palo Alto, CA;
Description: Seismic corrections;
Initial estimated completion date: 11/30/2006;
Estimated completion date as of August 2009: 5/27/2011;
Change in months[A]: 54.
Location: San Juan, PR;
Description: Seismic corrections;
Initial estimated completion date: 10/30/2002;
Estimated completion date as of August 2009: 11/18/2009;
Change in months[A]: 84.5.
Source: GAO analysis of VA data.
[A] The change in months is rounded to the nearest half month.
[End of table]
Cost Increases and Schedule Delays Result from a Number of Factors:
Cost increases and schedule delays have been caused by factors that
have generally occurred before construction of the project begins.
These factors include initial estimates that were not thorough because
they were completed quickly, scope changes that occurred after the
initial estimate, and unforeseen events and market conditions such as a
rise in construction costs.
Some Cost Estimates Were Not Thorough:
The CARES process required VA to quickly provide initial cost estimates
for about 30 major construction projects. Specifically, in 2004 VA had
about 3 months to provide initial cost estimates to Congress so that
Congress could consider authorizing these projects and appropriating
funds for them in fiscal years 2004 and 2005. According to VA, a number
of VA staff worked to produce these initial estimates, including staff
that had limited cost estimating expertise. The 30 projects included
three new large medical centers in Las Vegas, Nevada; Denver, Colorado;
and Orlando, Florida. Estimates prepared for these 30 projects were
prepared quickly and sometimes based on rudimentary designs. For
example, VHA officials in Syracuse told us that they had about 6 weeks
to prepare their initial estimate for a new spinal cord injury center,
which they did by using analogous estimating techniques such as the
cost-per-square foot of new construction in Syracuse. As a result, the
initial estimate was only a rough order-of-magnitude estimate. We have
reported that, while it is possible to develop a rough order-of-
magnitude estimate in days, a first-time budget-quality estimate would
likely require many months.[Footnote 12] VA officials in Syracuse who
worked to prepare this estimate told us that they were surprised when
the project was included in VA's fiscal year 2005 budget request
because they knew that the estimate was only a rough order-of-magnitude
estimate.
Some Cost Estimates and Schedules Were Affected by Scope Changes:
In two of our case studies, the scope of the project changed
substantially after VA submitted its estimate to Congress. VA officials
also told us that scope changes have occurred in other projects. In Las
Vegas, the initial estimate to Congress was based on plans for a large
VA clinic. However, VA later determined that a much larger medical
center was needed in Las Vegas after it became clear that an inpatient
medical facility it shares with the Department of Defense would not be
adequate to serve the medical needs of local veterans. This decision
greatly increased the cost, delayed the completion date of the project,
and required a modified authorization and an additional appropriation
from Congress. Since the estimate for the Las Vegas medical center was
based on a preliminary design for an expanded clinic, additional
functions had to be added to the clinic design to provide the services
necessary for the medical center. This expansion of the scope of the
project resulted in both a cost increase and schedule delay for the
project.
In Syracuse, New York, the original design of a new Spinal Cord Injury/
Disease (SCI/D) center that is being built on the campus of the VA
medical center did not include money for additional parking. However,
after the project had been authorized by Congress and was in design, VA
officials in Syracuse commissioned a study to examine future parking
needs at the medical center. The study concluded that, based on the new
SCI/D center and projected demand from patients and staff, there should
be an additional 429 to 528 parking spaces at the medical center. As a
result of this study, VA officials in Syracuse decided to add two
floors to the existing parking garage at an estimated cost of $10
million. Based on the parking garage addition and other changes to the
project, VA received a modified authorization in 2006 and an
appropriation of $23.8 million in fiscal year 2008 for the SCI/D
center. More information about the new SCI/D center in Syracuse is in
appendix IV.
Failure to involve stakeholders early in the process can also lead to
changes in scope. In Syracuse, the Paralyzed Veterans of America (PVA)
objected to some aspects of the design of the SCI/D center. For
example, PVA advocated for a dedicated entrance from the parking garage
to the SCI/D center, which is being built on the fourth floor of the
medical center. This dedicated entrance would allow veterans with
spinal cord injuries to enter the center directly from the parking
garage, without requiring the veterans to go down to the street from
the parking garage, outside to the main entrance of the medical center,
then up to the 4th floor of the medical center for treatment. According
to VA staff in Syracuse, VA agreed to make changes that would improve
access to the facility, and this increased the cost of the project.
Some Cost Estimates Were Affected by Market Conditions and Some
Schedules Were Affected by Unforeseen Events:
Changes in construction market conditions can escalate the costs of VA
construction projects. The cost of many materials used in construction--
from concrete to electrical equipment--increased more than the consumer
price index (indicating that construction costs increased more than
other costs) from 2003 through 2007. Specifically, the cost of these
construction materials increased over 28 percent between 2003 and 2007,
whereas the consumer price index increased about 13 percent over the
same period.[Footnote 13] Hurricane Katrina drove up the cost of
construction materials nationwide because the high demand for
construction in the New Orleans region strained supplies of material
and labor. In Las Vegas, several large billion-dollar projects created
competition for construction services, and this area experienced an
even greater cost increase as the demand for new construction exceeded
supply of materials and labor.
The schedule for one of our case studies was delayed by land
acquisition issues. In Cleveland, while the project remains within
budget, the project schedule was delayed 9 months because a property
acquisition took longer than expected. Part of the land that the bed
tower is being built on had been donated to the City of Cleveland for
use as parkland. The city could not give the land to VA until the city
was able to change the designated use of the donated land from parkland
to a more general use. More information about the construction project
in Cleveland is in appendix III.
VA Is Working to Improve Estimates but Could Better Assess Risks to
Costs and Schedules:
VA has developed a new process for determining its initial estimates
that allows for more time between VA approving a project and submitting
a cost and schedule estimate to Congress. However, VA does not analyze
cost risks to examine the changing assumptions on the cost estimate. VA
also does not have an integrated master schedule, which includes both
VA and contractor effort for all phases of the entire project, and does
not conduct a schedule risk analysis to help determine when projects
will be completed. While VA is not required to develop an integrated
master schedule and cost and schedule risk analyses,[Footnote 14] we
have identified these steps as best practices in project scheduling and
cost estimating.[Footnote 15]
VA Is Working to Improve Initial Estimates, but Needs to Analyze Cost
Risks:
VA has developed a new process to improve its initial estimates for
major construction projects. This new process allows VA to increase the
time between VA approving a project and submitting that project, and
its initial estimate, to Congress. According to VA officials, with this
additional time, VHA will be able to gather more information about a
project and begin preliminary design work. These officials noted that
VA will ideally have as much as 35 percent of the design work completed
before the project's first estimate is submitted to Congress. Cost
estimators can then use these designs to develop the initial cost
estimate that VA sends to Congress. According to VA officials, the
initial estimate should be more precise than estimates provided to
Congress in the past because the scope of the project will be more
developed.
Until the fiscal year 2010 budget cycle, field staff in VHA produced
the first estimate for a project. Beginning with the fiscal year 2010
budget cycle, for any project in the top 10 of the priority list, CFM
will work with VHA staff in the field to produce the first estimate of
the project's cost. CFM staff includes professionals with estimating
and construction engineering skills, whereas VHA staff in the field
generally does not possess these skills.
These new requirements were not in effect when the projects we studied
were developed. Therefore, we were not able to evaluate the process.
While it is unclear how much design work will actually occur before VA
submits a project and its estimate to Congress, the new process holds
promise to improve VA's initial estimates, particularly if the new
process requires early stakeholder input on a proposed project so that
any resulting changes in the project scope can be incorporated into the
estimate before it is submitted to Congress.
After a project has been authorized and funded based on VA's initial
estimate, VA hires an architect/engineering firm to design the major
construction project. The firm hires a contractor to develop a cost
estimate for the project. We visited three major construction sites--
Cleveland, Ohio, Las Vegas, Nevada, and Syracuse, New York. At these
sites, we found that these cost estimates were generally comprehensive
and well documented. Specifically, the estimate included an estimating
plan, structure, purpose, and documentation. However, we also found
that the cost estimates for projects in Cleveland and Las Vegas were
not adequately maintained during construction because they did not
include updated information based on actual costs as the project
progressed.
We also found that the estimates for projects in Syracuse and Las Vegas
did not include a cost risk analysis to examine the effect of changing
assumptions on the cost estimate. Conducting a cost risk analysis is
particularly important because only by quantifying cost risk can
management make informed decisions about risk mitigation strategies.
Quantifying cost risk also provides a benchmark for measuring future
progress. We identified best practices for estimating and managing
program costs in a cost assessment guide we issued in 2009.[Footnote
16] As we note in our cost assessment guide, agencies should begin to
follow these best practices at the earliest stages of the cost
estimation process, which includes the preparation of the initial
estimate submitted to Congress. Our cost estimating guide has been
endorsed by OMB. More information on the cost estimates for these three
sites is in appendices III through V.
VA Generally Follows Best Practices for Construction Schedules at the
Projects We Visited, but Does Not Conduct a Schedule Risk Analysis:
After the design is complete, VA hires a contractor to construct the
project by the completion date set in the contract. The contractor then
develops a construction schedule that details all of the activities
that the contractor plans to finish by the completion date. Generally,
the contractor must finish by the completion date or face financial
penalties. At the sites we visited, we found that these schedule
estimates, which occur after VA has submitted its initial estimate to
Congress, generally followed best practices for scheduling. For
example, we found that the contractor regularly updated the
construction schedule with actual dates as the work progressed. All
best practices for schedules, and the extent that they were met at our
site visits, are in table 3. More detailed information is included in
appendices III through V.
Table 3: Extent Construction Schedules Met Best Practices:
Best practice: Capturing key activities;
Cleveland, OH: Met;
Las Vegas, NV: Substantially met;
Syracuse, NY: Met.
Best practice: Sequencing key activities;
Cleveland, OH: Met;
Las Vegas, NV: Substantially met;
Syracuse, NY: Met.
Best practice: Assigning resources to key activities;
Cleveland, OH: Met;
Las Vegas, NV: Substantially met;
Syracuse, NY: Met.
Best practice: Establishing the duration of key activities;
Cleveland, OH: Met;
Las Vegas, NV: Met;
Syracuse, NY: Met.
Best practice: Integrating schedule activities horizontally and
vertically;
Cleveland, OH: Met;
Las Vegas, NV: Met;
Syracuse, NY: Met.
Best practice: Establishing the critical path for all activities;
Cleveland, OH: Met;
Las Vegas, NV: Substantially met;
Syracuse, NY: Met.
Best practice: Identifying the float between activities[A];
Cleveland, OH: Met;
Las Vegas, NV: Met;
Syracuse, NY: Met.
Best practice: Conducting a schedule risk analysis;
Cleveland, OH: Not met;
Las Vegas, NV: Not met;
Syracuse, NY: Not met.
Best practice: Updating the schedule using logic and duration to
determine dates;
Cleveland, OH: Met;
Las Vegas, NV: Partially met;
Syracuse, NY: Met.
Source: GAO analysis of VA data.
[A] "Float" is the amount of time an activity can slip before delaying
the entire project.
[End of table]
Although VA met or partially met nearly all scheduling best practices
at the three sites, VA does not conduct a schedule risk analysis of its
major construction projects, and therefore cannot predict a project's
completion date with confidence. A schedule risk analysis, which is one
of our best practices in project scheduling, uses statistical
techniques to predict a level of confidence in meeting a project's
completion date. The objective of the analysis is to develop a
probability distribution of possible completion dates that reflect the
project and its quantified risks. This analysis can help project
managers both understand the most important risks to the project and to
focus on mitigating these risks. We conducted a schedule risk analysis
of the construction schedule for the new medical center in Las Vegas,
Nevada, that is scheduled to be completed on August 22, 2011. We
conducted on-site interviews with staff who are working on the project
in Las Vegas and asked them to discuss potential risks to the project,
including how the risk would affect the project's timeline and the
likelihood of the risk occurring. Using this information, we developed
a list of risks to the project (such as the chance that the design is
inadequate or that labor is not available) and how each risk would
impact the duration of specific activities in the schedule. We then
used modeling software to run a Monte Carlo[Footnote 17] simulation,
which consisted of the computer-generated results of 3,000 estimates of
the future schedule based on the activities in the schedule, the chance
that some activities would be affected by some risks, and the predicted
affect of those risks on the duration of each activity. This analysis
showed that there is a 50 percent probability that the project will be
completed by March 1, 2012 (about 6 months after the current estimated
completion date) and an 80 percent probability that the project will be
completed by May 17, 2012 (about 9 months after the current estimated
completion date). Although we did not conduct a schedule risk analysis
for other VA major construction projects, the result of our analysis
for the Las Vegas Medical Center project shows the types of risks that
major construction projects face and the impact those risks can have on
meeting project milestones. More information on our schedule risk
analysis can be found in appendix V.
We shared the results of our schedule risk analysis with CFM staff in
Las Vegas. Specifically, we noted that we found the two biggest risks
to the project are that the design may be inadequate and that the
occupancy needs may change. CFM staff in Las Vegas told us that they
are working to mitigate the risk of inadequate design and have
discovered architectural drawings that do not include utilities. As a
result, CFM has directed the architect/engineer firm to revise the
drawings to include utilities. CFM staff also stated that they can deny
any changes to the project scope and that they can choose not to allow
changes that will affect the scheduled completion date.
VA Does Not Have an Integrated Master Schedule for Major Construction
Projects:
VA does not require an integrated master schedule for major
construction projects that encompasses both VA and contractor effort
for all phases of the entire project and shows the relationships
between various project phases (such as design, construction, and when
the project is "activated" for occupancy and use). However, we have
stated that the success of any project depends, in part, on having an
integrated and reliable schedule.[Footnote 18] Without a fully
integrated and reliably derived schedule, it is difficult to estimate
the overall cost and schedule of a project. In addition, individual
phases of a multiphase project can be completed on time, but the
project as a whole can be delayed and construction phases that are not
part of an integrated master schedule may not be completed in the most
efficient manner. For example, a VA nursing home in Las Vegas was
completed in 2009 but cannot be put into service until another phase of
the construction project--the on-site medical center--is completed and
can provide medical care to residents of the nursing home. The medical
center is scheduled to be completed in 2011. According to VA officials,
VA decided to construct the new nursing home because construction costs
in Las Vegas were escalating quickly, and VA officials thought that
they could save money by constructing the nursing home as soon as
possible. However, construction costs have recently decreased in the
Las Vegas area, and VA must pay to maintain the new nursing home from
2009 to 2011 even though the nursing home will not be used for VA
patients.
Conclusions:
Estimates for major construction projects, like any estimate of a
future activity, can never be exact. Some of VA's past estimates have
been off-base, although the reasons for this are sometimes outside of
VA's control. These imprecise estimates resulted in Congress
authorizing and appropriating millions of dollars for projects based on
estimates that proved to be inaccurate. In some of these cases, VA was
forced to change the scope of the project in order to stay within the
original estimate or the projects' authorizations were modified and
Congress has had to appropriate more funds to allow VA to finish some
projects.
VA is taking steps to make its initial estimates more accurate in the
future. VA is working to complete some preliminary design work on
projects and improve initial estimates so that they are more likely to
be closer to the actual costs and schedules of a project, but the
effect of these changes on VA's initial estimates remains to be seen.
While VA is taking steps to improve its initial estimates, it does not
always conduct a cost risk analysis, which would allow project managers
to better identify issues that could lead to cost escalation and
improve managers' ability to make informed decisions on how to minimize
cost risks. VA has also not used a schedule risk analysis to determine
the likelihood of a major project being completed on time. We recognize
that conducting a cost risk and schedule risk analysis takes both
financial resources and some time and that it may only be appropriate
to conduct these analyses when a project is particularly costly,
complex, or has a compressed schedule. However, the overall effect of
the analyses is to provide VA, congressional decisionmakers, and other
stakeholders with more precise information about when a project will be
completed and the main risks to a project being completed on time. With
this information, VA could provide more accurate schedule estimates to
stakeholders and could also work to mitigate risks to the project and
ensure that the project is completed on time. We have identified cost
risk and schedule risk analysis as best practices in our cost
assessment guide, which has been endorsed by OMB.
While the construction schedules we reviewed generally met best
practices, VA's lack of an integrated master schedule--which would
integrate VA and contractor effort for all phases of a project,
including all design and construction work--hampers VA's ability to
provide accurate information on the schedule for a project. Many
factors that can delay a project, such as changes in scope and
unforeseen site conditions, occur before construction begins. The use
of an integrated master schedule could assist VA in monitoring the
progress of a major construction project before construction begins and
allow VA to increase the accuracy of its schedule estimates.
Recommendations for Executive Action:
To improve estimates of the cost of a major construction project as
well as the risks that may influence the cost and how these risks can
be mitigated, GAO recommends that the Secretary of Veterans Affairs
direct CFM to conduct a cost risk analysis of major construction
projects.
To provide a realistic estimate of when a construction project may be
completed as well as the risks to the project that could be mitigated,
we recommend that the Secretary of Veterans Affairs direct CFM to take
the following two actions. First, require the use of an integrated
master schedule for all major construction projects. This schedule
should integrate all phases of project design and construction. Second,
conduct a schedule risk analysis, when appropriate, based on the
project's cost, schedule, complexity, or other factors. Such a risk
analysis should include a determination of the largest risks to the
project, a plan for mitigating those risks, and an estimate of when the
project will be finished if the risks are not mitigated.
Agency Comments and Our Evaluation:
We provided a draft of this report to VA for review and comment. VA
generally agreed with our conclusions and concurred with our
recommendations. In reference to our statement that some cost increases
and schedule delays were attributable to scope changes, VA stated that
it is important to note that VA followed all applicable laws and
congressional notification requirements during the execution of the
projects, and maintained the integrity and intent of each project as
authorized by Congress. While we did not find any instances where VA
did not follow applicable laws or congressional notification
requirements, we did not specifically evaluate VA's compliance with
such laws and requirements because this was outside the scope of our
review. VA's letter is contained in appendix II. In addition, VA made a
number of technical corrections, which we incorporated as appropriate.
We are sending copies of this report to the Secretary of Veterans
Affairs. Additional copies will be sent to interested congressional
committees. The report will also be available at no charge on the GAO
Web site at [hyperlink, http://www.gao.gov].
If you have any questions about this report, please contact me at (202)
512-2834 or at dornt@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made key contributions to this
report are listed in appendix VII.
Sincerely yours,
Signed by:
Terrell G. Dorn:
Director, Physical Infrastructure Issues:
[End of section]
Appendix I: Scope and Methodology:
In this report, we examined: (1) how costs and schedules of current
Veterans Affairs (VA) major medical construction projects have changed
since they were first submitted to Congress,[Footnote 19] (2) the
reasons for cost and schedule changes in VA's major medical
construction projects, and (3) the actions VA has taken to address cost
increases and schedule delays as well as the challenges VA faces in
managing its major medical construction program.
To address these issues, we reviewed pertinent laws relating to
construction, authorization and appropriation of VA projects. We also
examined the documents VA submitted to Congress, including the Office
of Management and Budget's form 300 provided with VA's budget that has
been required since 2006 and a project prospectus. We obtained and
analyzed data that VA provided on the status of VA's active major
medical construction projects, as of August, 2009. We also reviewed
VA's management of construction projects at three locations and
interviewed VA headquarters' officials from the Veterans Health
Administration (VHA) and the Office of Construction and Facilities
Management (CFM) as well as project managers at the construction sites
we visited.
To determine how costs and schedules of current VA major medical
construction projects have changed since they were first submitted to
Congress, we reviewed VA data on current major medical construction
projects, including the original cost estimates and completion dates
submitted to Congress and the projects' current status as of August
2009. We analyzed the current cost and completion dates against the
information provided to Congress to determine the increase in costs and
the extent to which projects exceeded or were expected to exceed the
original time allotted and summarized the results. VA officials
confirmed the reliability of the data provided for these projects.
To identify the reasons for cost and schedule changes in VA's
construction projects, we interviewed VA headquarters officials
regarding the status of all projects and examined project documents and
interviewed on-site managers and engineers at three projects we
selected. We selected projects based on VA-provided data on all of VA's
ongoing major medical construction projects as of March 2009. The data
included a short project description, project location, the original
and current total cost of the project, the original and current
completion date, and the percent of construction completed. VA
officials confirmed the reliability of the data provided. We selected
projects for site visits based on the following criteria and the
results cannot be applied to all of VA's major construction projects:
* Construction projects were between 20 percent and 70 percent
completed.
* Projects were estimated to cost $75 million or more.
* Projects were among those experiencing the greatest cost increases or
schedule delays relative to other VA major medical construction
projects.
* Projects were of different types of major construction projects
because there could be factors in cost and scheduling that relate to
one project type or factors that are systemic trends that occur across
all project types. Project types include new construction, renovation
of existing structures, expansion, or a combination of project types.
* Projects were selected from each of VA's three regions to account for
differences in management at VA regional offices that could impact cost
increases and schedule delays.
Based on our criteria, we selected three major medical construction
sites:
* consolidation of the Brecksville Veterans Affairs Medical Center and
the Wade Park Veterans Affairs Medical Center and construction of a new
90-bed tower for patient care in Cleveland, Ohio, estimated to cost
$102.3 million and to be completed by September 2009 and now scheduled
for February 2011;
* construction of Spinal Cord Injury Center, surgical suite renovation,
and expansion of the parking garage in Syracuse, New York, originally
estimated to cost $53.4 million and be completed by December 2009 and
now estimated to cost $84,969,000 and be completed by May 19, 2012;
and:
* construction of a new, comprehensive Medical Center Complex in Las
Vegas, Nevada, that will include a nursing home, ambulatory care
center, primary and specialty care, surgery, mental health,
rehabilitation, geriatric and extended care. Originally estimated to
cost $286 million and be completed by September 2009, it is now
expected to open in March 2012 and cost $600.4 million. The Las Vegas
project will also include administrative and support functions and
Veterans Benefits Administration offices.
To identify the actions VA has taken to address cost increases and
schedule delays as well as the challenges VA faces in managing its
major medical construction program we reviewed the procedures that VA's
Office of Construction and Facilities Management put in place beginning
in 2007. We also reviewed documentation and interviewed VA headquarters
officials and project managers for the sites we visited to determine
how estimated costs and schedules had been prepared. We then analyzed
the cost estimates and schedules prepared for the three projects we
visited and interviewed VA project managers and engineers, contractors,
and cost estimators and schedulers to ascertain the extent to which
their estimates and schedules compared with the best practices
identified in previous GAO work.
We used the GAO Cost Estimating and Assessment Guide[Footnote 20] (GAO-
09-3SP), as criteria to analyze cost estimates. For this guide, GAO
cost experts assessed 12 measures consistently applied by cost-
estimating organizations throughout the federal government and industry
and considered best practices for developing reliable cost-estimates.
We analyzed the cost estimating practices used by VA in developing its
cost estimates against these 12 best practices. After reviewing
documentation submitted by the VA and information obtained during
interviews, we determined the extent that the cost estimates met the
characteristics of cost estimating best practices for the three
projects we reviewed. For the purpose of this review, we grouped these
practices into four characteristics of a high-quality and reliable cost
estimate. They are:
* Comprehensive: The cost estimates should include both government and
contractor costs of the project over its full life cycle, from
inception of the project through design, development, deployment, and
operation and maintenance to retirement of the project. They should
also provide a level of detail appropriate to ensure that cost elements
are neither omitted nor double counted, and they should document all
cost-influencing ground rules and assumptions.
* Well-documented: The documentation should address the purpose of the
estimate, the project background and system description, its schedule,
the scope of the estimate (in terms of time and what is and is not
included), the ground rules and assumptions, all data sources,
estimating methodology and rationale, the results of the risk analysis,
and a conclusion about whether the cost estimate is reasonable.
Therefore, a good cost estimate--while taking the form of a single
number--is supported by detailed documentation that describes how it
was derived and how the expected funding will be spent in order to
achieve a given objective. For example, the documentation should
capture in writing such things as the source data used and their
significance, the calculations performed and their results, and the
rationale for choosing a particular estimating method or reference.
Moreover, this information should be captured in such a way that the
data used to derive the estimate can be traced back to, and verified
against their sources. Finally, the cost estimate should be reviewed
and accepted by management to ensure that there is a high level of
confidence in the estimate and the estimating process.
* Accurate: The cost estimates should provide for results that are
unbiased, and they should not be overly conservative or optimistic.
Estimates are accurate when they are based on an assessment of most
likely costs, adjusted properly for inflation, and contain few, if any,
minor mistakes. In addition, the estimates should be updated regularly
to reflect material changes in the project, such as when schedules or
other assumptions change so that the estimate is always reflecting
current status. Among other things, the estimate should be grounded in
documented assumptions and a historical record of cost estimating and
actual experiences on other comparable projects.
* Credible: The cost estimates should discuss any limitations of the
analysis because of uncertainty or biases surrounding data or
assumptions. Major assumptions should be varied, and other outcomes
recomputed to determine how sensitive they are to changes in the
assumptions. Risk and uncertainty analysis should be performed to
determine the level of risk associated with the estimate. Further, the
estimate's results should be crosschecked, and an independent cost
estimate conducted by a group outside the acquiring organization should
be developed to determine whether other estimating results produce
similar results.
Our review of project schedules was based on research that identified a
range of best practices associated with effective schedule estimating.
[Footnote 21] In addition, we obtained the consulting services of David
Hulett, Ph.D., to assist in our risk analysis of the Las Vegas Medical
Center project schedule.[Footnote 22] We analyzed documentation
submitted by the VA project office and construction staff for three of
VA's major medical construction projects. We also conducted multiple
interviews with project managers, contractors, and schedulers to
determine the extent that projects' current schedule met the best
practice criteria. These practices include:
* Capturing all activities: The schedule should reflect all activities
(steps, events, outcomes, etc.) as defined in the project's work
breakdown structure, to include activities to be performed by both the
government and its contractors.
* Sequencing all activities: The schedule should be planned so that it
can meet project critical dates. To meet this objective, activities
need to be logically sequenced in the order that they are to be carried
out. In particular, activities that must finish prior to the start of
other activities (i.e., predecessor activities) as well as activities
that cannot begin until other activities are completed (i.e., successor
activities) should be identified. Identifying interdependencies among
activities that collectively lead to the accomplishment of events or
milestones can be used as a basis for guiding work and measuring
progress.
* Assigning resources to all activities: The schedule should
realistically reflect what resources (i.e., labor, material, and
overhead) are needed to do the work, whether all required resources
will be available when they are needed, and whether any funding or time
constraints exist.
* Establishing the duration of all activities: The schedule should
reflect how long each activity will take to execute. In determining the
duration of each activity, the same rationale, data, and assumptions
used for cost estimating should be used for preparing the schedule.
Further, these durations should be as short as possible and should have
specific start and end dates. Excessively long periods needed to
execute an activity should prompt further decomposition of the activity
so that shorter execution durations will result.
* Integrating schedule activities horizontally and vertically: The
schedule should be horizontally integrated, meaning that it should link
the products and outcomes associated with already sequenced activities
(see previous section). These links are commonly referred to as "hand
offs" and serve to verify that activities are arranged in the right
order to achieve aggregated products or outcomes. The schedule should
also be vertically integrated, meaning that traceability exists among
varying levels of activities and supporting tasks and sub-tasks. Such
mapping or alignment among levels can enable different groups to work
to the same master schedule.
* Establishing the critical path for all activities: Using scheduling
software the critical path--the longest duration path through the
sequenced list of activities--should be identified. The establishment
of a project's critical path is necessary for examining the effects of
any activity slipping along this path. Potential problems that may
occur on or near the critical path should also be identified and
reflected in the scheduling of the time for high-risk activities (see
float below).
* Identifying float between activities: The schedule should identify
float--the time that a predecessor activity can slip before the delay
affects successor activities--so that schedule flexibility can be
determined. As a general rule, activities along the critical path
typically have the least amount of float.
* Conducting a schedule risk analysis: A schedule risk analysis uses a
good critical path method schedule and data about project schedule
risks as well as Monte Carlo simulation techniques to predict the level
of confidence in meeting a project's completion date, the amount of
time contingency needed for a level of confidence, and the
identification of high-priority risks. This analysis should focus not
only on critical path activities but also on other schedule paths that
may become critical. A schedule/cost risk assessment recognizes the
inter-relationship between schedule and cost and captures the risk that
schedule durations and cost estimates may vary due to, among other
things: limited data, optimistic estimating, technical challenges, lack
of qualified personnel, and other external factors. As a result, the
baseline schedule should include a buffer or a reserve of extra time.
Schedule reserve for contingencies should be calculated by performing a
schedule risk analysis. As a general rule, the reserve should be held
by the project manager and applied as needed to those activities that
take longer than scheduled because of the identified risks. Reserves of
time should not be apportioned in advance to any specific activity
since the risks that will actually occur and the magnitude of their
impact is not known in advance.
* Updating the schedule using logic and durations to determine the
dates: The schedule should use logic and durations in order to reflect
realistic start and completion dates for project activities. The
schedule should be continually monitored to determine when forecasted
completion dates differ from the planned dates, which can be used to
determine whether schedule variances will affect downstream work.
Maintaining the integrity of the schedule logic is not only necessary
to reflect true status, but is also required before conducting a
schedule risk analysis. The schedule should avoid logic overrides and
artificial constraint dates that are chosen to create a certain result
on paper. Individuals trained in critical path method scheduling should
be responsible for updating the schedule.
Based on our work, we determined the extent that estimates and
schedules for the three projects we selected met the best practices
criteria.
* Not Met--Project officials provided no evidence that satisfies any of
the criterion,
* Minimally Met--Project officials provided evidence that satisfies a
small portion of the criterion,
* Partially Met--Project officials provided evidence that satisfies
about half of the criterion,
* Substantially Met--Project officials provided evidence that satisfies
a large portion of the criterion, and:
* Met--Project officials provided complete evidence that satisfies the
entire criterion.
We conducted this performance audit from October 2008 through December
2009 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained meets these standards.
[End of section]
Appendix II: Comments from the Department of Veterans Affairs:
The Secretary Of Veterans Affairs:
Washington:
December 7, 2009:
Mr. Terrell G. Dorn:
Director, Physical Infrastructure Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Dorn:
The Department of Veterans Affairs (VA) has reviewed the Government
Accountability Office's (GAO) draft report, VA Construction: VA is
Working to Improve Initial Project Cost Estimates, but Should Analyze
Cost and Schedule Risks (GAO-10-189) and generally agrees with GAO's
conclusions and concurs with GAO's recommendations to the Department.
The report states that cost increases and schedule delays for some of
the projects are attributable to scope changes. It is important to note
that VA followed all applicable laws and congressional notification
requirements during the execution of the projects, and maintained the
integrity and intent of each project as authorized by Congress. Such
projects enable VA facilities to provide a safe and modern environment,
which improves health care delivery services for Veterans.
The enclosure specifically addresses GAO's recommendations and provides
additional comments to the draft report. VA appreciates the opportunity
to comment on your draft report.
Sincerely,
Signed by:
John R. Gingrich:
Chief of Staff:
Enclosure:
[End of letter]
Enclosure:
Department of Veterans Affairs (VA) Comments to Government
Accountability Office (GAO) Draft Report, VA Construction: VA is
Working to Improve Initial Project Cost Estimates, but Should Analyze
Cost and Schedule Risks (GAO-10-189):
GAO Recommendation 1: To improve estimates of the cost of a major
construction project as well as the risks that may influence the cost
and how these risks can be mitigated, GAO recommends that the Secretary
of Veterans Affairs direct the Office of Construction and Facilities
Management (CFM) to conduct a cost risk analysis of major construction
projects.
VA comments to the draft report: Concur. While VA assesses risk in both
cost and schedule, we do not currently perform the type of risk
analysis recommended by GAO. VA will develop a plan and implement GAO's
recommendations by the third quarter of Fiscal Year 2010.
The VA's major construction program for health care has grown
significantly since 2004 when the initial results of the CARES studies
were released. As pointed out in the report, some of the projects were
included in budget requests without the benefit of sufficient planning
and design information to accurately estimate the cost; and many of
these have subsequently experienced cost
growth. The Department has already recognized this problem and is
taking steps to develop a multi-year construction plan that will enable
development of appropriate plans and early designs to ensure that the
budget requests to the Congress more accurately predict the cost.
As pointed out in the report, not all of the cost growth can be
attributed to the lack of prior planning and design. The projects under
review were in design and in some cases in the construction procurement
phase during a time when the construction economy in the United States
was extremely robust. A large amount of work was available for
construction companies in the commercial sector which resulted in a
small number of companies competing for VA contracts. In addition, the
cost of labor and building materials was rising dramatically. This was
exacerbated by the impact on the construction economy of the hurricanes
of 2004 and 2005 ” most notably Katrina.
GAO Recommendation 2: To provide a realistic estimate of when a
construction project may be completed as well as the risks to the
project that could be mitigated, [GAO] recommends that the Secretary of
Veterans Affairs direct CFM to take the following two actions:
* Require the use of an integrated master schedule for all major
construction projects. This schedule should integrate all phases of
project design and construction, and;
* Conduct a schedule risk analysis, when appropriate, based on the
project's cost, schedule, complexity, or other factors. Such a risk
analysis should include a determination of the largest risks to the
project, a plan for mitigating those risks, and an estimate of when the
project will be finished if the risks are not mitigated.
VA comments to the draft report: Concur. VA is pleased the report
highlights that most of the GAO 12 best practices are being met by VA.
However, the report recommends that VA implement two of the best
practices that VA currently does not include in its process. These are
to conduct a cost and schedule risk analysis and maintain an integrated
master schedule. The Department accepts GAO's recommendations and the
Office of Construction and Facilities Management (CFM) will begin to
take steps to implement these process modifications by the third
quarter of Fiscal Year 2010.
[End of section]
Appendix III: Consolidation and Expansion of Medical Centers in
Cleveland, Ohio:
Project Overview:
The major construction project in Cleveland includes consolidating the
Brecksville Veterans Affairs Medical Center and the Wade Park Veterans
Affairs Medical Center, which are 26 miles apart. As part of this
consolidation, a new bed tower is being built at the Wade Park Medical
Center. This bed tower will contain a nursing home and space for
psychiatric patients. The project is divided into two phases. Phase I
includes the construction of an energy center and phase II includes the
construction of a bed tower addition.
Reasons for the Project:
The project was first initiated by the VA under the Capital Asset
Realignment for Enhanced Services (CARES) process in 2004 to save money
through consolidation and to provide better health care for veterans.
[Footnote 23] According to VA officials, the two medical centers
frequently worked together to provide health care for veterans. The
Brecksville medical center was primarily a nursing home care unit and
psychiatric care facility and the Wade Park medical center was
primarily a surgical care facility. According to VA, it was very
expensive to operate and maintain the two physical locations. Patients
needing immediate care at the Brecksville medical center were sometimes
taken to local area hospitals instead of the Wade Park medical center
because of the distance between the two medical centers. Maintaining
the two medical centers resulted in duplication of services, decreased
operational efficiencies, and issues of continuity of care between the
two medical centers. Other inefficiencies included ambulance and
wheelchair van costs and outdated modes of providing health care.
VA also intended for the project to meet rising demand for services in
the Cleveland area and noted that the total number of unique patients
at these 2 medical centers had increased. After considering four
alternatives, the medical center staff determined that consolidating
the two medical centers at Wade Park would lead to better health care
for veterans and provide significant cost savings and other
efficiencies. Specifically, consolidation would allow VA to avoid
approximately $41 million in non-recurring maintenance and
infrastructure improvements at the Brecksville medical center and gain
approximately $10.6 million in operational savings per year.
Project Cost:
The cost estimate to consolidate the two facilities and construct a new
bed tower at Wade Park has remained constant at $102.3 million.
According to VA officials, the cost estimate is still reasonable for
the project through completion. Of the $102.3 million, $15 million was
appropriated in fiscal year 2004 and $87.3 million was appropriated in
fiscal year 2008.[Footnote 24] To keep costs within budget, the VA
closely monitored and reduced the scope of the major construction
project. Some of the work was also shifted to a minor construction
project. The medical center modified the design plans to eliminate 30
beds and one floor from the bed tower. The 30 beds will instead be
relocated in the main hospital where space is being renovated to
accommodate them. The funding for the 30 beds will not come from the
appropriated construction funds. Rather, the 30 beds will be funded out
of non-recurring maintenance (NRM) funds, which can be used to renovate
spaces and purchase equipment needed as a result of that renovation.
Our analysis of how the cost estimate met best practices is in table 4.
Table 4: Extent That Bed Tower Cost Estimate Met Best Practices:
Step One: Define the Estimate's Purpose;
The purpose of a cost estimate is determined by its intended use, and
its intended use determines its scope and detail. Cost estimates have
two general purposes: (1) to help managers evaluate affordability and
performance against plans, as well as the selection of alternative
systems and solutions, and (2) to support the budget process by
providing estimates of the funding required to efficiently execute a
program. The scope of the cost estimate will be determined by such
issues as the time involved, what elements of work need to be
estimated, who will develop the cost estimates, and how much cost
estimating detail will be included. A life cycle cost estimate provides
an exhaustive and structured accounting of all resources and associated
cost elements required to develop, produce, deploy, and sustain a
particular program. As such a life cycle cost estimate encompasses all
past (or sunk), present, and future costs for every aspect of a
program, regardless of funding source. Life cycle costing enhances
decision making, especially in early planning and concept formulation
of acquisition. Design trade-off studies conducted in this period can
be evaluated on a total cost basis as well as on a performance and
technical basis. A life cycle cost estimate can support budgetary
decisions, key decision points, milestone reviews, and investment
decisions. Because they encompass all possible costs, life cycle cost
estimates provide a wealth of information about how much programs are
expected to cost over time. Thus, having full life cycle costs is
important for successfully planning program resources and making wise
decisions.
1. Is the purpose and scope of the cost estimate defined and
documented? Have all costs been estimated, including life cycle costs?
Met; the purpose of the cost estimate is documented and defined at a
level that would enable VA to submit a quality cost estimate. The
purpose of the cost estimate is to be the basis of comparison for the
bids responding to the CARES tower construction request for proposal
(RFP). The scope covers only the construction of the tower; it is not
required to include complete life-cycle costs. The scope of the initial
estimate is defined through a cost estimate steering committee that
included construction cost estimators, architects, and engineers that
were familiar with the design and earlier construction phases. Life
cycle costs are represented in the OMB Exhibit 300. The scope of the
estimate is defined by VA policy. The Manual for Preparation of Cost
Estimates for VA Facilities states that:
1.1.1 - A project estimate shall show the current cost of construction
on the date of the estimate. The estimate should reflect current costs
on the date the estimate is received and anticipated local escalation
to the midpoint of construction, i.e., date of estimate plus half of
construction duration;
1.1.2 - The level of detail for this estimate shall be consistent with
the degree of completeness of the drawings being submitted. Simply
stated, this means that if a construction element is shown, it must be
priced; if it is shown in detail, it must be priced in detail. For
detailed elements, "lump sum" or "allowance" figures will not be
acceptable. Project estimates will include all elements within the
contractor's bid such as insurance, bonds, hazardous abatement and any
other such items.
Step Two: Develop the Estimating Plan:
An analytic approach to cost estimates typically entails a written
study plan detailing a master schedule of specific tasks, responsible
parties, and due dates. Enough time should be scheduled to collect
data, including visits to contractor sites to further understand the
strengths and limitations of the data that have been collected. If
there is not enough time, then the schedule constraint should be
clearly identified in the ground rules and assumptions, so that
management understands the effect on the estimate's quality and
confidence.
2. Did the team develop a written study plan?
Met; the estimating team is from a centralized cost estimating firm
that specializes in hospital construction and the estimate follows cost
estimate preparation guidance published by the VA. CFM publishes
guidance on preparing cost estimates that details how construction cost
estimates should be created, structured, and presented. The manual also
explains roles and responsibilities, units of measure, and guidance on
master specifications. The CARES tower cost estimate was created by an
independent consultant to the architect as directed by VA contractual
requirements. The consulting estimating firm specializes in major
construction cost estimates, particularly hospital construction.
Officials stated that the cost estimators have extensive experience in
the regional marketplace and in creating estimates for high-cost
medical centers. Senior cost estimators for the project have 30 years
of experience estimating construction costs and are members of a
professional cost engineering society. As outlined in the Manual for
Preparation of Cost Estimates for VA Facilities:
1.1.1 - A project estimate shall show the current cost of construction
on the date of the estimate. The estimate should reflect current costs
on the date the estimate is received and anticipated local escalation
to the midpoint of construction, i.e., date of estimate plus ½ of
construction duration;
1.1.2 - The level of detail for this estimate shall be consistent with
the degree of completeness of the drawings being submitted. Simply
stated, this means that if a construction element is shown, it must be
priced; if it is shown in detail, it must be priced in detail. For
detailed elements, "lump sum" or "allowance" figures will not be
acceptable. Project estimates will include all elements within the
contractor's bid such as insurance, bonds, hazardous abatement and any
other such items.
Step Three: Define the Program Characteristics:
Key to developing a credible estimate is having an adequate
understanding of the acquisition program--the acquisition strategy,
technical definition, characteristics, system design features, and
technologies to be included in its design. The cost estimator can use
this information to identify the technical and program parameters that
will bind the cost estimate. The amount of information gathered
directly affects the overall quality and flexibility of the estimate.
Less information means more assumptions must be made, increasing the
risk associated with the estimate. Therefore, the importance of this
step must be emphasized, because the final accuracy of the cost
estimate depends on how well the program is defined.
3. Is there a documented technical baseline description?
Met; the detailed architectural drawings, which served as the technical
baseline for the estimate, were continually updated to reflect the
latest design changes. The master architect plan was used as a
technical baseline for the estimate. The master plan was created by
consultants to the architect/engineering (A/E) firm. The plan consisted
of four volumes of design information that was used as the basis for
the cost estimate. The cost estimate was developed as the design was
changing. Officials noted that, ideally, architectural changes would be
sent to the cost estimator 2 to 3 weeks in advance to give estimators
time to update the estimate. However, because VA required the drawings
and the estimate due at the same time, estimators only had 9 days to
update the estimate for changes. Officials stated the technical
baseline went through a fact-check to make sure all changes were
incorporated. Other technical baseline documents to be referenced in
the development of a VA cost estimate are defined by VA policy. These
documents, listed and defined in The Manual for Preparation of Cost
Estimates for VA Facilities, include Practice Design Manuals, Master
Specifications, Architect/Engineer Checklists, Design and Quality
Alerts, Design Guides, Design and Construction Procedures, Physical
Security Design Manuals, and Technical Summaries. The Cost Estimate
Manual also includes the cost breakdown categories to be used in the
estimate.
Step Four: Determine the Estimating Structure:
A work breakdown structure (WBS) is the cornerstone of every program
because it defines in detail the work necessary to accomplish a
program's objectives. A WBS is a valuable communication tool between
systems engineering, program management, and other functional
organizations because it provides a clear picture of what needs to be
accomplished and how the work will be done. Accordingly, it is an
essential element for identifying activities in a program's integrated
master schedule and it provides a consistent framework for planning and
assigning responsibility for the work. Initially set up when the
program is established, the WBS becomes successively detailed over time
as more information because known about the program. A WBS deconstructs
a program's end product into successive levels with smaller specific
elements until the work is subdivided to a level suitable for
management control. By breaking the work down into smaller elements,
management can more easily plan and schedule the program's activities
and assign responsibility for the work. It also facilitates
establishing a schedule, cost, and earned value management (EVM)
baseline. Establishing a product-oriented WBS is a best practice
because it allows a program to track cost and schedule by defined
deliverables, such as a hardware or software component. This allows a
program manager to more precisely identify which components are causing
cost or schedule overruns and to more effectively mitigate the root
cause of the overruns.
4. Is there a defined WBS and/or cost element structure?
Met; the estimate clearly describes how the various sub-elements are
summed to produce the amounts for each cost category, thereby ensuring
that all pertinent costs are included and no costs are double counted.
While the WBS is not considered product-oriented by program officials,
the breakout of work is based on a required VA element structure. The
WBS is based on the standardized WBS on VA form HO-18B/C. Both the
architect and cost estimators are required to use this format. The WBS
breaks the construction costs into standardized systems such as
foundation, substructure, superstructure, and roofing, as well as
subsystems such as slab on grade, stair construction, and elevators.
These system descriptions are also used in the schedule. The HO-18 WBS
elements are defined in the Manual for Preparation of Cost Estimates
for VA Facilities by CFM.
Step Five: Identify Ground Rules and Assumptions:
Cost estimates are typically based on limited information and therefore
need to be bound by the constraints that make estimating possible.
These constraints usually take the form of assumptions that bind the
estimate's scope, establishing baseline conditions the estimate will be
built from. Ground rules represent a common set of agreed on estimating
standards that provide guidance and minimize conflicts in definitions.
Without firm ground rules, the analyst is responsible for making
assumptions that allow the estimate to proceed. Assumptions represent a
set of judgments about past, present, and future conditions postulated
as true in the absence of positive proof. The analyst must ensure that
assumptions are not arbitrary, that they are founded on expert
judgments rendered by experienced program and technical personnel. Many
assumptions profoundly influence cost; the subsequent rejection of even
a single assumption by management could invalidate many aspects of the
estimate. Therefore, it is imperative that cost estimators brief
management and document all assumptions well, so that management fully
understands the conditions the estimate was structured on. Failing to
do so can lead to overly optimistic assumptions that heavily influence
the overall cost estimate, to cost overruns, and to inaccurate
estimates and budgets.
5. Are there defined ground rules and assumptions that document the
rationale and any historical data to back up any claims?
Met; cost-influencing ground rules and assumptions, such as the
programs schedule, labor rates, and inflation rates are documented, and
market surveys were conducted to describe variability in material and
labor prices. Assumptions and ground rules are documented and are
included within the cost estimate. The ground rules and assumptions
were created by the independent cost estimating firm and vetted with VA
engineers and architects at the cost estimate steering committee
meetings. Officials stated that the assumptions regarding escalation
rates were particularly hard to agree upon between members of the
steering committee. At the time, hurricane Katrina had recently struck
and material prices were volatile. The final escalation rates were
based off independent material price research performed by the cost
estimating firm, which were nearly double the original rates proposed
by the architect. In addition, the cost estimate includes market survey
information to describe the volatility in labor and material prices and
the risk of obtaining sufficient labor resources for the project.
Step Six: Obtain the Data:
Data are the foundation of every cost estimate. How good the data are
affects the estimate's overall credibility. Depending on the data
quality, an estimate can range anywhere from a mere guess to a highly
defensible cost position. Credible cost estimates are rooted in
historical data. Rather than starting from scratch, estimators usually
develop estimates for new programs by relying on data from programs
that already exist and adjusting for any differences. Thus, collecting
valid and useful historical data is a key step in developing a sound
cost estimate. The challenge in doing this is obtaining the most
applicable historical data to ensure that the new estimate is as
accurate as possible. One way of ensuring that the data are applicable
is to perform checks of reasonableness to see if the results are
similar. Different data sets converging toward one value provides a
high degree of confidence in the data.
6. Were the data gathered from historical actual cost, schedule, and
program and technical sources?
Met; cost estimators used actual costs from similar programs,
incorporated vendor quotes, and conducted market surveys to develop
material and labor price estimates. Officials stated that historical
data were the foundation of the estimate and were based on experience
by the cost estimating firm on similar high-profile hospital estimates.
The cost estimators leveraged their experience and data from these
previous projects to estimate the costs of the VA tower. The initial
concept estimates were based primarily on verbal discussions with
vendors. As the designs were finalized and the estimate took shape,
actual paper quotes for labor and material were submitted by vendors;
these were and assessed for appropriateness and realism by the cost
estimators. In addition, a market survey was delivered as part of the
final cost estimate. The market survey describes the volatility in
labor and material prices and the risk of obtaining sufficient labor
resources for the project.
Step Seven: Develop the Point Estimate and Compare It to an Independent
Cost Estimate:
Step 7 pulls all the information together to develop the point
estimate--the best guess at the cost estimate, given the underlying
data. High-quality cost estimates usually fall within a range of
possible costs, the point estimate being between the best and worst
case extremes. The cost estimator must perform several activities to
develop a point estimate: develop the cost model by estimating each WBS
element, using the best methodology, from the data collected; include
all estimating assumptions in the cost model; express costs in constant-
year dollars; time-phase the results by spreading costs in the years
they are expected to occur, based on the program schedule; and add the
WBS elements to develop the overall point estimate. Having developed
the overall point estimate, the cost estimator must then validate it by
thoroughly understanding and investigating how the cost model was
constructed. For example, all WBS cost estimates should be checked to
verify that calculations are accurate (no double counting) and account
for all costs, including indirect costs. Moreover, proper escalation
factors should be used to inflate costs so that they are expressed
consistently and accurately. Finally, the cost estimator should compare
the cost estimate against the independent cost estimate and examine
where and why there are differences; perform cross-checks on cost
drivers to see if results are similar; and update the model as more
data become available or as changes occur and compare the results
against previous estimates.
7. Did the cost estimator consider various cost estimating methods like
analogy, engineering build up, parametric, extrapolating from actual
costs, and expert opinion (if none of the other methods can be used)?
Met; the cost estimate is based on a detailed engineering buildup
methodology using estimated labor and material prices, and crosschecked
against an independent unit-cost level assessment. The estimate was
vetted through experts to ensure costs were appropriately captured. The
construction cost estimate is based on engineering buildup of vendor
quotes for material and labor dollars. Due to hurricane Katrina, the
prices of copper and steel were especially volatile. The cost
estimating firm conducted its own market research into material prices
to create escalation rates. Moreover, officials stated that contingency
factors for the estimate were tailored to the building and the
construction situation rather than employing standard rules-of-thumbs.
Officials stated that an independent cost estimate was not performed by
either the VA or the architect. However, a consultant to the cost
estimating firm did perform an assessment, and the two estimates were
reconciled. Officials stated that the assessment was performed using a
high-level unit cost methodology, which was compared to the original
cost estimate's bottom-up engineering methodology. The draft estimates
were created in spreadsheets and reviewed multiple times by senior cost
estimators. Officials stated that the cost estimate is reviewed
multiple times for errors internally because the estimate must meet
requirements imposed by insurance companies (referred to as
"professional liability"). Moreover, the estimate was reviewed
periodically by the cost estimate steering committee. Officials noted
that the largest pitfall to the VA estimating process is that the
budget is already set far in advance of the cost estimate. The cost
would be estimated independently, and if the price exceeded the
budgeted amount, the cost estimators worked with the engineers and
architects to reduce or eliminate costs through the value engineering
process.
Step Eight: Conduct a Sensitivity Analysis:
Sensitivity analysis should be included in all cost estimates because
it examines the effects of changing assumptions and ground rules. Since
uncertainty cannot be avoided, it is necessary to identify the cost
elements that represent the most risk and, if possible, cost estimators
should quantify the risk using both a sensitivity and uncertainty (see
step 9) analysis. In order for sensitivity analysis to reveal how the
cost estimate is affected by a change in a single assumption, the cost
estimator must examine the effect of changing one assumption or cost
driver at a time while holding all other variables constant. By doing
so, it is easier to understand which variable most affects the cost
estimate.
8. Did the cost estimate included a sensitivity analysis that
identified using a range of possible costs the effects of changing key
cost driver assumptions or factors?
Partially met; while a sensitivity analysis was not conducted by the
VA, the estimate identifies volatility in material and labor costs and
utilizes conservative escalation rates. Program officials stated that a
sensitivity analysis was not performed on the estimate. However, they
noted that the estimate utilized conservative escalation rates because
officials were well aware of the consequences of underestimated
material costs. In addition, the cost estimate includes market survey
information to describe the volatility in labor and material prices and
the risk of obtaining sufficient labor resources for the project. Given
the volatility of material prices at the time--officials stated that
steel was anywhere from $2,200 to $4,500 a ton--a sensitivity analysis
on escalation rates would formally document the sensitivity of the
overall estimate to small or large changes in material prices.
Step Nine: Conduct a Risk and Uncertainty Analysis:
Because cost estimates predict future program costs, uncertainty is
always associated with them. Moreover, a cost estimate is usually
composed of many lower-level WBS elements, each of which comes with its
own source of error. Once these elements are added together, the
resulting cost estimate can contain a great deal of uncertainty. Risk
and uncertainty refer to the fact that because a cost estimate is a
forecast, there is always a chance that the actual cost will differ
from the estimate. A lack of knowledge about the future is only one
possible reason for the difference. Another equally important reason is
the error resulting from historical data inconsistencies, assumptions,
cost estimating equations, and factors typically used to develop an
estimate. In addition, biases are often found in estimating program
costs and developing program schedules. The biases may be cognitive--
often based on estimators' inexperience--or motivational, where
management intentionally reduces the estimate or shortens the schedule
to make the project look good to stakeholders. Recognizing the
potential for error and deciding how best to quantify it is the purpose
of risk and uncertainty analysis. Since cost estimates are uncertain,
making good predictions about how much funding a program needs to be
successful is difficult. In a program's early phases, knowledge about
how well technology will perform, whether the estimates are unbiased,
and how external events may affect the program is imperfect. For
management to make good decisions, the program estimate must reflect
the degree of uncertainty, so that a level of confidence can be given
about the estimate. Quantitative risk and uncertainty analysis provide
a way to assess the variability in the point estimate. Using this type
of analysis, a cost estimator can model such effects as schedules
slipping, missions changing, and proposed solutions not meeting user
needs, allowing for a known range of potential costs. Having a range of
costs around a point estimate is more useful to decision makers,
because it conveys the level of confidence in achieving the most likely
cost and also informs them on cost, schedule, and technical risks.
9. Was a risk and uncertainty analysis conducted that quantified the
imperfectly understood risks and identified the effects of changing key
cost driver assumptions and factors?
Not met; a risk and uncertainty analysis was not conducted so that a
level of confidence about the estimate could be determined. Program
officials stated that an uncertainty analysis was not performed on the
estimate. Given the volatility of material prices at the time--
officials stated that steel was anywhere from $2,200 to $4,500 a ton
and copper was falling--an uncertainty analysis on labor and material
price estimates would formally document the risks and uncertainty of
the overall estimate.
Step Ten: Document the Estimate:
Documentation provides total recall of the estimate's detail so that it
can be replicated by someone other than those who prepared it. It also
serves as a reference to support future estimates. Documenting the cost
estimate makes available a written justification showing how it was
developed and aiding in updating it as key assumptions change and more
information becomes available. Estimates should be documented to show
all parameters, assumptions, descriptions, methods, and calculations
used to develop a cost estimate. A best practice is to use both a
narrative and cost tables to describe the basis for the estimate, with
a focus on the methods and calculations used to derive the estimate.
With this standard approach, the documentation provides a clear
understanding of how the cost estimate was constructed. Moreover, cost
estimate documentation should explain why particular methods and data
sets were chosen and why these choices are reasonable. It should also
reveal the pros and cons of each method selected. Finally, there should
be enough detail so that the documentation serves as an audit trail of
backup data, methods, and results, allowing for clear tracking of a
program's costs as it moves through its various life-cycle phases.
10. Did the documentation describe the cost estimating process, data
sources, and methods step by step so that a cost analyst unfamiliar
with the program could understand what was done and replicate it?
Partially met; while the documentation for the most part provided
detailed material and labor build up, we were not able to trace the
data back based on the documentation alone. While officials stated that
the estimate was based off data from previous estimates, the cost
estimate documentation delivered to VA does not trace estimated values
to raw or normalized data. For instance, the delivered cost estimate
documentation does not provide a basis or supporting data for included
bidding contingency, markup, or escalation rates that would allow an
analyst unfamiliar with the project to recreate them.
Step Eleven: Present Estimate to Management for Approval:
A cost estimate is not considered valid until management has approved
it. Since many cost estimates are developed to support a budget request
or make a decision between competing alternatives, it is vital that
management is briefed on how the estimate was developed, including
risks associated with the underlying data and methods. Therefore, the
cost estimator should prepare a briefing for management with enough
detail to easily defend the estimate by showing how it is accurate,
complete, and high in quality. The briefing should present the
documented life-cycle cost estimate with an explanation of the
program's technical and program baseline.
11. Was there a briefing to management that included a clear
explanation of the cost estimate so as to convey its level of
competence?
Met; the estimate was approved by internal management and vetted
through a cost estimating steering committee consisting of project
engineers and architects. The draft estimates were created in
spreadsheets and reviewed multiple times by senior cost estimators.
Officials stated that the cost estimate is reviewed multiple times for
errors internally because the estimate must meet requirements imposed
by insurance companies (referred to as "professional liability").
Moreover, the estimate was periodically vetted through VA engineers and
architects familiar with the project at cost estimate steering
committee meetings.
Step Twelve: Update the Estimate to Reflect Actual Costs and Changes:
The cost estimate should be regularly updated to reflect all changes.
Not only is this a sound business practice; it is also a requirement
outlined in OMB's Capital Programming Guide. The purpose of updating
the cost estimate is to check its accuracy, defend the estimate over
time, shorten turnaround time, and archive cost and technical data for
use in future estimates. After the internal agency and congressional
budgets are prepared and submitted, it is imperative that cost
estimators continue to monitor the program to determine whether the
preliminary information and assumptions remain relevant and accurate.
Keeping the estimate fresh gives decision makers accurate information
for assessing alternative decisions. Cost estimates must also be
updated whenever requirements change, and the results should be
reconciled and recorded against the old estimate baseline. The
documented comparison between the current estimate (updated with actual
costs) and old estimate allows the cost estimator to determine the
level of variance between the two estimates. In other words, it allows
estimators to see how well they are estimating and how the program is
changing over time.
12. Is there a process for the estimating team to update the estimate
with actual costs as it becomes available?
Not met; the VA does not require the cost estimating firm to update the
construction cost estimate with actual costs once the project is
underway. Officials from the cost estimating firm stated that while
"adds and deducts" were inserted into the estimate as the design
changed, the estimate is not updated once construction begins.
Officials stated that they are not privy to actual costs incurred by VA
general contractors, but that they wish they were in order to check the
accuracy of their estimates. Regardless of what type of contract or
what organization is managing costs, the purpose of updating the cost
estimate is to check its accuracy, defend the estimate over time,
shorten turnaround time of future estimates, and archive cost and
technical data for use in future estimates.
Source: GAO analysis of VA information.
[End of table]
Project Schedule and Changes in Scope:
The project was originally a one-phase project and scheduled to be
completed in September 2008 but is now a two-phase project and is
scheduled to be completed in February 2011. Before construction began,
the project was broken into two phases because there was insufficient
power capacity to keep the existing hospital functioning while the
construction was being completed. As a result, an energy center was
added to the design plan and its construction was separated from that
of the bed tower. In addition, a property acquisition that took longer
than expected delayed the project schedule by nine months. Part of the
land that the bed tower is being built on was donated to the City of
Cleveland for use as parkland. The acquisition process was prolonged
because the City had to change the use of the donated land before the
VA could begin construction. Phasing the project and the delayed
property acquisition fostered a change in scope of the project and the
project's original completion date was moved from September 1, 2008, to
November 9, 2010. The projected completion date was again extended to
February 1, 2011, due to unforeseen site conditions. Specifically,
during the construction of the bed tower, crews discovered and had to
move a sewer line before they could continue. According to VA
officials, February 1, 2011, is still the projected date for project
completion. However, it was not possible for us to determine if the
completion date is reasonable because the project's construction
schedule has not undergone a schedule risk analysis. We have identified
a schedule risk analysis as a best practice in scheduling. As of August
2009, VA has completed the energy center and is constructing the bed
tower addition.
The construction schedule for this project generally followed best
practices but, as stated, did not include a schedule risk analysis.
Specifically, while the schedule met eight of nine scheduling best
practices, the schedule did not undergo a risk analysis to determine
the major risks to the schedule and the likelihood of the project being
completed on time. Our analysis of how the schedule met best practices
is in table 5.
Table 5: Extent That Bed Tower Construction Schedule Met Best
Practices:
Best practice: Capturing activities;
Explanation: The schedule should reflect all activities as defined in
the project's work breakdown structure, which defines in detail the
work necessary to accomplish a project's objectives, including
activities to be performed by both the owner and contractors;
Met?: Met;
GAO analysis: The schedule is required by contract to include
approximately 2,500 activities in order to sufficiently detail the
level of work required (the actual schedule has 2,725, approximately 75
detail activities per milestone). Each activity is mapped to an
activity ID number, building area, and work trade, which allows the
scheduler to quickly filter the schedule by type of work or
subcontractor. The schedule is reviewed by the VA CFM for completeness
to ensure all necessary activities and milestones are included.
Best practice: Sequencing activities;
Explanation: The schedule should be planned so that critical project
dates can be met. To meet this objective, activities need to be
logically sequenced--that is, listed in the order in which they are to
be carried out. In particular, activities that must be completed before
other activities can begin (predecessor activities), as well as
activities that cannot begin until other activities are completed
(successor activities), should be identified. This helps ensure that
interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be established and used as a
basis for guiding work and measuring progress;
Met?: Met;
GAO analysis: All detail activities and milestones are properly
sequenced. Out of 2,378 remaining detail activities, we found only 2
activities that were not properly driving the start date of a
predecessor activity. There are no lags, hard constraints, or soft
constraints in the schedule, as required by contract specifications.
Officials stated that, if the project runs late, the VA requires the
baseline schedule to show the slip in the finish milestone (as opposed
to constraining the finish milestone and simply reporting negative
float). The VA also requires a diagram of the schedule network, similar
to a PERT diagram, that clearly displays the relationships between
tasks.
Best practice: Assigning resources to activities;
Explanation: The schedule should reflect what resources (e.g., labor,
materials, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist;
Met?: Met;
GAO analysis: The VA requires schedules to be cost loaded with prorated
overhead and profit, and the total price loaded into the schedule must
equal the total contract price. Each detail activity has an associated
manpower requirement.
Best practice: Establishing the duration of activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, historical data, and assumptions used for
cost estimating should be used. Durations should be as short as
possible and have specific start and end dates. The schedule should be
continually monitored to determine when forecasted completion dates
differ from planned dates; this information can be used to determine
whether schedule variances will affect subsequent work;
Met?: Met;
GAO analysis: As required by VA schedule contract specifications,
activity durations are 20 days or less, except for procurement
activities. Our analysis shows the median task duration is 10 days.
Less than 1% of the remaining activities are 1 day in duration.
Activity durations are estimated using input from subcontractors who
will be performing the work. All activities are based on a standard 5-
day workweek with holidays.
Best practice: Integrating activities horizontally and vertically;
Explanation: The schedule should be horizontally integrated, meaning
that it should link products and outcomes associated with other
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that the dates for
starting and completing activities in the integrated master schedule
should be aligned with the dates for supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to work
to the same master schedule;
Met?: Met;
GAO analysis: Our analysis shows the schedule to be horizontally
integrated due to the high number of straightforward finish-start
links, realistic float, and valid critical path. The schedule is
vertically integrated, with all activities subsumed under organized
higher levels. Each activity is mapped to an area and trade, clearly
indicating which subcontractor is responsible for what work in each
area at any time.
Best practice: Establishing the critical path for activities;
Explanation: Scheduling software should be used to identify the
critical path, which represents the chain of dependent activities with
the longest total duration. Establishing a project's critical path is
necessary to examine the effects of any activity slipping along this
path. Potential problems along or near the critical path should also be
identified and reflected in scheduling the duration of high-risk
activities;
Met?: Met;
GAO analysis: Officials stated the critical path is calculated by the
scheduling software and is a crucial tool for managing the construction
project. The critical path and activities near the critical path (the
"hot list") are reviewed in management meetings on a monthly basis. Our
analysis shows the critical path to be structurally sound, running the
length of the schedule and encompassing several major milestones.
Best practice: Identifying the float between activities;
Explanation: The schedule should identify the float--the amount of time
by which a predecessor activity can slip before the delay affects
successor activities--so that a schedule's flexibility can be
determined. As a general rule, activities along the critical path have
the least float. Total float is the total amount of time by which an
activity can be delayed without delaying the project's completion (if
everything else goes according to plan);
Met?: Met;
GAO analysis: Total float represents the amount of time an activity can
slip before it affects the project finish date. It is therefore a
crucial tool for resource allocation and risk mitigation. There appear
to be excessive values of total float in the schedule, but officials
stated that they were not concerned with these float values. Officials
told us that in a construction project, many tasks can be performed in
any order, and the only float values of real concern for management was
float on or near the critical path.
Best practice: Conducting a schedule risk analysis;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
project's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can affect the project's status;
Met?: Not met;
GAO analysis: The program has not performed a schedule risk analysis
(SRA). Officials stated that they see value in an SRA particularly if
it is performed very early in the program, for example, during the OMB
300 budget request procedure. However, best practices suggest that even
at the construction bid phase, an SRA can be used to determine a level
of confidence in meeting the completion date or whether proper reserves
have been incorporated into the schedule. An SRA will calculate
schedule reserve, which can be set aside for those activities
identified as high-risk. Without this reserve, the program faces the
risk of delays to the scheduled completion date if any delays were to
occur on critical path activities.
Best practice: Updating the schedule using logic and durations to
determine dates;
Explanation: [Empty];
Met?: Met;
GAO analysis: The schedule is reviewed and updated formally in monthly
management meetings. During these meetings, officials update and
examine actual start and finish dates and remaining durations of
activities. The schedule is manually updated through "progress
reporting." During progress reporting, the general contractor
physically walks through the job site and updates the remaining
durations on each ongoing activity. Our analysis found no date
anomalies in the schedule. Date anomalies are errors such as actual
finish dates in the future, outdated tasks that have no actual start
date, and completed tasks in the past with no actual finish date.
Source: GAO analysis of VA information.
[End of table]
[End of section]
Appendix IV: Construction of Spinal Cord Injury/Disease Center in
Syracuse, New York:
Project Overview:
This project includes the construction of a 30-bed center for treating
spinal cord injuries to be attached to the current VA medical center in
Syracuse, New York. The project also includes adding two levels to the
current parking garage. The project is divided into two phases. Phase I
includes the addition on the parking garage and Phase II includes the
construction of the Spinal Cord Injury/Disease (SCI/D) center.
Reasons for the Project:
VA initiated this project under the Capital Asset Realignment for
Enhanced Services (CARES) process in February 2004 because the
Veterans' Integrated Service Network (VISN) did not have the ability to
treat acute spinal cord injuries. Syracuse had the only in-patient
rehabilitation unit and SCI/D expertise within the VISN; so, VA decided
to put the new SCI/D center in Syracuse.
Project Cost:
The project cost has increased from the original estimate submitted to
Congress of $53.9 million to $84,969,000 (an increase of 58 percent).
According to VA officials in Syracuse, this estimate was developed in
about 6 weeks and was based on the total square footage required
multiplied by the cost per square foot of new construction. Congress
authorized $53.9 million for the project in 2004[Footnote 25] and
appropriated about $53.4 million in funds in the Consolidated
Appropriations Act for FY 2005.[Footnote 26]
According to VA officials in Syracuse, the main reason for the cost
increase is that the initial estimate did not fully consider several
factors. The original design of a new SCI/D center did not include
money for additional parking. However, when the project had been
approved by Congress and was in design, VA officials in Syracuse
commissioned a study to examine future parking needs at the Syracuse
medical center. The study concluded that, based on the new SCI/D center
and projected demand from patients and staff, there should be an
additional 429 to 528 parking spaces at the medical center. As a result
of this study, VA officials in Syracuse decided to add two floors onto
the existing parking garage at an estimated cost of $10 million.
In addition to parking, stakeholders identified needed scope changes in
the project. Specifically, the Paralyzed Veterans of America (PVA)
insisted that there be a dedicated entrance from the parking garage to
the SCI/D center, which is being built on the 4th floor of the medical
center. This dedicated entrance would allow veterans with spinal cord
injuries to enter the center directly from the parking garage, without
requiring the veterans to go down to the street from the parking
garage, outside to the main entrance of the medical center, then up to
the 4th floor of the medical center for treatment. According to VA
staff in Syracuse, VA agreed to make changes that would improve access
to the facility, and this increased the cost of the project and delayed
the project's schedule. As a result of these changes to the project's
scope, VA received an additional $23.8 million from Congress in fiscal
year 2008.[Footnote 27] Our analysis of how the cost estimate for the
SCI/D center met best practices is in table 6.
Table 6: Extent That SCI/D Center Cost Estimate Met Best Practices:
Step One: Define the Estimate's Purpose:
The purpose of a cost estimate is determined by its intended use, and
its intended use determines its scope and detail. Cost estimates have
two general purposes: (1) to help managers evaluate affordability and
performance against plans, as well as the selection of alternative
systems and solutions, and (2) to support the budget process by
providing estimates of the funding required to efficiently execute a
program. The scope of the cost estimate will be determined by such
issues as the time involved, what elements of work need to be
estimated, who will develop the cost estimates, and how much cost
estimating detail will be included. A life-cycle cost estimate provides
an exhaustive and structured accounting of all resources and associated
cost elements required to develop, produce, deploy, and sustain a
particular program. As such a life cycle cost estimate encompasses all
past (or sunk), present, and future costs for every aspect of a
program, regardless of funding source. Life-cycle costing enhances
decision making, especially in early planning and concept formulation
of acquisition. Design trade-off studies conducted in this period can
be evaluated on a total cost basis as well as on a performance and
technical basis. A life-cycle cost estimate can support budgetary
decisions, key decision points, milestone reviews, and investment
decisions. Because they encompass all possible costs, life cycle cost
estimates provide a wealth of information about how much programs are
expected to cost over time. Thus, having full life cycle costs is
important for successfully planning program resources and making wise
decisions.
1. Is the purpose and scope of the cost estimate defined and
documented? Have all costs been estimated, including life cycle costs?
Substantially Met; the purpose of the cost estimate is documented and
clearly defined at a level that would enable VA Syracuse to submit a
quality cost estimate: however, the cost estimate does not cover the
full life cycle and therefore does not account for all costs. The
purpose of the VA cost estimate is to support the basis for the budget
request for the Syracuse Spinal Cord Injury/Disease (SCI/D) center. As
they work through the process they further refined the estimate. VA
Syracuse did not do a life-cycle cost estimate (LCCE), they only
addressed design and construction in their estimate. In the OMB Exhibit
300, VA only showed costs for the acquisition base year and 3
additional years. No costs were reported for maintenance. The scope of
the estimate is defined by VA policy. In the Manual for Preparation of
Cost Estimates for VA Facilities (00CFM1B) June 2007, page 1, sections
1.1.1 - 1.1.2 says:
1.1.1 - A project estimate shall show the current cost of construction
on the date of the estimate. The estimate should reflect current costs
on the date the estimate is received and anticipated local escalation
to the midpoint of construction, i.e., date of estimate plus ½ of
construction duration;
1.1.2 - The level of detail for this estimate shall be consistent with
the degree of completeness of the drawings being submitted. Simply
stated, this means that if a construction element is shown, it must be
priced; if it is shown in detail, it must be priced in detail. For
detailed elements, "lump sum" or "allowance" figures will not be
acceptable. Project estimates will include all elements within the
contractor's bid such as insurance, bonds, hazardous abatement, and any
other such items.
Step Two: Develop the Estimating Plan:
An analytic approach to cost estimates typically entails a written
study plan detailing a master schedule of specific tasks, responsible
parties, and due dates. Enough time should be scheduled to collect
data, including visits to contractor sites to further understand the
strengths and limitations of the data that have been collected. If
there is not enough time, then the schedule constraint should be
clearly identified in the ground rules and assumptions so that
management understands the effect on the estimate's quality and
confidence.
2. Did the team develop a written study plan?
Met; the cost estimator followed the process for developing the
estimate as outlined in the Manual for Preparation of Cost Estimates
for VA Facilities (00CFM1B) June 2007. The Phoenix Engineering staff
who worked on the estimate is very experienced. The cost estimator has
worked in the construction industry for 30 years and has been doing
cost estimates for the past 7 years. The senior electrical estimator
has 40 years of experience in construction. Most of the cost
estimators' team came from the construction industry. The team follows
a systematic approached outlined in the Manual for Preparation of Cost
Estimates for VA Facilities (00CFM1B) June 2007, page 1, sections 1.1.1
- 1.1.2, which says:
1.1.1 - A project estimate shall show the current cost of construction
on the date of the estimate. The estimate should reflect current costs
on the date the estimate is received and anticipated local escalation
to the midpoint of construction, i.e., date of estimate plus half of
construction duration;
1.1.2 - The level of detail for this estimate shall be consistent with
the degree of completeness of the drawings being submitted. Simply
stated, this means that if a construction element is shown, it must be
priced; if it is shown in detail, it must be priced in detail. For
detailed elements, "lump sum" or "allowance" figures will not be
acceptable. Project estimates will include all elements within the
contractor's bid such as insurance, bonds, hazardous abatement, and any
other such items. Submission requirements are indicated in the VA Cost
Estimating Guide.
Step Three: Define the Program Characteristics:
Key to developing a credible estimate is having an adequate
understanding of the acquisition program--the acquisition strategy,
technical definition, characteristics, system design features, and
technologies to be included in its design. The cost estimator can use
this information to identify the technical and program parameters that
will bind the cost estimate. The amount of information gathered
directly affects the overall quality and flexibility of the estimate.
Less information means more assumptions must be made, increasing the
risk associated with the estimate. Therefore, the importance of this
step must be emphasized, because the final accuracy of the cost
estimate depends on how well the program is defined.
3. Is there a documented technical baseline description?
Met; a technical baseline has been documented that includes
requirements, purpose, and system design features; Phoenix engineering,
the cost estimating firm, worked with QPK, the architectural and
engineering firm, to obtain design specifications and clarifications
when needed for the estimates. QPK provided draft copies of each of the
submissions to the VA 2 weeks before the schematic design and design
development milestones were reached. Both the medical center staff and
central office staff commented on the design before it was released.
For example, there were questions about the specifications for the
depth of the caissons in the amount of concrete needed. The caissons
are the underground supports for the columns in the garage. They are
large diameter holes in the ground filled with concrete and rebar that
in some cases, go as deep as 30 feet. QPK reviewed the estimate before
it was released to the VA. This is an iterative process that is defined
by the VA Project Guide. The Manual for Preparation of Cost Estimates
for VA Facilities (00CFM1B) June 2007, sections 1.4 - 1.4.2.3 outlines
the technical resources and their descriptions. In addition, the VA has
set of master specifications based on the in Construction Specification
Institute format. Midstream during the estimating process the VA
switched to the newer CSI format which has 50 divisions versus the
previous 16 divisions. The most recent cost estimate is based on the VA
Manual, which discusses foundation, frame, floor structure, etc., which
is standard in the construction industry.
Step Four: Determine the Estimating Structure:
A work breakdown structure (WBS) is the cornerstone of every program
because it defines in detail the work necessary to accomplish a
program's objectives. A WBS is a valuable communication tool between
systems engineering, program management, and other functional
organizations because it provides a clear picture of what needs to be
accomplished and how the work will be done. Accordingly, it is an
essential element for identifying activities in a program's integrated
master schedule and it provides a consistent framework for planning and
assigning responsibility for the work. Initially set up when the
program is established, the WBS becomes successively detailed over time
as more information because known about the program. A WBS deconstructs
a program's end product into successive levels with smaller specific
elements until the work is subdivided to a level suitable for
management control. By breaking the work down into smaller elements,
management can more easily plan and schedule the program's activities
and assign responsibility for the work. It also facilitates
establishing a schedule, cost, and earned value management (EVM)
baseline. Establishing a product-oriented WBS is a best practice
because it allows a program to track cost and schedule by defined
deliverables, such as a hardware or software component. This allows a
program manager to more precisely identify which components are causing
cost or schedule overruns and to more effectively mitigate the root
cause of the overruns.
4. Is there a defined WBS and/or cost element structure?
Met; the estimate describes how the various sub-elements are summed to
produce the amounts for each cost category, thereby ensuring that all
pertinent costs are included and no costs are double counted. The
backbone of the VA work breakdown structure is based on the
Construction Specifications Institute (CSI). The VA has a master
specification based on 16 divisions of costs from CSI. Midstream during
the estimating process the VA switched to the newer CSI format, which
has 50 divisions versus the previous 16 divisions. The latest estimate
that Phoenix engineering prepared was made against the VA manual
specifications. This is different from what the construction industry
has. It is VA's attempt to track costs for the future. The VA guide
discusses foundations, frames, floor structure etc. The cost estimate
WBS is product oriented to the extent possible and it is at a level of
detail to ensure that costs are neither omitted nor double counted. The
WBS is in a standard format and is consistently used. The WBS is also
used by the contractor for cost estimates and scheduling. A view of the
work breakdown structure down to at least level 3 is shown below:
Base Building--Spinal Cord Injury Center;
A. Foundation;
B. Sub-structure--includes the cost of slab on grade and basement
walls;
C. Super structure--includes the cost of floor and roof construction,
interstitial and stair construction as well as structural steel;
D. Exterior structure--includes the cost of exterior walls, doors and
windows;
E. Roofing;
F. Interior construction--includes the cost of partitions and interior
finishes;
G. Conveying systems--includes the cost of elevators, moving stairs and
walkways, dumbwaiters, pneumatic tubing and conveying systems;
H. Mechanical systems--includes the cost of plumbing, HVAC, fire
protection, medical gas system, sewage treatment and solar mechanical
system;
I. Electrical systems--includes the cost of base materials, lighting,
electrical systems, communication systems as well as heating systems;
J. General conditions;
K. Equipment--includes the cost of equipment, special construction and
furnishings;
L. Sitework--includes the cost of site preparation, site improvements,
site utilities, and other specified off-site work.
Step Five: Identify Ground Rules and Assumptions:
Cost estimates are typically based on limited information and therefore
need to be bound by the constraints that make estimating possible.
These constraints usually take the form of assumptions that bind the
estimate's scope, establishing baseline conditions the estimate will be
built from. Ground rules represent a common set of agreed on estimating
standards that provide guidance and minimize conflicts in definitions.
Without firm ground rules, the analyst is responsible for making
assumptions that allow the estimate to proceed. Assumptions represent a
set of judgments about past, present, and future conditions postulated
as true in the absence of positive proof. The analyst must ensure that
assumptions are not arbitrary, that they are founded on expert
judgments rendered by experienced program and technical personnel. Many
assumptions profoundly influence cost; the subsequent rejection of even
a single assumption by management could invalidate many aspects of the
estimate. Therefore, it is imperative that cost estimators brief
management and document all assumptions well, so that management fully
understands the conditions the estimate was structured on. Failing to
do so can lead to overly optimistic assumptions that heavily influence
the overall cost estimate, to cost overruns, and to inaccurate
estimates and budgets.
5. Are there defined ground rules and assumptions that document the
rationale and any historical data to back up any claims?
Met; the estimator identified ground rules and assumptions as well its
escalation rates. In preparing the estimates Phoenix engineering made
some assumptions regarding costs. They supported their assumptions with
information from the designers and discussions with the VA. For
example, for this estimate they assumed that inflation would be tied to
the Boechk index. The escalation worksheets that the cost estimator
provided show what assumptions were made. Phoenix engineering also
reviewed the schedule assumptions for the cost estimate and had
concerns about whether the 1,000 day schedule was realistic. For
example, there will be three major phases to the project which the firm
believed would take about 4 years to complete: Phase 1 involves
removal/site prep/foundation/underground utilities (1 year due to life
safety & infection control); Phase 2 will be for construction (2 years
due to weather related issues); and Phase 3 will be for the renovation
of the 4th floor (1 year to complete). The cost estimate was prepared
using current prices as if bids were received on the date of the
estimate. The cost estimate was then escalated to the planned
construction contract award date using rates established by OMB. The
cost estimator demonstrated in their escalation paper the gap between
OMB mandated escalation rates and actual market conditions reflected by
the Boechk Index.
Step Six: Obtain the Data:
Data are the foundation of every cost estimate. How good the data are
affects the estimate's overall credibility. Depending on the data
quality, an estimate can range anywhere from a mere guess to a highly
defensible cost position. Credible cost estimates are rooted in
historical data. Rather than starting from scratch, estimators usually
develop estimates for new programs by relying on data from programs
that already exist and adjusting for any differences. Thus, collecting
valid and useful historical data is a key step in developing a sound
cost estimate. The challenge in doing this is obtaining the most
applicable historical data to ensure that the new estimate is as
accurate as possible. One way of ensuring that the data are applicable
is to perform checks of reasonableness to see if the results are
similar. Different data sets converging toward one value provides a
high degree of confidence in the data.
6. Was the data gathered from historical actual cost, schedule, and
program and technical sources?
Met; program office took well-documented steps to obtain data. For all
four iterations of the cost estimate, with the first being done in 2005
and the last in spring 2009, Phoenix engineering used the PROEST
construction software package. PROEST updates the software with the
updates from the RS Means which is an industry-standard estimating
database. They validate and supplement the RS Means data with internal
cost information that they have obtained from quotes from vendors on
other jobs that they have been involved in or data from research on
other projects. The only drawback to the software is that they had to
convert the cost estimate to Excel in order to manipulate it. Phoenix
engineering also updated PROEST data with changes in material costs.
PROEST data was also used for estimating recurring costs. For example,
data from PROEST was used to estimate the cost of slabs used for
elevator shafts by multiplying the costs by the number of floors. The
cost of fuel also had an impact on project costs so they added a
surcharge for fuel that would reflect the expected inflation rate for
fuel. The cost of fuel also affected the costs for concrete and steel.
Because quotes from vendors were not local, the data was normalized by
using a location factor that adjusted prices for Syracuse. Phoenix
engineering used several resources to look at square footage costs and
historical data that they have in-house for recent construction. They
also used construction industry data from sources such as RS Means or
Dodge Design Cost Data. Phoenix engineering worked with QPK to get
design specifications and clarifications when needed for the estimates.
QPK provided draft copies of each of the submissions to the VA 2 weeks
before the schematic design and design development milestones were
reached. Both the medical center staff and central office staff
commented on the design before it was released. For example, there were
questions about the specifications for the depth of the caissons in the
amount of concrete needed. The caissons are the underground supports
for the columns in the garage. They are large diameter holes in the
ground filled with concrete and rebar that in some cases go as deep as
30 feet.
Step Seven: Develop the Point Estimate and Compare It to an Independent
Cost Estimate:
Step 7 pulls all the information together to develop the point
estimate--the best guess at the cost estimate, given the underlying
data. High-quality cost estimates usually fall within a range of
possible costs, the point estimate being between the best and worst
case extremes. The cost estimator must perform several activities to
develop a point estimate: develop the cost model by estimating each WBS
element, using the best methodology, from the data collected; include
all estimating assumptions in the cost model; express costs in constant-
year dollars; time-phase the results by spreading costs in the years
they are expected to occur, based on the program schedule; and add the
WBS elements to develop the overall point estimate. Having developed
the overall point estimate, the cost estimator must then validate it by
thoroughly understanding and investigating how the cost model was
constructed. For example, all WBS cost estimates should be checked to
verify that calculations are accurate (no double counting) and account
for all costs, including indirect costs. Moreover, proper escalation
factors should be used to inflate costs so that they are expressed
consistently and accurately. Finally, the cost estimator should compare
the cost estimate against the independent cost estimate and examine
where and why there are differences; perform cross-checks on cost
drivers to see if results are similar; and update the model as more
data become available or as changes occur and compare the results
against previous estimates.
7. Did the cost estimator consider various cost estimating methods like
analogy, engineering build up, parametric, extrapolating from actual
costs, and expert opinion (if none of the other methods can be used)?
Met; the cost estimate is based on an engineering buildup. The initial
estimate for the SCI/D was $56 million. This initial estimate was a
gross level (rough order of magnitude) based on the aggregate square
footage and historical costs per square foot. This estimate was
developed in a rough manner using the square footage multiplied by the
per square foot cost of market-based construction. In addition, the
estimate included contingencies and overhead. Contingencies and
overhead came to about $2.1 million and the construction cost was
estimated at $53.9 million. The life cycle costs included in the OMB
300 were developed by medical center staff and were done in a hurry. In
2006 it became clear that the original estimate would be too little too
complete the spinal cord injury center. The Construction and Facilities
Management Office came up with the revised estimate. The new estimate
was for approximately $78 million or $24 million more than the original
estimate, and medical center staff were not informed of how this cost
estimate was developed. The medical center received $23.8 million in
2007 supplemental appropriations bill. In the summer of 2008, Phoenix
Engineering estimated the costs would be much higher because the cost
estimate was updated to reflect a more detailed design and reflected
delays in the project. As a result, when the project went out for bid,
the medical center found it would have to break the project up into
smaller increments because there was not enough budget to do it all at
once. These increments would be listed in the bid package as bid
alternates (i.e., additional projects) that potential contractors would
also provide bids for along with the base bid. In developing their
estimate, Phoenix Engineering used an engineering build up method based
on square feet times the labor rate plus analogous material costs.
Phoenix engineering used square feet data from QPK and converted it to
a cost by multiplying the square feet by the labor rate using data from
RS Means. They also checked the labor rates against union labor
agreements and the Davis Bacon Act, which mandates minimum labor rates
to check their estimate for consistency. The $77.7 million appropriated
for the construction includes the cost of the garage. The garage
contract is for almost $10.6 million, which leaves only $66.1 million
for the SCI/D. There were several internal reviews of the cost estimate
that were done to scrub the numbers. First, QPK reviewed the estimate,
then VA reviewed it, and finally Alpha performed a peer review of the
cost estimate. There was no independent cost estimate, but the Alpha
peer-review served that purpose. Phoenix made some changes that had a
major impact on the estimate. For example, Phoenix changed the estimate
to reflect a lower inflation rate and less design contingency. Phoenix
felt that the decrease was acceptable because the recession has caused
the cost of materials to decrease and contractors were lowering their
labor rates and fees in this competitive environment. Phoenix
engineering initially delivered a conservative estimate so there was
room to lower the cost estimate. In addition, there were no engineering
change orders because the VA would not accept any changes. Only one
state, New York, will budget a contingency for change orders for
construction. As a crosscheck, Phoenix engineering had someone gather
labor rates that were applicable to Syracuse. Their estimate was based
on a composite rate for the crew that would do the work.
Step Eight: Conduct a Sensitivity Analysis:
Sensitivity analysis should be included in all cost estimates because
it examines the effects of changing assumptions and ground rules. Since
uncertainty cannot be avoided, it is necessary to identify the cost
elements that represent the most risk and, if possible, cost estimators
should quantify the risk using both a sensitivity and uncertainty (see
step 9) analysis. In order for sensitivity analysis to reveal how the
cost estimate is affected by a change in a single assumption, the cost
estimator must examine the effect of changing one assumption or cost
driver at a time while holding all other variables constant. By doing
so, it is easier to understand which variable most affects the cost
estimate.
8. Did the cost estimate included a sensitivity analysis that
identified using a range of possible costs the effects of changing key
cost driver assumptions or factors?
Not met; while a sensitivity analysis was not conducted in their risk
analysis, VA Syracuse did identify major cost drivers. There was no
sensitivity analysis done. Square footage was dictated by VA. The only
changes were scope changes. Similarly, there was no sensitivity
analysis of the inflation index. The fuel surcharge that was included
in the estimate, which had been volatile, was taken out because fuel
prices had leveled off. They did not vary the composite labor rate.
While the VA Syracuse project followed most of the best practice steps,
they do not perform a full sensitivity analysis. However, the
estimating team did develop a risk analyses that identified major cost
drivers that could adversely affect the project.
Step Nine: Conduct a Risk and Uncertainty Analysis:
Because cost estimates predict future program costs, uncertainty is
always associated with them. Moreover, a cost estimate is usually
composed of many lower-level WBS elements, each of which comes with its
own source of error. Once these elements are added together, the
resulting cost estimate can contain a great deal of uncertainty. Risk
and uncertainty refer to the fact that because a cost estimate is a
forecast, there is always a chance that the actual cost will differ
from the estimate. A lack of knowledge about the future is only one
possible reason for the difference. Another equally important reason is
the error resulting from historical data inconsistencies, assumptions,
cost estimating equations, and factors typically used to develop an
estimate. In addition, biases are often found in estimating program
costs and developing program schedules. The biases may be cognitive--
often based on estimators' inexperience--or motivational, where
management intentionally reduces the estimate or shortens the schedule
to make the project look good to stakeholders. Recognizing the
potential for error and deciding how best to quantify it is the purpose
of risk and uncertainty analysis. Since cost estimates are uncertain,
making good predictions about how much funding a program needs to be
successful is difficult. In a program's early phases, knowledge about
how well technology will perform, whether the estimates are unbiased,
and how external events may affect the program is imperfect. For
management to make good decisions, the program estimate must reflect
the degree of uncertainty, so that a level of confidence can be given
about the estimate. Quantitative risk and uncertainty analysis provide
a way to assess the variability in the point estimate. Using this type
of analysis, a cost estimator can model such effects as schedules
slipping, missions changing, and proposed solutions not meeting user
needs, allowing for a known range of potential costs. Having a range of
costs around a point estimate is more useful to decision makers,
because it conveys the level of confidence in achieving the most likely
cost and also informs them on cost, schedule, and technical risks.
9. Was a risk and uncertainty analysis conducted that quantified the
imperfectly understood risks and identified the effects of changing key
cost driver assumptions and factors?
Partially met; while VA Syracuse followed most of the best practice
steps, they do not perform an uncertainty analysis: however, they did
perform risk analyses for the various alternatives. Phoenix Engineering
collected risks in excel spreadsheets and identified the likelihood of
the risks occurring as well as the impact of the risks. In addition,
the VA Cares Risk Analysis included a Risk Control Plan which
identified the risks, probability of occurring, and the internal
mitigation resources as well as the responsible parties. In the list of
risks identified by VA Cares Syracuse, the majority of the issues
focused on the construction of the new operating room (OR) while the
current ORs still need to be in operation while the project is under
construction. See list of risks identified below;
Risk Categories:
i. Schedule - Operating Room construction may fall behind schedule;
ii. Initial Costs - Costs associated with Operating Room construction
may be higher than expected;
iii. Life Cycle Costs - Expenses to run the Operating Room may be
greater than predicted;
iv. Technical Obsolescence - New Operating Room equipment could become
available soon after project completion;
v. Feasibility - Issues In operating room design, execution, or
functioning could result in financial feasibility problems;
vi. Reliability of Systems - Equipment fails to perform as designed;
vii. Dependencies & Interoperabilities - Facility is incapable of
supporting operating room equipment;
viii. Surety (Asset Protection) - Risk of vandalism (intentional
damage);
ix. Risk of Creating a Monopoly - Dependence on operating room vendors
for upgrades/repairs;
x. Capability of Agency to Manage the Project - Expertise unavailable
or has many competing projects in addition to operating room project.
Step Ten: Document the Estimate:
Documentation provides total recall of the estimate's detail so that it
can be replicated by someone other than those who prepared it. It also
serves as a reference to support future estimates. Documenting the cost
estimate makes available a written justification showing how it was
developed and aiding in updating it as key assumptions change and more
information becomes available. Estimates should be documented to show
all parameters, assumptions, descriptions, methods, and calculations
used to develop a cost estimate. A best practice is to use both a
narrative and cost tables to describe the basis for the estimate, with
a focus on the methods and calculations used to derive the estimate.
With this standard approach, the documentation provides a clear
understanding of how the cost estimate was constructed. Moreover, cost
estimate documentation should explain why particular methods and data
sets were chosen and why these choices are reasonable. It should also
reveal the pros and cons of each method selected. Finally, there should
be enough detail so that the documentation serves as an audit trail of
backup data, methods, and results, allowing for clear tracking of a
program's costs as it moves through its various life cycle phases.
10. Did the documentation describe the cost estimating process, data
sources, and methods step by step so that a cost analyst unfamiliar
with the program could understand what was done and replicate it?
Partially met; while the VA Syracuse office provided some
documentation, the cost estimator only provided the cost estimate and
without any supporting backup documentation. Design drawings were
logged in with the dates they were prepared. Phoenix Engineering also
discussed the format of the software that was used to document where
the labor rates came from and they provided 3 to 5 pages of explanation
along with the document log sheet. In the estimate documentation we
found that Phoenix Engineering compared the last estimate to the
current estimate and the reasons for any differences. Between design
and construction drawings, there were some changes in the way the
estimate was organized, but the cost difference was minimal. There
were, however, changes in the alternate designs. The VA Syracuse team
provided electronic files for the estimate summary and detailed
estimate, which included the WBS based on a required VA element
structure. In both the summary and detail estimate documentation, the
WBS broke down the construction costs into standardized systems such as
foundations, substructure, superstructure, exterior closure and roofing
as well as interior construction, conveying systems, mechanical &
electrical systems, equipment, and site work.
Step Eleven: Present Estimate to Management for Approval:
A cost estimate is not considered valid until management has approved
it. Since many cost estimates are developed to support a budget request
or make a decision between competing alternatives, it is vital that
management is briefed on how the estimate was developed, including
risks associated with the underlying data and methods. Therefore, the
cost estimator should prepare a briefing for management with enough
detail to easily defend the estimate by showing how it is accurate,
complete, and high in quality. The briefing should present the
documented life cycle cost estimate with an explanation of the
program's technical and program baseline;
11. Was there a briefing to management that included a clear
explanation of the cost estimate so as to convey its level of
competence?
Met; VA Syracuse reviewed the cost estimate many times before it became
final. The estimate was presented via written documentation and Excel
spreadsheet. Phoenix engineering made changes based on comments
received from QPK and VA (including comments from Alpha Engineering,
who conducted a peer review of the estimate). The reviewers provided
written comments to Phoenix via the 'doctor checks' system. This system
provided a centralized place to record comments. Comments were sent
electronically to Washington, D.C., so that others could review them.
Step Twelve: Update the Estimate to Reflect Actual Costs and Changes:
The cost estimate should be regularly updated to reflect all changes.
Not only is this a sound business practice; it is also a requirement
outlined in OMB's Capital Programming Guide. The purpose of updating
the cost estimate is to check its accuracy, defend the estimate over
time, shorten turnaround time, and archive cost and technical data for
use in future estimates. After the internal agency and congressional
budgets are prepared and submitted, it is imperative that cost
estimators continue to monitor the program to determine whether the
preliminary information and assumptions remain relevant and accurate.
Keeping the estimate fresh gives decision makers accurate information
for assessing alternative decisions. Cost estimates must also be
updated whenever requirements change, and the results should be
reconciled and recorded against the old estimate baseline. The
documented comparison between the current estimate (updated with actual
costs) and old estimate allows the cost estimator to determine the
level of variance between the two estimates. In other words, it allows
estimators to see how well they are estimating and how the program is
changing over time.
12. Is there a process for the estimating team to update the estimate
with actual costs as it becomes available?
Partially met; the program's cost estimate is not updated with actual
costs once the project is underway; however, that is not the
contractor's fault because once the contract is awarded the contractor
is no longer apart of the process. Thus far, the estimate has been a
living document. For example, the cost estimator has been working on
the document since 2005, making revisions and updates as warranted.
However, once the contract is awarded, Phoenix Engineering's services
will no longer be employed; therefore, they cannot update the estimate
with actual costs.
Source: GAO analysis of VA information.
[End of table]
Project Schedule:
VA initially estimated that the project would be completed by December
6, 2009. VA awarded the contract to construct the SCI/D center on
August 12, 2009, and estimates that the SCI/D center will be completed
in May 2012, or 29 months after the first estimated completion date.
The schedule delays and cost increases occurred before construction
began, and once construction commenced we found that the construction
schedule for this project generally followed best practices.
Specifically, the schedule met eight of nine scheduling best practices
but did not include a schedule risk analysis. The schedule did not
undergo a risk analysis to determine the major risks to the schedule
and the likelihood of the project being completed on time. Our analysis
of how the schedule met best practices is in table 7.
Table 7: Extent That Parking Garage Schedule Met Best Practices:
Best practice: Capturing activities;
Explanation: The schedule should reflect all activities as defined in
the project's work breakdown structure, which defines in detail the
work necessary to accomplish a project's objectives, including
activities to be performed by both the owner and contractors;
Met?: Met;
GAO analysis: The parking garage schedule follows a specific work
breakdown schedule that includes bonds, insurance, mobilization,
demolition, earthwork, sidewalks, chainlink fence, pavement markings,
concrete, cast-in-place, precast pre-stressed, architectural precast,
masonry, structural steel, waterproofing, roofing,
doors/frames/hardware, painting, interior signage, dry stand pipe,
elevators, plumbing/mechanical, and electrical. This level of breakdown
ensures that all of the work has been identified. Moreover, the
schedule is fully resource-loaded and all the resources in the schedule
add up to the cost estimate providing the scheduler with confidence
that it includes all of the work. In developing the schedule, the
scheduler used activity codes to develop the work and also used a
description of each trade, so that it is possible to sort on an
activity, by person responsible, and check the progress against what
the contractor has been paid. The activity codes confirm with account
numbers. In addition, the VA follows a disciplined process for
incorporating changes to ensure that new activities are described and
the cost for the change are identified so that changes can be
incorporated quickly into the schedule.
Best practice: Sequencing activities;
Explanation: The schedule should be planned so that critical project
dates can be met. To meet this objective, activities need to be
logically sequenced--that is, listed in the order in which they are to
be carried out. In particular, activities that must be completed before
other activities can begin (predecessor activities), as well as
activities that cannot begin until other activities are completed
(successor activities), should be identified. This helps ensure that
interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be established and used as a
basis for guiding work and measuring progress;
Met?: Met;
GAO analysis: The schedule includes the sequencing of activities. The
program's schedule includes both the predecessor and successor
activities to ensure that the interdependencies among activities are
used as a basis for guiding work and measuring progress. For example,
VA provided us with a detailed network diagram of the schedule that
depicts all of the work that the contractor needs to complete to finish
the garage. The driving logic for Syracuse is the weather which
dictates what work can start and when. In evaluating the schedule,
while we found a few activities that were missing logic links, the
scheduler was able to demonstrate why this was the case and the answers
were reasonable. All of the activities in the schedule have a finish to
start relationship that is a VA specification that is in line with best
practices.
Best practice: Assigning resources to activities;
Explanation: The schedule should reflect what resources (e.g., labor,
materials, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist;
Met?: Met;
GAO analysis: The schedule allocates resources, such as labor costs and
materials, to all activities. For example, the schedule includes
specific activity codes that correspond to a description of each trade
making it possible to sort by activity or person responsible. By having
this capability, the contractor can easily check progress and determine
if resources are in short supply or overallocated. Moreover, since the
schedule is fully resource loaded it can also be used to track costs.
Best practice: Establishing the duration of activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, historical data, and assumptions used for
cost estimating should be used. Durations should be as short as
possible and have specific start and end dates. The schedule should be
continually monitored to determine when forecasted completion dates
differ from planned dates; this information can be used to determine
whether schedule variances will affect subsequent work;
Met?: Met;
GAO analysis: The schedule establishes the durations of activities
based on contractor expert opinion and historical data. The schedule
activity durations were determined by identifying the various crews and
methods necessary for getting specific work done. Using this
information, the scheduler relied on historical data from RS Means to
identify the number of days necessary for each task. Developing the
duration estimates was a complex effort for the garage because weather
was always an issue---the garage is open to the elements and limited
crew space creates inefficiencies that needed to be considered.
Finally, most of the activities were of short duration (i.e., less than
21 days) in keeping with best practices. There was one long duration
activity (i.e., 125 days), but it was not driving the critical path.
This activity was in the schedule to remind the contractor to turn over
the drawings to the VA when the garage was complete.
Best practice: Integrating activities horizontally and vertically;
Explanation: The schedule should be horizontally integrated, meaning
that it should link products and outcomes associated with other
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that the dates for
starting and completing activities in the integrated master schedule
should be aligned with the dates for supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to work
to the same master schedule;
Met?: Met;
GAO analysis: The schedule is horizontally integrated, meaning that the
activities across the multiple teams are arranged in the right order to
achieve aggregated products or outcomes. We tested horizontal
integration by extending a non-critical activity 200 additional days to
see if it showed up on the project's critical path. The activity became
critical and as a result the completion date for the project was pushed
out to July 2010 providing us with confidence that the schedule was
horizontally dynamic. The schedule was also vertically integrated in
that traceability existed among varying levels of the activities
allowing multiple trades to work to the same master schedule.
Best practice: Establishing the critical path for activities;
Explanation: Scheduling software should be used to identify the
critical path, which represents the chain of dependent activities with
the longest total duration. Establishing a project's critical path is
necessary to examine the effects of any activity slipping along this
path. Potential problems along or near the critical path should also be
identified and reflected in scheduling the duration of high-risk
activities;
Met?: Met;
GAO analysis: The project's critical path has been defined using
scheduling software and includes, among other things, installing steel
and pre-cast concrete planks, pouring cement, and installing a snow
melt system. Program officials demonstrated the critical path using
their scheduling tool. Specifically, they showed us which remaining
activities fell on the critical path. For example, elevator extension
work, refurbishment of cab controls, testing elevators, form tower
columns, reinforcing steel columns, rebar and place concrete columns to
roof, and install masonry shell and enclosure were all critical path
activities.
Best practice: Identifying the float between activities;
Explanation: The schedule should identify the float--the amount of time
by which a predecessor activity can slip before the delay affects
successor activities--so that a schedule's flexibility can be
determined. As a general rule, activities along the critical path have
the least float. Total float is the total amount of time by which an
activity can be delayed without delaying the project's completion (if
everything else goes according to plan);
Met?: Met;
GAO analysis: The contractor's overall schedule process enabled good
visibility into the float between activities and demonstrated that
float is actively managed. While we found some instances of high float,
the contractor had valid reasons for it. For example, we found one
activity that had 154 days of float but the contractor explained to us
that this activity was for a change order that was overcome by a
supplemental agreement. According to the contractor, supplemental
agreements cancel out change orders; however, they like to keep change
orders in the schedule for tracking purposes. Similarly, there was also
high float for activities associated with the guardrails and sidewalks,
but these activities do not drive any critical work and just need to be
done by the end of the project so the high float was justified. For the
remaining tasks, we found that most had less than 40 days float and we
found no instances of negative float.
Best practice: Conducting a schedule risk analysis;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
project's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can affect the project's status;
Met?: Not met;
GAO analysis: The project did not perform a schedule risk analysis that
would determine the level of confidence in meeting the program's
completion date, even though the project's Exhibit 300 identified the
probability of falling behind schedule as a medium risk. As a result,
the project did not identify any schedule reserve which should be
calculated by performing a schedule risk analysis, and set aside for
those activities identified as high-risk. Without this reserve, if any
delays were to occur on any activities on the critical path, the
program faces the risk of further delays to the scheduled completion
date. There have been schedule slips in the parking garage construction
project. Most notably, the Request for Proposal assumed that the
contract would be awarded on May 23, 2008, with a Notice to Proceed
occurring about 2 weeks later. Relying on these assumptions, the
contractor estimated that construction would be complete by July 19,
2009. However, these assumptions did not hold true. For example, while
the contract was awarded in the middle of June 2008, the Notice to
Proceed did not occur until the end of July 2008--a delay of almost 46
days. This slip had major ramifications on the schedule as the
contractor had planned to pour concrete in the fall so that structural
steel could be laid by the time winter set in. Instead, the contractor
had to pour concrete during the winter, which was problematic as the
extreme cold not only affected worker productivity but also the time it
took for the concrete to cure. Thus, the almost 2-month slip caused an
approximate 4-month delay due to productivity being hampered by the
cold weather. Due to these problems and other change orders, the
completion date has been extended to November 2009. The delay in
completing the project also has been increasing costs to the medical
center because they have to pay for remote parking and a shuttle bus
for employees who are unable to park in the parking garage.
Best practice: Updating the schedule using logic and durations to
determine dates;
Explanation: [Empty];
Met?: Met;
GAO analysis: The schedule is updated on a monthly basis using logic
and durations to determine the dates. As a result, the schedule is a
good tool to identify the critical path so that VA can use it for
making management decisions. VA is briefed monthly on the schedule via
contractor progress reports that discuss major variances, percent
complete, duration changes, change orders and how they affect logic and
durations, supplemental agreements, and time extensions that must be
incorporated into the schedule network. The monthly report also
identifies what activities are driving the critical path so that
management can focus its attention on them. Using this approach, the VA
ensures that the schedule is accurate and current to the contractor's
plan of construction. There are two construction reasons for the
extended contract completion delay to November 2009. First, during
construction the contractor found a buried cable conduit and it took
some time to figure out what it was, who it belonged to, and where to
move it. This effort took about 42 days to complete. Second, while the
construction crew was renovating the walls around the elevators on all
floors of the garage, they found blue flexible pipe that was once used
in construction but is no longer up to code. As a result, they had to
replace and reroute the tubing in the garage, causing a slip of 17
days.
Source: GAO analysis of VA information.
[End of table]
[End of section]
Appendix V: Construction of New Medical Center Complex in Las Vegas,
Nevada:
Project Overview:
This project involves construction of a comprehensive Medical Center
Complex in Las Vegas, Nevada. The complex will consist of up to 90
inpatient beds, a 120-bed Nursing Home Care Unit, Ambulatory Care
Center, primary and specialty care, surgery, mental health,
rehabilitation, geriatrics and extended care, as well as administrative
and support functions. VA also plans to include Veterans Benefits
Administration offices attached to the medical center. The project is
divided into four phases. Phase I includes the construction of a new
utility building and related infrastructure such as streets, sewers,
and connections to electric and water utilities that are miles away
from the construction site. Phase II includes the construction of the
foundation of the new medical center. Phase III includes the
construction of the Nursing Home Care Unit and Phase IV includes the
construction of the medical center and the Veterans Benefits Office.
Reasons for the Project:
VA initiated the medical center project under the Capital Asset
Realignment for Enhanced Services (CARES) process between 2003 and 2004
because, according to VA officials, the increase in the number of Iraq
war veterans needing medical care combined with the growth in the Las
Vegas area supported building a large medical center.[Footnote 28] Out-
patient medical care for veterans in the area was provided at 15 leased
primary care clinics located throughout the Las Vegas area. In-patient
services were provided under a joint venture with the Air Force's Mike
O'Callaghan Federal Hospital located at Nellis Air Force Base. However,
some VA patients had to be sent to other VA hospitals for care that
could not be provided at the Mike O'Callaghan hospital such as spinal
cord injuries. VA officials said they initially sought to expand its
medical services and construct a nursing home at Nellis Air Force Base
in 2004, but the Air Force would not agree to such an expansion and
advised VA that the number of veterans' in-patient beds would likely
have to be reduced in the future. As a result, VA decided to construct
a new comprehensive medical complex, including a nursing home care
unit.
Project Cost:
The cost of the medical center has increased from an initial estimate
of $286 million in 2004 to a current estimate of $600.4 million (an
increase of 110 percent). In accordance with these increased cost
estimates, Congress has appropriated $600.4 million for the medical
center, providing $60 million for fiscal year 2004,[Footnote 29] an
additional $199 million for fiscal year 2006,[Footnote 30] and $341.4
million for fiscal year 2008.[Footnote 31] The original estimate to
Congress was based on plans for a large VA clinic. However, VA later
determined that a much larger medical center was needed in Las Vegas
after it became clear that an inpatient medical facility it shares with
DOD would not be adequate to serve the medical needs of local veterans.
Since the estimate for the Las Vegas medical center was based on a
preliminary design for an expanded clinic, additional functions had to
be added to the clinic design to provide the services necessary for the
medical center. This expansion of the scope of the project resulted in
both a cost increase and schedule delay for the project. According to
VA officials, a lack of planning and the omission of key facilities
contributed to the cost increases. Specifically, VA officials stated
that the original cost estimate did not correctly anticipate the amount
of preparation that the site needed. For example, the original estimate
did not include funding for the roads and street lights required for
the facility. In addition, the medical center could not anticipate that
the Department of Homeland Security would institute new requirements
for federal facilities as part of its continuing response to the events
of September 11, 2001, which resulted in further cost increases. VA
officials also explained that the nationwide increase in construction,
the rebuilding in the New Orleans area since hurricane Katrina, and the
local building boom in Las Vegas have driven up the cost of material
and labor. The Las Vegas area had several multi-billion dollar projects
underway. Locally, construction costs increased over 20 percent in 2006
and 2007 while the standard that VA uses for contingencies is 5
percent. To illustrate, VA staff told us that Las Vegas builders had
tied up almost 80 percent of the nation's large cranes used to build
tall buildings.
According to VA officials, in response to the increasing costs, the VA
and the architectural/engineering firm preparing the medical center
design reduced the scope of work for the final phase of the project.
Gross square footage was reduced from about 900,000 square feet to
785,000 square feet and they eliminated extra space between floors for
mechanical and electrical cables that would have made maintenance
easier. They also reduced warehouse space and space for administrative
offices because estimators were concerned that the project could not be
completed with the funds available. The medical center warehouse, which
is used to store maintenance and medical supplies, was reduced to one-
third of its originally proposed size. As, a result, the hospital will
need to acquire warehouse storage and procure warehouse management
services from contractors outside of the VA facility.
The economic recession that began in 2008 led several companies to
suspend their construction projects in Las Vegas, and there was greater
competition among construction firms to construct the hospital. This
change in the construction market led to a significantly lower cost of
construction than VA staff had anticipated, and VA now estimates that
the total project will cost about $100 million less than estimated. As
a result, VA officials explained they are taking steps to add these
features back into the medical center prior to completion. For example,
a utility tunnel running from the utility building to the medical
center was added back to the project once the construction contract was
awarded and VA saw they had funds available. Adding this tunnel will
reduce operating and maintenance costs for the medical center. VA
officials are also reviewing their options for adding back features
that had been eliminated such as administrative offices. This would
save operating costs by eliminating the need to lease office space. Our
analysis of VA's current cost estimate for the construction of the
medical center is in table 8.
Table 8: Extent That Cost Estimate for Las Vegas Medical Center Met
Best Practices:
Step One: Define the Estimate's Purpose:
The purpose of a cost estimate is determined by its intended use, and
its intended use determines its scope and detail. Cost estimates have
two general purposes: (1) to help managers evaluate affordability and
performance against plans, as well as the selection of alternative
systems and solutions, and (2) to support the budget process by
providing estimates of the funding required to efficiently execute a
program. The scope of the cost estimate will be determined by such
issues as the time involved, what elements of work need to be
estimated, who will develop the cost estimates, and how much cost
estimating detail will be included. A life-cycle cost estimate provides
an exhaustive and structured accounting of all resources and associated
cost elements required to develop, produce, deploy, and sustain a
particular program. As such a life-cycle cost estimate encompasses all
past (or sunk), present, and future costs for every aspect of a
program, regardless of funding source. Life-cycle costing enhances
decision making, especially in early planning and concept formulation
of acquisition. Design trade-off studies conducted in this period can
be evaluated on a total cost basis as well as on a performance and
technical basis. A life-cycle cost estimate can support budgetary
decisions, key decision points, milestone reviews, and investment
decisions. Because they encompass all possible costs, life-cycle cost
estimates provide a wealth of information about how much programs are
expected to cost over time. Thus, having full life-cycle costs is
important for successfully planning program resources and making wise
decisions.
1. Is the purpose and scope of the cost estimate defined and
documented? Have all costs been estimated, including life-cycle costs?
Met; the purpose of the cost estimate is documented and clearly defined
at a level that would enable VA Las Vegas to submit a quality cost
estimate. The cost estimate is an estimate of probable cost, developed
by an independent consultant to the architect for the purposes of
comparing incoming bid proposals. The estimate covers the construction
of Phase IV at an estimated construction cost of $365 million. The
estimate was not required to include complete lifecycle costs estimate;
it includes the construction of Phase IV up to the hand-over of the
keys to the building. Life cycle costs are included in the OMB Exhibit
300. The scope of the estimate is defined by VA policy. The Manual for
Preparation of Cost Estimates for VA Facilities states that:
1.1.1 - A project estimate shall show the current cost of construction
on the date of the estimate. The estimate should reflect current costs
on the date the estimate is received and anticipated local escalation
to the midpoint of construction, i.e., date of estimate plus half of
construction duration;
1.1.2 - The level of detail for this estimate shall be consistent with
the degree of completeness of the drawings being submitted. Simply
stated, this means that if a construction element is shown, it must be
priced; if it is shown in detail, it must be priced in detail. For
detailed elements, "lump sum" or "allowance" figures will not be
acceptable. Project estimates will include all elements within the
contractor's bid such as insurance, bonds, hazardous abatement, and any
other such items. Submission requirements are indicated in VA Cost
Estimating Guide.
Step Two: Develop the Estimating Plan:
An analytic approach to cost estimates typically entails a written
study plan detailing a master schedule of specific tasks, responsible
parties, and due dates. Enough time should be scheduled to collect
data, including visits to contractor sites to further understand the
strengths and limitations of the data that have been collected. If
there is not enough time, then the schedule constraint should be
clearly identified in the ground rules and assumptions, so that
management understands the effect on the estimate's quality and
confidence.
2. Did the team develop a written study plan?
Met; the estimating team is from a centralized cost estimating firm
that specializes in construction and the estimate follows cost estimate
preparation guidance published by the VA. Officials stated that the
estimate includes the full cost of construction regardless of funding
source. The VA Office of Construction and Facilities Management (CFM)
publishes guidance on preparing cost estimates that details how
construction cost estimates should be created, structured, and
presented. The manual also explains roles and responsibilities, units
of measure, and guidance on master specifications. The cost estimate
documentation includes introductory notes that explain the overall cost
estimating approach. Officials stated that the process for developing
estimates begins with the contract documents, then the pricing of
materials and overhead, collection of historical data, gathering of
current market pricing, and conducting market studies. As consultants
to the architect, the independent cost estimating firm has a staff of
25 cost estimators dedicated to developing construction cost estimates.
The 25 estimators have extensive backgrounds ranging from new hires to
individuals with 40+ years of experience in the cost estimating
discipline. The basic approach of the estimate is defined by VA policy.
The Manual for Preparation of Cost Estimates for VA Facilities states
that:
1.1.1 - A project estimate shall show the current cost of construction
on the date of the estimate. The estimate should reflect current costs
on the date the estimate is received and anticipated local escalation
to the midpoint of construction, i.e., date of estimate plus ½ of
construction duration;
1.1.2 - The level of detail for this estimate shall be consistent with
the degree of completeness of the drawings being submitted. Simply
stated, this means that if a construction element is shown, it must be
priced; if it is shown in detail, it must be priced in detail. For
detailed elements, "lump sum" or "allowance" figures will not be
acceptable. Project estimates will include all elements within the
contractor's bid such as insurance, bonds, hazardous abatement and any
other such items. Submission requirements are indicated in VA Cost
Estimating Guide.
Step Three: Define the Program Characteristics:
Key to developing a credible estimate is having an adequate
understanding of the acquisition program--the acquisition strategy,
technical definition, characteristics, system design features, and
technologies to be included in its design. The cost estimator can use
this information to identify the technical and program parameters that
will bind the cost estimate. The amount of information gathered
directly affects the overall quality and flexibility of the estimate.
Less information means more assumptions must be made, increasing the
risk associated with the estimate. Therefore, the importance of this
step must be emphasized, because the final accuracy of the cost
estimate depends on how well the program is defined.
3. Is there a documented technical baseline description?
Met; a technical baseline has been documented that includes
requirements, purpose, and system design features. The Technical
Baseline is based on the construction drawings and specifications used
by the architect to design the hospital. These construction documents
were approved by the architect, and officials stated that the VA
provided written approval to the architect regarding the technical
baseline. Other technical baseline documents to be referenced in the
development of a VA cost estimate are defined by VA policy. These
documents, listed and defined in The Manual for Preparation of Cost
Estimates for VA Facilities, include Practice Design Manuals, Master
Specifications, Architect/Engineer Checklists, Design and Quality
Alerts, Design Guides, Design and Construction Procedures, Physical
Security Design Manuals, and Technical Summaries. The Cost Estimate
Manual also includes the cost breakdown categories to be used in the
estimate.
Step Four: Determine the Estimating Structure:
A work breakdown structure (WBS) is the cornerstone of every program
because it defines in detail the work necessary to accomplish a
program's objectives. A WBS is a valuable communication tool between
systems engineering, program management, and other functional
organizations because it provides a clear picture of what needs to be
accomplished and how the work will be done. Accordingly, it is an
essential element for identifying activities in a program's integrated
master schedule and it provides a consistent framework for planning and
assigning responsibility for the work. Initially set up when the
program is established, the WBS becomes successively detailed over time
as more information because known about the program. A WBS deconstructs
a program's end product into successive levels with smaller specific
elements until the work is subdivided to a level suitable for
management control. By breaking the work down into smaller elements,
management can more easily plan and schedule the program's activities
and assign responsibility for the work. It also facilitates
establishing a schedule, cost, and earned value management (EVM)
baseline. Establishing a product-oriented WBS is a best practice
because it allows a program to track cost and schedule by defined
deliverables, such as a hardware or software component. This allows a
program manager to more precisely identify which components are causing
cost or schedule overruns and to more effectively mitigate the root
cause of the overruns.
4. Is there a defined WBS and/or cost element structure?
Met; the estimate clearly describes how the various sub-elements are
summed to produce the amounts for each cost category, thereby ensuring
that all pertinent costs are included and no costs are double counted.
The cost estimate categorizes construction costs into a required VA
element structure that breaks the building down into systems and
subsystems. The WBS is based on the standardized WBS on VA form HO-
18B/C. The WBS breaks the construction costs into standardized systems
such as foundation, substructure, superstructure, and roofing, as well
as subsystems such as slab on grade, stair construction, and elevators.
These system descriptions are also used in the schedule. The HO-18 WBS
elements are defined in the Manual for Preparation of Cost Estimates
for VA Facilities.
Step Five: Identify Ground Rules and Assumptions:
Cost estimates are typically based on limited information and therefore
need to be bound by the constraints that make estimating possible.
These constraints usually take the form of assumptions that bind the
estimate's scope, establishing baseline conditions the estimate will be
built from. Ground rules represent a common set of agreed on estimating
standards that provide guidance and minimize conflicts in definitions.
Without firm ground rules, the analyst is responsible for making
assumptions that allow the estimate to proceed. Assumptions represent a
set of judgments about past, present, and future conditions postulated
as true in the absence of positive proof. The analyst must ensure that
assumptions are not arbitrary, that they are founded on expert
judgments rendered by experienced program and technical personnel. Many
assumptions profoundly influence cost; the subsequent rejection of even
a single assumption by management could invalidate many aspects of the
estimate. Therefore, it is imperative that cost estimators brief
management and document all assumptions well, so that management fully
understands the conditions the estimate was structured on. Failing to
do so can lead to overly optimistic assumptions that heavily influence
the overall cost estimate, to cost overruns, and to inaccurate
estimates and budgets.
5. Are there defined ground rules and assumptions that document the
rationale and any historical data to back up any claims?
Met; cost-influencing ground rules and assumptions, such as the
programs schedule, labor rates, and inflation rates are documented. The
cost estimate documentation provides an overview of basic assumptions
underlying the estimate. For example, the documentation notes which
construction drawings were the basis of the estimate as well as the
number of assumed bids. The cost estimate also notes the areas in which
contingency was removed and the reasons for its removal. Officials
stated that the cost estimate is based on prevailing labor wage rates
as well as local pricing of material. In addition, the cost estimate
outlines the specific items that are not included in the estimate. The
Manual for Preparation of Cost Estimates for VA Facilities specifically
notes what costs should be included for each system and subsystem
category.
Step Six: Obtain the Data:
Data are the foundation of every cost estimate. How good the data are
affects the estimate's overall credibility. Depending on the data
quality, an estimate can range anywhere from a mere guess to a highly
defensible cost position. Credible cost estimates are rooted in
historical data. Rather than starting from scratch, estimators usually
develop estimates for new programs by relying on data from programs
that already exist and adjusting for any differences. Thus, collecting
valid and useful historical data is a key step in developing a sound
cost estimate. The challenge in doing this is obtaining the most
applicable historical data to ensure that the new estimate is as
accurate as possible. One way of ensuring that the data are applicable
is to perform checks of reasonableness to see if the results are
similar. Different data sets converging toward one value provides a
high degree of confidence in the data.
6. Was the data gathered from historical actual cost, schedule, and
program and technical sources?
Met; cost estimators used local pricing of labor and material as well
as local escalation rates. Historical data were used when applicable,
but the Las Vegas area has not had major hospital construction in
recent years. Officials stated the cost estimate is based on local
pricing of labor and material. They stated that although the cost
estimating firm has data collected over 20 years, the hospital estimate
is not entirely based on historical data. Officials told us this is
because there have been no large-scale hospitals built in Las Vegas in
recent years; and the last major VA hospital was constructed over 20
years ago. Due to project uniqueness, pricing was based primarily on
quotes and estimates. The estimate used local escalation rates instead
of OMB rates because of the high real estate costs in the Las Vegas
area at the time of the estimate construction. Officials stated that
the estimates include pricing for labor and detailed materials, such as
linear feet of wire, light fixtures, cubit years of concrete, and
pounds of steel and rebar. In addition, officials stated that material
estimates went through peer reviews.
Step Seven: Develop the Point Estimate and Compare it to an independent
cost estimate:
Step 7 pulls all the information together to develop the point
estimate--the best guess at the cost estimate, given the underlying
data. High-quality cost estimates usually fall within a range of
possible costs, the point estimate being between the best and worst
case extremes. The cost estimator must perform several activities to
develop a point estimate: develop the cost model by estimating each WBS
element, using the best methodology, from the data collected; include
all estimating assumptions in the cost model; express costs in constant-
year dollars; time-phase the results by spreading costs in the years
they are expected to occur, based on the program schedule; and add the
WBS elements to develop the overall point estimate. Having developed
the overall point estimate, the cost estimator must then validate it by
thoroughly understanding and investigating how the cost model was
constructed. For example, all WBS cost estimates should be checked to
verify that calculations are accurate (no double counting) and account
for all costs, including indirect costs. Moreover, proper escalation
factors should be used to inflate costs so that they are expressed
consistently and accurately. Finally, the cost estimator should compare
the cost estimate against the independent cost estimate and examine
where and why there are differences; perform cross-checks on cost
drivers to see if results are similar; and update the model as more
data become available or as changes occur and compare the results
against previous estimates.
7. Did the cost estimator consider various cost estimating methods like
analogy, engineering build up, parametric, extrapolating from actual
costs, and expert opinion (if none of the other methods can be used)?
Met; the cost estimate is based on a detailed engineering buildup
methodology using estimated labor and material prices, and cross
checked against independent cost assessments. The estimate was vetted
through experts to ensure costs were appropriately captured. The cost
estimate is based on a detailed engineering buildup methodology using
estimated labor and material prices. Officials stated that parametric
methodologies were used to conduct crosschecks during early design,
when details were not that well defined. At the request of the VA, two
additional cost estimates were performed and compared against the
original estimate. These independent third-party estimates were
performed using price databases and parametric techniques. The
crosscheck estimates were provided to the VA in February and April of
2008, several weeks before the final detailed estimate was delivered in
May 2008. The detailed estimate is vetted through layers of experts,
including the architect, outside peer reviews by third-party
consultants, and VA resident engineers. In addition, officials stated
that the required breakout by the VA ensures transparency and documents
that all costs are properly captured.
Step Eight: Conduct a Sensitivity Analysis:
Sensitivity analysis should be included in all cost estimates because
it examines the effects of changing assumptions and ground rules. Since
uncertainty cannot be avoided, it is necessary to identify the cost
elements that represent the most risk and, if possible, cost estimators
should quantify the risk using both a sensitivity and uncertainty (see
step 9) analysis. In order for sensitivity analysis to reveal how the
cost estimate is affected by a change in a single assumption, the cost
estimator must examine the effect of changing one assumption or cost
driver at a time while holding all other variables constant. By doing
so, it is easier to understand which variable most affects the cost
estimate.
8. Did the cost estimate included a sensitivity analysis that
identified using a range of possible costs the effects of changing key
cost driver assumptions or factors?
Not met; a sensitivity analysis was not performed. While officials
noted that market surveys were conducted for the cost estimate and
contingency was included, a formal sensitivity analysis was not
performed because it was not requested by the VA. The cost estimating
firm performs market surveys at each stage of design, evaluating local
capital and the availability of trade skills in the local market.
However, officials stated that the final product to VA is a point
estimate because they are not they are not afforded the luxury of
providing ranges of costs.
Step Nine: Conduct a Risk and Uncertainty Analysis:
Because cost estimates predict future program costs, uncertainty is
always associated with them. Moreover, a cost estimate is usually
composed of many lower-level WBS elements, each of which comes with its
own source of error. Once these elements are added together, the
resulting cost estimate can contain a great deal of uncertainty. Risk
and uncertainty refer to the fact that because a cost estimate is a
forecast, there is always a chance that the actual cost will differ
from the estimate. A lack of knowledge about the future is only one
possible reason for the difference. Another equally important reason is
the error resulting from historical data inconsistencies, assumptions,
cost estimating equations, and factors typically used to develop an
estimate. In addition, biases are often found in estimating program
costs and developing program schedules. The biases may be cognitive--
often based on estimators' inexperience--or motivational, where
management intentionally reduces the estimate or shortens the schedule
to make the project look good to stakeholders. Recognizing the
potential for error and deciding how best to quantify it is the purpose
of risk and uncertainty analysis. Since cost estimates are uncertain,
making good predictions about how much funding a program needs to be
successful is difficult. In a program's early phases, knowledge about
how well technology will perform, whether the estimates are unbiased,
and how external events may affect the program is imperfect. For
management to make good decisions, the program estimate must reflect
the degree of uncertainty, so that a level of confidence can be given
about the estimate. Quantitative risk and uncertainty analysis provide
a way to assess the variability in the point estimate. Using this type
of analysis, a cost estimator can model such effects as schedules
slipping, missions changing, and proposed solutions not meeting user
needs, allowing for a known range of potential costs. Having a range of
costs around a point estimate is more useful to decision makers,
because it conveys the level of confidence in achieving the most likely
cost and also informs them on cost, schedule, and technical risks.
9. Was a risk and uncertainty analysis conducted that quantified the
imperfectly understood risks and identified the effects of changing key
cost driver assumptions and factors?
Partially met; while cost estimators did not perform a formal
uncertainty analysis, risk assessments were developed on the
availability of trades. Officials stated that while VA does not require
a formal uncertainty analysis, cost estimators did perform an internal
risk analysis evaluating at-risk trades. From that risk analysis,
estimators stated they had a low level of confidence in the
availability of mechanical, plumbing and electrical trades. Officials
stated that part of the risk was based on how competitive the market
was in Las Vegas at the time they prepared the estimate. Officials told
us that the VA does not require an uncertainty analysis and the
analysis is, generally speaking, not a construction industry best
practice.
Step Ten: Document the Estimate:
Documentation provides total recall of the estimate's detail so that it
can be replicated by someone other than those who prepared it. It also
serves as a reference to support future estimates. Documenting the cost
estimate makes available a written justification showing how it was
developed and aiding in updating it as key assumptions change and more
information becomes available. Estimates should be documented to show
all parameters, assumptions, descriptions, methods, and calculations
used to develop a cost estimate. A best practice is to use both a
narrative and cost tables to describe the basis for the estimate, with
a focus on the methods and calculations used to derive the estimate.
With this standard approach, the documentation provides a clear
understanding of how the cost estimate was constructed. Moreover, cost
estimate documentation should explain why particular methods and data
sets were chosen and why these choices are reasonable. It should also
reveal the pros and cons of each method selected. Finally, there should
be enough detail so that the documentation serves as an audit trail of
backup data, methods, and results, allowing for clear tracking of a
program's costs as it moves through its various life-cycle phases.
10. Did the documentation describe the cost estimating process, data
sources, and methods step by step so that a cost analyst unfamiliar
with the program could understand what was done and replicate it?
Partially met; while the documentation for the most part provided
detailed material and labor build up, we were not able to trace the
data back based on the documentation alone. While officials stated that
the estimate was in part based off data from previous estimates and
market surveys, the cost estimate documentation delivered to VA does
not trace estimated values to raw or normalized data. For instance, the
delivered cost estimate documentation does not provide a basis or
supporting data for included labor dollars or general conditions markup
that would allow an analyst unfamiliar with the project to recreate
them.
Step Eleven: Present Estimate to Management for Approval:
A cost estimate is not considered valid until management has approved
it. Since many cost estimates are developed to support a budget request
or make a decision between competing alternatives, it is vital that
management is briefed on how the estimate was developed, including
risks associated with the underlying data and methods. Therefore, the
cost estimator should prepare a briefing for management with enough
detail to easily defend the estimate by showing how it is accurate,
complete, and high in quality. The briefing should present the
documented life cycle cost estimate with an explanation of the
program's technical and program baseline.
11. Was there a briefing to management that included a clear
explanation of the cost estimate so as to convey its level of
competence?
Met; the estimate is vetted through layers of experts, including the
architect, outside peer reviews by third-party consultants, and VA
resident engineers. Officials stated that the cost estimate was first
reviewed by the architect responsible for the detailed design of the
hospital. After this initial review, the estimate is then presented to
the VA. The cost estimate is part of the milestone submittal outlined
in the contractual requirements between the architectural firm and the
VA. Officials stated that the cost estimate review is a month-long
process. The detailed estimate is vetted through layers of experts,
including the architect, outside peer reviews by third-party
consultants, and VA resident engineers. These reviews helped refine the
estimate and its underlying assumptions. For example, officials stated
that one third-party reviewer took issue with the assumed price of
steel in the estimate. VA officials stated that they use information
from the peer reviews prior to giving the estimate their approval. At
the request of the VA, two additional cost estimates were performed and
compared against the original estimate. These independent third-party
estimates were performed at a unit-level using pricing databases. The
crosscheck estimates were provided to the VA in February and April of
2008, several weeks before the final detailed estimate was delivered in
May 2008.
Step Twelve: Update the Estimate to Reflect Actual Costs and Changes:
The cost estimate should be regularly updated to reflect all changes.
Not only is this a sound business practice, it is also a requirement
outlined in OMB's Capital Programming Guide. The purpose of updating
the cost estimate is to check its accuracy, defend the estimate over
time, shorten turnaround time, and archive cost and technical data for
use in future estimates. After the internal agency and congressional
budgets are prepared and submitted, it is imperative that cost
estimators continue to monitor the program to determine whether the
preliminary information and assumptions remain relevant and accurate.
Keeping the estimate fresh gives decision makers accurate information
for assessing alternative decisions. Cost estimates must also be
updated whenever requirements change, and the results should be
reconciled and recorded against the old estimate baseline. The
documented comparison between the current estimate (updated with actual
costs) and old estimate allows the cost estimator to determine the
level of variance between the two estimates. In other words, it allows
estimators to see how well they are estimating and how the program is
changing over time.
12. Is there a process for the estimating team to update the estimate
with actual costs as it becomes available?
Not met; the VA does not require the cost estimating firm to update the
construction cost estimate with actual costs once the project is
underway. The estimate is not updated once construction begins.
Officials from the cost estimating firm stated that they attempt to
collect past bid results when possible; however, the tracking and
reporting of actual costs by the estimator is not part of the
contractual requirements between the VA and the A/E firm. VA officials
stated that because this is a fixed price contract, the contractor is
responsible for managing the costs. However, regardless of what type of
contract or what organization is managing costs, the purpose of
updating the cost estimate is to check its accuracy, defend the
estimate over time, shorten turnaround time of future estimates, and
archive cost and technical data for use in future estimates.
Source: GAO analysis of VA information.
[End of table]
Project Schedule:
The first two phases of the project have been completed and, according
to VA officials, Phase III will be completed in February 2010. However,
the nursing home completed in Phase III of the project will not be open
for patient care until the medical center becomes operational in 2012,
as the nursing home relies upon the hospital for patient medical care
and food service. Since the nursing home will be vacant for about 2
years before the medical center opens, VA may use part of the nursing
home for administrative offices.
The final phase of the project, the construction of the new medical
center, is underway with completion scheduled for August 2011.
According to VA officials, the medical center is scheduled to become
operational in the spring of 2012, depending upon how quickly the
equipment for the hospital can be purchased and the additional
personnel can be hired. Our analysis of the construction schedule of
the medical center is in table 9.
Table 9: Extent That Construction Schedule for Las Vegas Hospital Met
Best Practices:
Best practice: Capturing activities;
Explanation: The schedule should reflect all activities as defined in
the project's work breakdown structure, which defines in detail the
work necessary to accomplish a project's objectives, including
activities to be performed by both the owner and contractors;
Met?: Substantially met;
GAO analysis: The schedule is required by contract to include
approximately 2,500 to 3,000 activities in order to sufficiently detail
the level of work required (the actual schedule has 6,089 activities,
approximately 16 detail activities per milestone). Each activity is
mapped to an activity ID number, building area, and work trade, which
allows the scheduler to quickly filter the schedule by type of work or
subcontractor. The schedule is reviewed by the VA CFM for completeness
to ensure all necessary activities and milestones are included.
Construction drawings and specifications are used to create the
schedule, which officials stated helps to ensure the entire scope is
included. However, we found several key activities were missing from
the approved baseline schedule, including redesign for ductwork;
submit, approve, fabrication, and delivery of electrical equipment;
contractor approval time for changes above $100,000; government
furnished equipment delivery milestones; systemwide testing;
and effort related to telecommunications.
Best practice: Sequencing activities;
Explanation: The schedule should be planned so that critical project
dates can be met. To meet this objective, activities need to be
logically sequenced--that is, listed in the order in which they are to
be carried out. In particular, activities that must be completed before
other activities can begin (predecessor activities), as well as
activities that cannot begin until other activities are completed
(successor activities), should be identified. This helps ensure that
interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be established and used as a
basis for guiding work and measuring progress;
Met?: Substantially met;
GAO analysis: All detail activities and milestones in the baseline
schedule are properly sequenced. Each activity--except the start and
finish milestones--has at least one predecessor or successor. Out of
5,701 detail activities, we found less than 1 percent that were not
properly driving the start date of a predecessor activity start date.
There are no lags in the schedule, as required by contract
specifications. There is one hard Finish No Later Than constraint on
the finish milestone, which the VA CFM recommended be removed. Program
management officials stated that the constraint is used solely to
calculate negative float. The VA requires a diagram of the schedule
network, similar to a PERT diagram, which clearly displays the
relationships between tasks. However, because the schedule is missing
several key activities, it is uncertain whether or not all activities
are scheduled in the correct order.
Best practice: Assigning resources to activities;
Explanation: The schedule should reflect what resources (e.g., labor,
materials, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist;
Met?: Substantially met;
GAO analysis: The VA requires schedules to be cost loaded with prorated
overhead and profit, and the total price loaded into the schedule must
equal the total contract price. Officials stated that the term
"resources" is defined as manpower by the VA schedule specifications.
Accordingly, each detail activity has an associated manpower
requirement. However, because the baseline schedule is missing key
contractor activities such as ductwork redesign, systemwide testing,
and telecommunications effort, it is uncertain how or whether resources
are properly allocated.
Best practice: Establishing the duration of activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, historical data, and assumptions used for
cost estimating should be used. Durations should be as short as
possible and have specific start and end dates. The schedule should be
continually monitored to determine when forecasted completion dates
differ from planned dates; this information can be used to determine
whether schedule variances will affect subsequent work;
Met?: Met;
GAO analysis: As required by VA schedule contract specifications,
activity durations are 20 days or less, except for procurement
activities. Our analysis shows the median task duration is 5 days.
Approximately 8 percent of the remaining activities are 1 day in
duration. All activities are based on a standard 5-day workweek with
holidays.
Best practice: Integrating activities horizontally and vertically;
Explanation: The schedule should be horizontally integrated, meaning
that it should link products and outcomes associated with other
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that the dates for
starting and completing activities in the integrated master schedule
should be aligned with the dates for supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to work
to the same master schedule;
Met?: Met;
GAO analysis: The schedule is vertically integrated, with all
activities subsumed under organized higher levels. Each activity is
mapped to an area and trade, clearly indicating which subcontractor is
responsible for what work in each area at any time. Our analysis shows
the schedule to be, in general, horizontally integrated due to the high
number of straightforward finish-start links and continuous critical
path.
Best practice: Establishing the critical path for activities;
Explanation: Scheduling software should be used to identify the
critical path, which represents the chain of dependent activities with
the longest total duration. Establishing a project's critical path is
necessary to examine the effects of any activity slipping along this
path. Potential problems along or near the critical path should also be
identified and reflected in scheduling the duration of high-risk
activities;
Met?: Substantially met;
GAO analysis: Officials stated the critical path is calculated by the
scheduling software and will become a crucial tool for managing the
construction project once the project is fully underway. Our analysis
shows the existing critical path to be structurally sound, running the
length of the schedule and encompassing several major milestones.
However, because the schedule is missing key activities, we cannot be
certain the activities are sequenced logically. It is uncertain whether
or not missing activities would appear on the critical path once
inserted into the network.
Best practice: Identifying the float between activities;
Explanation: The schedule should identify the float--the amount of time
by which a predecessor activity can slip before the delay affects
successor activities--so that a schedule's flexibility can be
determined. As a general rule, activities along the critical path have
the least float. Total float is the total amount of time by which an
activity can be delayed without delaying the project's completion (if
everything else goes according to plan);
Met?: Met;
GAO analysis: Total float represents the amount of time an activity can
slip before it affects the project finish date. It is therefore a
crucial tool for resource allocation and risk mitigation. There appear
to be excessive values of total float in the schedule. But officials
stated that the project schedule will have excessive float in some
areas. For instance, mobilization tasks in the beginning of the project
will have high float. Furthermore, officials stated that this
construction project is unique because the hospital's foundation and
control plant were constructed in prior phases. Therefore, upfront
Phase IV tasks related to work performed in earlier phases may appear
to have excessive float.
Best practice: Conducting a schedule risk analysis;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
project's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can affect the project's status;
Met?: Not met;
GAO analysis: The program has not performed a schedule risk analysis
(SRA) and officials stated that SRAs are not typically done in the
construction industry. However, best practices suggest that even at the
construction bid phase, an SRA can be used to determine a level of
confidence in meeting the completion date or whether proper reserves
have been incorporated into the schedule. An SRA will calculate
schedule reserve, which can be set aside for those activities
identified as high risk. Without this reserve, the program faces the
risk of delays to the scheduled completion date if any delays were to
occur on critical path activities.
Best practice: Updating the schedule using logic and durations to
determine dates;
Explanation: [Empty];
Met?: Partially met;
GAO analysis: At the time of the analysis the baseline schedule had not
been fully statused. Notice to Proceed (NTP) was given to the general
contractor on October 22, 2008, and work began in October. The contract
requires the general contractor to submit a network schedule to the VA
within 60 days of the NTP; yet, the schedule was not received until
April 21, 2009--181 days after the NTP. Moreover, the schedule was
received 50 days after the project executive notified the contractor on
March 2, 2009, that the schedule was over 60 days late. The VA CFM
recommended the first submitted schedule be rejected by the local
resident engineer's office. A second schedule was submitted on June 15,
2009, and the VA CFM recommended the schedule for approval on June 29,
2009. General contractor officials stated that they have been tracking
progress on an internal schedule since October 2008. The VA is
requiring the general contractor to retro-actively status the approved
schedule for the previous months. By September 2009, contractor
officials had retroactively statused the schedule up to August 2009.
Source: GAO analysis of VA information.
[End of table]
Schedule Risk Analysis:
The sole best practice that the schedule did not meet is conducting a
schedule risk analysis (SRA), which is not required by the VA schedule
specifications. VA officials told us that they do not conduct SRAs and
that a risk analysis is typically not performed in the construction
industry. In August and September 2009, we performed our own schedule
risk analysis on the construction schedule. Specifically, we analyzed
the C07P schedule, which was the latest statused version available to
us at the time of the analysis.
A schedule risk analysis uses statistical techniques to predict a level
of confidence in meeting a program's completion date. This analysis
focuses on critical path activities and on near-critical and other
activities, since any activity may potentially affect the program's
completion date. The objective of the simulation is to develop a
probability distribution of possible completion dates that reflect the
program and its quantified risks. From the cumulative probability
distribution, the organization can match a date to its degree of risk
tolerance. For instance, an organization might want to adopt a program
completion date that provides a 70 percent probability that it will
finish on or before that date, leaving a 30 percent probability that it
will overrun, given the schedule and the risks. The organization can
thus adopt a plan consistent with its desired level of confidence in
the overall integrated schedule. This analysis can give valuable
insight into what-if drills and quantify the impact of program changes.
In developing a schedule risk analysis, probability distributions for
each activity's duration have to be established. Further, risk in all
activities must be evaluated and included in the analysis. Some people
focus only on the critical path, but because we cannot know the
durations of the activities with certainty, we cannot know the true
critical path. Consequently, it would be a mistake to focus only on the
deterministic critical path when some off-critical path activity might
become critical if a risk were to occur. Typically, three-point
estimates--that is, best, most likely, and worst case estimates--are
used to develop the probability distributions for the duration of
workflow activities.
Once the distributions have been established, a Monte Carlo simulation
uses random numbers to select specific durations from each activity
probability distribution and calculates a new critical path and dates,
including major milestone and program completion. The Monte Carlo
simulation continues this random selection thousands of times, creating
a new program duration estimate and critical path each time. The
resulting frequency distribution displays the range of program
completion dates along with the probabilities that these dates will
occur. Table 10 provides a range of dates and the probability of the
project completing on those dates or earlier, based on our 3,000-
iteration Monte Carlo simulation. For example, according to our SRA,
there is a 5 percent chance that the project will finish on or before
December 1, 2011. Likewise, there is an 80 percent chance that the
project will finish on or before May 17, 2012. Because completion on
any date is uncertain, it is more realistic to show a range of possible
completion dates than to focus on a single date. In deciding which
percentile to use for prudent scheduling, there is no international
best practice standard. The chosen percentile depends on the riskiness
and maturity of the project. For some projects we emphasize the 80th
percentile as a conservative promise date. While the 80th percentile
may appear overly conservative, it is a useful promise date if a number
of new but presently unknown risks (i.e., "unknown unknowns") are
anticipated. The 50th percentile date may expose the project to
overruns.
Table 10: Probability of Project Completion:
Finish date:
As scheduled[A]: Oct. 20, 2011;
5th Percentile: Dec. 1, 2011;
50th Percentile: Mar. 1, 2012;
80th Percentile: May 17, 2012;
95th Percentile:: Aug. 23, 2012.
Months beyond scheduled finish date;:
As scheduled[A]: [Empty];
5th Percentile: 1.4;
50th Percentile: 4.4;
80th Percentile: 6.9;
95th Percentile: 10.1.
Source: GAO analysis of VA data.
[A] The actual "as scheduled" finish date in the schedule is August 29,
2011. However, this finish date does not include the 21 working days of
negative float that were in the schedule at the time of our analysis.
Moreover, an additional 22 working days (1 month) were added to allow
time for overall system commissioning. VA officials told us that the
system commissioning would take at least 1 month to complete beyond the
scheduled finish date.
[End of table]
In the case of the medical center construction schedule, our analysis
concludes that VA should realistically expect turnover from the general
contractor between March 1, 2012, and May 17, 2012, the 50th and 80th
percentiles, respectively. The must finish date of August 29, 2011, is
very unlikely.[Footnote 32] Our analysis shows the probability of
completing medical center turnover by October 20, 2011, is less than 1
percent with the current schedule without risk mitigation.
Identified risks:
The project executive identified 22 different risks as a preliminary
exercise to our SRA. Using these risks as a basis for discussion, we
interviewed 14 experts familiar with the project, including VA resident
engineers, general contractor officials, and A/E consultants. Each
interviewee was asked four general questions:
1. To provide an estimate of the probability an identified risk will
occur on the project in such a way that some activity durations are
affected. The estimated probability is translated into the percentage
of the iterations that are chosen at random during the simulation. For
example, if the expert believed weather has a 10 percent chance of
affecting some activities, then, on average, weather risk will occur in
10 percent of the Monte Carlo iterations.
2. If the interviewee believed the risk could occur, the interviewee
was asked to identify which activities' durations would be affected.
For example, activities related to steel erection or concrete pouring
may be affected if the weather risk occurs.
3. Upon identifying affected activities, interviewees were then asked
to provide a 3-point estimate for the impact on duration. These are
low, most likely, and high impact estimates. Estimates were provided as
percentages, which were applied to the activity durations in the Monte
Carlo simulation if the risk occurred. For example, if the weather risk
occurs, a 10-day steel erection activity may be affected a minimum of
110 percent, a most likely of 150 percent, or a maximum of 200 percent
(i.e., the 3-point estimates for steel erection if weather risk occurs
are 11 days minimum, 15 days most likely, and 20 days maximum). If the
risk does not occur, there is no change to the original estimated
duration.
4. Finally, interviewees were asked to identify any risks they believe
we did not account for.
We began the interviews with 22 risks and through the interview process
identified 11 more risks. During data analysis, some risks were
consolidated with others or eliminated due to a low amount of data. In
all, 20 risks were identified and incorporated into the Monte Carlo
simulation. These include 18 risk drivers, 1 schedule duration
risk,[Footnote 33] and 1 overall system commissioning activity that was
not included in the baseline schedule. The final risk drivers used in
the SRA are:
* Occupancy needs may change.
* Design may be inadequate.
* Steel design may be inadequate.
* Medical technology may change.
* Work may be misfabricated.
* Equipment may not meet design requirements.
* Subcontractors may fail.
* Suppliers may not deliver equipment on time.
* Resident Engineer (RE) staffing may be inadequate.
* Contractor field office staffing may be inadequate.
* Architect/Engineer (A/E) staffing may be inadequate.
* Labor may not be available.
* Contractor coordination problems may exist.
* Communication between RE, contractor, and A/E may be ineffective.
* May experience problems testing systems.
* Construction disciplines may not be coordinated.
* Vendor drawings may not be submitted on time.
* Change orders under $100,000 may affect schedule.
Most risks received multiple responses during the interviews. During
data analysis, we combined and analyzed data from the interviews to
create ranges and probabilities for each of the 18 risk drivers.
Because risks are multiplicative, several risks occurring on the same
activity may overestimate the true risk. That is, in the Monte Carlo
simulation, risks occur in a series, one after another, so that an
activity that has several risks may be unrealistically extended if all
risks occur. For example, drawing approval activities may possibly be
affected by RE, contractor field office, or A/E staffing being
inadequate, as well as the schedule duration risk. If all risks
occurred, drawing approval activities will most likely be
overestimated. In reality, an activity may successfully recover from
two or more risks simultaneously, so that the actual risk is not
multiplicative. Therefore, to avoid overestimation of risk, the impact
ranges of risks that occur together are reduced. That is, the 3-point
duration estimates for risks that occur together frequently were
reduced; in this particular analysis, we decreased the estimated
duration impact ranges by a factor of 0.3. This adjustment helps temper
any over-estimated risk caused by a multiplication of risk factors.
Of the 6,098 activities in the schedule, 3,193 had risk drivers
assigned to them. Some activities had one or two risks assigned, but
some had as many as seven assigned.
Prioritizing risks and risk mitigation:
Risks can impact the schedule in several ways: they can have a high
probability of occurring, have a large percentage impact on the
durations of the activities they affect, and/or they can apply to risk-
critical paths, which may differ from the baseline deterministic
critical path. Beyond applying 20 risks to the schedule, we are
interested in identifying the marginal impact of each risk. That is, we
are interested in identifying which risks have the largest impact on
the schedule, because these are the risks that should be targeted first
for mitigation.
To find the marginal impact of a risk on the total project risk at a
certain percentile, the Monte Carlo simulation is performed with the
risk removed. The difference between the finish dates of the simulation
with all the risks and the simulation with the missing risk yields the
marginal impact of the risk. Table 11 gives the priority of risks at
the 80th percentile and the marginal impact of each risk.
Table 11: Risks at the 80th Percentile:
Risk: Design may be inadequate;
Marginal impact in calendar days: 59.
Risk: Occupancy needs may change;
Marginal impact in calendar days: 48.
Risk: Change orders under $100,000 may affect schedule;
Marginal impact in calendar days: 21.
Risk: Schedule duration estimates may be inaccurate;
Marginal impact in calendar days: 18.
Risk: Construction disciplines may not be coordinated;
Marginal impact in calendar days: 16.
Risk: System commissioning may take longer than a month;
Marginal impact in calendar days: 15.
Risk: Work may be misfabricated;
Marginal impact in calendar days: 8.
Risk: Labor may not be available;
Marginal impact in calendar days: 5.
Risk: Suppliers may not deliver equipment on time;
Marginal impact in calendar days: 6.
Risk: Steel design may be inadequate;
Marginal impact in calendar days: 9.
Risk: Remaining risks;
Marginal impact in calendar days: 5.
Risk: Total;
Marginal impact in calendar days: 210.
Source: GAO analysis of VA information.
[End of table]
The marginal impact directly translates to potential calendar days
saved if the risk is mitigated. Once risks are prioritized at the
percentile desired by management, a risk mitigation workshop can be
implemented to deal with the high-priority risks in order. The
prioritized list of risks will form the basis of the workshop, and risk
mitigation plans can be analyzed using the risk model to determine how
much time might be saved. Project managers cannot expect to completely
mitigate any one risk nor is it reasonable to expect to mitigate all
risks. In addition, risk mitigation will add to the project budget.
However, some opportunities may be available to partially mitigate
risks.
Schedule issues:
During our interviews with the local VA office in North Las Vegas, we
identified several missing activities:
* Redesign for ductwork.
* Submittal, approval, fabrication, and delivery of all Division 16
(electrical equipment).
* Effort related to building the tunnel from the central plant to the
hospital basement.
* VA-furnished equipment delivery to the general contractor.
* Systemwide testing.
* Effort related to telecommunications.
Missing activities will lead to an underestimation of schedule risk
because these activities may become critical either in the baseline
schedule or the SRA. In particular, the missing fabrication and
delivery of electrical equipment assumes that the equipment will be at
the construction site when needed. Since the schedule does not contain
activities for the delivery of this equipment, risks leading to delays
in delivery of electrical equipment are not reflected in the SRA
results.
Additionally, during our analysis, we identified 58 remaining
activities with finish dates that did not drive successor activities.
That is, the activities are open ended. This is a potential problem
because the open-ended activity can have an extended duration and not
drive any successor in the SRA simulation. However, officials stated
that they were aware of these open ends and they did not believe them
to be an issue.
[End of section]
Appendix VI: Cost Increases and Schedule Delays:
We found some projects that experienced both cost increases and
schedule delays, while other projects experienced only a cost increase
and still others experienced only a schedule delay. All projects, and
whether they experienced a cost increase, a schedule delay, or both,
are in table 12.
Table 12: Projects That Experienced a Cost Increase and/or a Schedule
Delay:
Location: American Lake, WA;
Description: Seismic corrections;
Cost increase: [Empty];
Schedule delay: [Check].
Location: Anchorage, AK;
Description: Outpatient clinic;
Cost increase: [Empty];
Schedule delay: [Check].
Location: Atlanta, GA;
Description: Modernize patient wards;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Bay Pines, FL;
Description: Outpatient clinic;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Biloxi, MS;
Description: Hospital restoration/consolidation;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Gulfport, MS;
Description: Environmental cleanup;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: Syracuse, NY;
Description: Spinal cord injury/disease center;
Cost increase: [Check];
Schedule delay: [Check].
Location: Fayetteville, AR;
Description: Clinical addition;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Cleveland, OH;
Description: Brecksville consolidation;
Cost increase: [Empty];
Schedule delay: [Check].
Location: Denver, CO;
Description: New medical facility;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Gainesville, FL;
Description: Renovate patient rooms;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Indianapolis, IN;
Description: Ward modernization;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: Columbia, MO;
Description: Operating suite replacement;
Cost increase: [Empty];
Schedule delay: [Check].
Location: Las Vegas, NV;
Description: New medical facility;
Cost increase: [Check];
Schedule delay: [Check].
Location: Long Beach, CA;
Description: Seismic corrections;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Martinsburg, WV;
Description: Capital region data center;
Cost increase: [Empty];
Schedule delay: [Check].
Location: New Orleans, LA;
Description: New medical facility;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Des Moines, IA;
Description: Extended care building;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Palo Alto, CA;
Description: Seismic corrections building 2;
Cost increase: [Check];
Schedule delay: [Check].
Location: Palo Alto, CA;
Description: Centers for ambulatory care and polytrauma rehabilitation;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: Pittsburgh, PA;
Description: Medical center consolidation;
Cost increase: [Check];
Schedule delay: [Empty].
Location: St. Louis, MO;
Description: Medical facility improvement and cemetery expansion;
Cost increase: [Check];
Schedule delay: [Empty].
Location: San Antonio, TX;
Description: Ward upgrades and expansion;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: San Antonio, TX;
Description: Polytrauma center and renovation of building 1;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: San Juan, PR;
Description: Seismic corrections;
Cost increase: [Check];
Schedule delay: [Check].
Location: San Juan, PR;
Description: Seismic corrections;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Tampa, FL;
Description: Upgrade electrical system;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: Tampa, FL;
Description: Polytrauma expansion;
Cost increase: [Check];
Schedule delay: [Check].
Location: Orlando, FL;
Description: New medical facility;
Cost increase: [Check];
Schedule delay: [Empty].
Location: Temple, T[Check];
Description: IT building;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: Walla Walla, WA;
Description: Multi specialty care;
Cost increase: [Empty];
Schedule delay: [Empty].
Location: Milwaukee, WI;
Description: SCI Center;
Cost increase: [Empty];
Schedule delay: [Check].
Location: Total;
Description: 32;
Cost increase: 18;
Schedule delay: 11.
Source: GAO analysis of VA data.
[End of table]
[End of section]
Appendix VII: GAO Contact and Staff Acknowledgments:
GAO Contact:
Terrell Dorn (202) 512-6923 or dornt@gao.gov:
Staff Acknowledgments:
In addition to the contact person named above, Tisha Derricotte, Colin
Fallon, Hazel S. Gumbs, Ed Laughlin, Jason T. Lee, Susan Michal-Smith,
Karen Richey, John W. Shumann, and Frank Taliaferro also made key
contributions to this report.
[End of section]
Footnotes:
[1] The term "major medical facility project" means a project for the
construction, alteration, or acquisition of a medical facility
involving the total expenditure of more than $10 million. See 38 U.S.C.
§ 8104. For purposes of this report, we are referring to these projects
as "major construction projects."
[2] According to 38 U.S.C. §§ 8101 and 8104, the prospectus, or initial
estimate, is sent to the House and Senate Committees on Veterans'
Affairs. For purposes of this report, we refer to this as sending the
prospectus to Congress.
[3] We refer to this prospectus as the "initial estimate" throughout
this report because the prospectus contains the first estimate that VA
provides to Congress.
[4] While the Veterans Benefits Administration (VBA) and the National
Cemetery Administration (NCA) are also authorized to construct major
projects, as of August 2009, VBA had no active major construction
projects and NCA had 20 projects with a total cost of about $450
million. For the purposes of this report, we focused on VHA projects.
[5] GAO, VA Health Care: Capital Asset Planning and Budgeting Need
Improvement, [hyperlink, http://www.gao.gov/products/GAO/T-HEHS-99-83]
(Washington, D.C.: Mar. 10, 1999).
[6] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[7] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[8] Office of Management and Budget, Capital Programming Guide,
Supplement to OMB Circular A-11, Part 7, "Planning, Budgeting,
Acquisition, and Management of Capital Assets" (Washington, D.C.: June
2006); and [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[9] In this report, we only discuss costs of construction. However, the
total cost of a capital asset is its full life cycle cost, including
all direct and indirect costs for planning, procurement, operations and
maintenance, and disposal in addition to construction. See OMB Circular
A-11, Capital Programming Guide.
[10] VA must notify the House and Senate Committees on Veterans'
Affairs at least 30 days before obligating funds for a major medical
construction project that would exceed the amount authorized in law by
more than 10 percent and provide the reasons for the amount being
exceeded. See 38 U.S.C. § 8104(c).
[11] VA officials told us that construction costs have decreased since
2008 largely because of the economic downturn and construction projects
that are awarded now and in the near future may be completed at a lower
cost than they had estimated. According to VA officials, VA is
considering using the remaining unobligated appropriated funds for the
project in Las Vegas for additional construction at the medical center
site, such as adding administrative offices or a utility tunnel.
[12] [hyperlink, http://www.gao.gov/products/GAO-09-03SP].
[13] We used national data from the Department of Labor, Bureau of
Labor Statistics' Inputs to Construction Industries Producer Price
Index to identify nationwide trends in the costs of many of the
materials used in construction.
[14] VA stated that it has cost and risk assessment guidance that
requires that risk assessments should be performed at the initial
concept stage and then monitored and controlled throughout the life
cycle of the project, and should include risk information from all
stakeholders. However, VA does not conduct cost or schedule risk
analyses that would allow it to quantify its level of confidence to
finish a project at a specific cost and time.
[15] [hyperlink, http://www.gao.gov/products/GAO-09-03SP]. While the
cost guide was issued in March 2009, the guide identifies best
practices that have been widely accepted in the cost estimating field
for many years.
[16] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[17] A Monte Carlo simulation involves the use of random numbers and
probability distributions to examine outcomes.
[18] GAO, Homeland Security: US-VISIT Exit Initiatives at Varying
Stages of Completion but Integrated and Reliable Schedule for
Completing Comprehensive Exit Project Needed, [hyperlink,
http://www.gao.gov/products/GAO-10-13] (Washington, D.C.: Oct. 28,
2009).
[19] The term "major medical facility project" means a project for the
construction, alteration, or acquisition of a medical facility
involving the total expenditure of more than $10 million. See 38 U.S.C.
§ 8104. For purposes of this report, we are referring to these projects
as "major construction projects."
[20] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[21] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[22] Hulett & Associates, LLC, Los Angeles, Calif.
[23] The Veterans Health Care, Capital Asset, and Business Improvement
Act of 2003 authorized the Secretary of VA to carry out major
construction projects specified in the final CARES report, which was to
be approved by the Secretary of VA. See Pub. L. No. 108-170, § 221, 117
Stat 2042, 2050 (2003). The Secretary's report dated May 20, 2004,
listed $15 million for design of phase I of the project, which was
authorized under § 221of Pub. L. No. 108-170. Additionally, in 2006 the
project's authorization was modified to an amount not to exceed
$102,300,000. See Veterans Benefits, Health Care, and Information
Technology Act of 2006, Pub. L. No. 109-461, § 802, 120 Stat. 3403,
3443 (2006).
[24] Consolidated Appropriations Act for FY 2004, Pub. L. No. 108-199,
Div. G, Title I, 118 Stat. 3, 367-368 (2004), and Consolidated
Appropriations Act for FY 2008, Pub. L. No. 110-161, Div. I, Title II,
121 Stat., 1844, 2267 (2007).
[25] The Veterans Health Care, Capital Asset, and Business Improvement
Act of 2003 authorized the Secretary of VA to carry out major
construction projects specified in the final CARES report, which was to
be approved by the Secretary of VA. See Pub. L. No. 108-170, § 221, 117
Stat 2042, 2050 (2003). The Secretary's report dated May 20, 2004,
listed $53.9 million for construction of a spinal cord injury center,
which was authorized under § 221 of Pub. L. No. 108-170. The project's
authorization was modified in 2006 to an amount not to exceed
$77,700,000. See Pub. L. No. 109-461 § 802.
[26] Consolidated Appropriations Act for FY 2005, Pub. L. No. 108-447,
Div. I, Title I, 118 Stat. 2809, 3289-3290 (2004).
[27] Pub. L. No. 110-161, 121 Stat. 1844, 2267 (2007).
[28] The Veterans Health Care, Capital Asset, and Business Improvement
Act of 2003 authorized the Secretary of VA to carry out major
construction projects specified in the final CARES report, which was to
be approved by the Secretary of VA. See Pub. L. No. 108-170, § 221, 117
Stat. 2042, 2050 (2003). The Secretary's report dated May 20, 2004,
listed $60 million for construction of a new medical facility, design
and land purchase, which was authorized under § 221 of Pub. L. No. 108-
170. Additionally, in 2006 the project's authorization was modified to
an amount not to exceed $406,000,000. See Pub. L. No. 109-461 § 802.
The project's authorization was modified again in 2008 to an amount not
to exceed $600,400,000. See The Veterans' Mental Health and Other Care
Improvements Act of 2008, Pub. L. No. 110-387, § 702, 122 Stat. 4110,
4137 (2008).
[29] Pub. L. No. 108-199, 118 Stat. 3, 367-368 (2004).
[30] Military Quality of Life and Veterans Affairs Appropriations Act
for FY 2006, Pub. L. No. 109-114, Title II, 119 Stat. 2372, 2386-2387
(2005).
[31] Pub. L. No. 110-161, 121 Stat. 1844, 2267 (2007).
[32] At the time of our analysis, the projected finish of August 29,
2011, did not include 21 days of negative float or a month of required
system commissioning. Moreover, VA officials told us in September,
after the SRA was completed, that delays in steel procurement had
already pushed the scheduled completion date to October 2011.
[33] The schedule duration risk is applied to each activity duration to
represent the inherent inaccuracy of scheduling.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: