Aviation Security: TSA’s Revised Cost Comparison Provides a More Reasonable Basis for Comparing the Costs of Private-Sector and TSA Screeners

GAO -11-375R:

United States Government Accountability Office:
GAO:

March 4, 2011:

The Honorable John L. Mica:
Chairman:
Committee on Transportation and Infrastructure:
House of Representatives:

The Honorable Daniel E. Lungren:
Chairman:
Subcommittee on Cybersecurity, Infrastructure Protection, and Security
Technologies:
Committee on Homeland Security:
House of Representatives:

Subject: Aviation Security: TSA’s Revised Cost Comparison Provides a
More Reasonable Basis for Comparing the Costs of Private-Sector and
TSA Screeners:

As you requested, this letter discusses the Transportation Security
Administration’s (TSA) efforts to address our January 2009
recommendation to TSA to improve its methodology for comparing the
cost and performance of screening services at commercial airports
using private-sector screeners under the Screening Partnership Program
(SPP airports) and at airports using federal screeners (non-SPP
airports).[Footnote 1] Our January 2009 report identified some design
strengths in TSA’s analysis comparing the cost and performance of
screening services at SPP and non-SPP airports, such as recognizing
that cost savings would be limited by the mandated structure of the
program, but also identified 10 limitations in TSA’s methodology that
could affect the accuracy and reliability of the cost and performance
comparisons, and its usefulness in informing future management
decisions. We recommended that if TSA plans to rely on its comparison
of cost and performance of SPP and non-SPP airports for future
decision making, the agency should update its analysis to address the
limitations we identified. TSA generally concurred with our findings
and recommendation.

In summary, on January 4, 2011, TSA provided us with an update on the
status of its efforts to address seven limitations related to cost we
cited in our report, as well as a revised comparison of costs for
screeners at SPP and non-SPP airports. This revised cost comparison
generally addresses three of the seven limitations and provides TSA
with a more reasonable basis for comparing the cost of SPP and non-SPP
airports. However, TSA needs to take additional actions or provide
additional documentation to address the remaining four limitations
related to cost and the three limitations related to performance that
we identified in our January 2009 report (see enclosure). In addition,
TSA reported that it does not plan to rely solely on its cost and
performance study for future management decisions related to the SPP.
As requested, this letter focuses on the status of TSA’s efforts
related to its revised cost comparison of SPP and non-SPP airports.

Enacted in response to the September 11, 2001, terrorist attacks, the
Aviation and Transportation Security Act (ATSA) established TSA and
charged it with responsibility for securing all modes of
transportation, including civil aviation.[Footnote 2] Prior to ATSA
and the establishment of TSA, passenger and baggage screening had been
performed by private screening companies under contract to airlines.
ATSA required TSA to create a federal workforce to assume the job of
conducting passenger and checked-baggage screening at the nation’s
commercial airports.[Footnote 3] In accordance with ATSA, TSA also
created the SPP to allow commercial airports an opportunity to apply
to TSA to use private-sector screeners.[Footnote 4] These private-
sector screeners are employed by TSA-approved private-sector screening
contractors. TSA funds all passenger and baggage screening activities
at SPP airports as well as airports with federal screeners. Federal
screeners are employed by TSA and are referred to as transportation
security officers.

The observations in this letter are based on a review of TSA’s updated
January 4, 2011, cost analysis as well as discussions with agency
officials. We compared TSA’s updated study methodology as described by
TSA officials to criteria in federal accounting standards for
managerial cost accounting, GAO and Office of Management and Budget
(OMB) guidance for estimating and managing program costs, and GAO’s
Standards for Internal Control in the Federal Government.[Footnote 5]
We did not independently verify TSA’s analysis for the revised cost
comparisons. We performed this work from September 2010 through
February 2011 in accordance with generally accepted government
auditing standards. Those standards require that we plan and complete
work to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our review
objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions.

We reported in January 2009, among other things, that TSA had
underestimated costs to the government for screeners at non-SPP
airports because the agency did not include all of the costs
associated with passenger and baggage screening services at these
airports, such as workers’ compensation and general liability
insurance, and certain retirement benefits to be paid by the Office of
Personnel Management to TSA retirees at non-SPP airports. Further, TSA
did not reflect the revenue received by the government from corporate
income taxes paid by SPP contractors. The omission of these factors
reduced the reliability of TSA’s 2009 cost estimate by increasing the
costs for private-contractor screeners relative to federal screeners.
According to Statement of Federal Financial Accounting Standards No.
4, Managerial Cost Accounting Concepts and Standards, the full cost of
programs is to be reported to assist Congress and executives in making
informed decisions on program resources and to ensure that programs
get expected and efficient results.

We believe TSA has made progress in addressing three of seven
limitations related to cost we identified in our January 2009 report
and now has a more reasonable basis for comparing the cost of SPP and
non-SPP airports. Specifically, TSA’s updated cost analysis includes:

* the potential cost impact of overlapping administrative personnel;
[Footnote 6]

* costs associated with passenger and baggage screening services at
non-SPP airports, such as workers’ compensation, general liability
insurance, certain retirement costs, and a corporate tax adjustment to
reflect revenue received by the government from corporate income taxes
paid by SPP contractors; and:

* cost comparisons for multiple fiscal years.

In all, these adjustments narrowed the reported cost differences
between SPP and non-SPP airports. Specifically, in 2007, TSA estimated
that SPP airports would cost about 17 percent more to operate than
airports using federal screeners. In its January 4, 2011, update, TSA
estimated that SPP airports would cost 3 percent more to operate in
2011 than airports using federal screeners. TSA has also taken actions
that partially address the four remaining limitations related to cost,
but needs to do more to fully address them. These actions and
remaining steps are discussed in greater detail in the enclosure.

TSA reports that it is working with the Department of Homeland
Security (DHS) to further refine its estimate, which will provide
important context for any future discussions regarding the use and
relative costs of screeners at SPP and non-SPP airports.[Footnote 7]
We continue to encourage the agency to seek assistance from DHS’s Cost
Analysis Division as it considers any additional technical refinements
to its SPP cost analysis.

We will continue to follow up on the limitations related to screener
performance that we cited in our January 2009 report and report on
them as appropriate. Among other things, our report raised questions
about the performance measures used to judge screener performance and
compare the performance of screeners at SPP and non-SPP airports. For
example, TSA’s study design did not include confidence levels related
to screening performance or account for factors, such as airport
configuration or size, that could contribute to differences in
performance between SPP and non-SPP airports.

We requested comments on a draft of this letter from the Administrator
of TSA. The agency did not provide written comments. However, in e-
mails received on February 14 and 17, 2011, the TSA liaison stated
that TSA has taken several steps to address the cost and performance
limitations noted in our 2009 report and provided additional
information. For example, the liaison stated that TSA has introduced
several additional performance metrics to address the performance
limitations noted in our report. These additional metrics are designed
to help mitigate threats, assess employee performance and staffing,
and ensure adherence to TSA headquarters programs. We have updated the
draft enclosure to reflect the information the TSA liaison provided by
modifying our assessment of TSA’s actions to address the first
performance limitation discussed in the enclosure. We also updated the
enclosure based on additional information the liaison provided related
to TSA’s actions to address the sixth cost limitation and all
performance limitations.

As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 5 days
from the date of this report. At that time we will send copies to
relevant House and Senate committees and subcommittees, the Secretary
of the Department of Homeland Security, the TSA Administrator, and
other interested parties. In addition, this report will be available
at no charge on GAO’s Web site at [hyperlink, http://gao.gov].

If you or your staff have any questions concerning this report, please
contact me at (202) 512-4379 or lords@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Individuals making key contributions to
this report were Glenn Davis, Assistant Director; Daren Sweeney,
Analyst-in-Charge; Chuck Bausell; Wendy Dye; Stan Kostyla; Thomas
Lombardi; Lara Miklozek; John Warner; and Yee Wong.

Signed by:

Stephen M. Lord, Director:
Homeland Security and Justice Issues:

Enclosure:

[End of section]

Enclosure: Status of TSA’s Efforts to Address SPP Cost and Performance
Limitations:

Table 1 below provides information on the 10 limitations we cited in
our January 2009 report regarding TSA’s methodology for comparing the
cost and performance of screening services at commercial airports
using private-sector screeners under the Screening Partnership Program
(SPP airports) and at airports using federal screeners (non-SPP
airports).[Footnote 8] Table 1 also provides information on the status
of TSA’s actions to address these limitations. Specifically, TSA has
generally addressed 3 of 4 cost limitations but needs to take
additional actions or provide additional documentation to address the
remaining 4 limitations related to cost and the 3 limitations related
to performance that we identified in our January 2009 report.

Table 1: Status of TSA’s Efforts to Address SPP Cost and Performance
Limitations:

Cost limitations: 1. TSA’s study design did not consider the impact of
overlapping administrative personnel on the costs of SPP airports;
Status: Generally addressed;
Description of actions taken: TSA adjusted its cost estimates to more
accurately account for overlapping contractor management and support
staff.

Cost limitations: 2. TSA underestimated costs to the government for
non-SPP airports by not including all costs associated with providing
passenger and baggage screening services;
Status: Generally addressed;
Description of actions taken: TSA adjusted its cost estimates to
include indirect costs, such as workers’ compensation, liability
insurance, certain retirement costs, and revenue received by the
government from corporate income taxes paid by SPP contractors.

Cost limitations: 3. TSA’s analysis included more uncertainty in the
cost estimates for non-SPP airports than for SPP airports;
Status: Partially addressed;
Description of actions taken: TSA officials stated that the DHS Cost
Analysis Division offered to help TSA develop a confidence interval to
better identify the level of uncertainty of its cost estimates. We
will continue to monitor TSA’s efforts to develop additional
documentation addressing this limitation.

Cost limitations: 4. TSA’s design did not include an analysis to
determine how changes in underlying assumptions would affect the size
of the estimate, and its cost comparison did not account for
differences in screening performance or analyze the costs associated
with a particular level of performance;
Status: Partially addressed;
Description of actions taken: TSA reported that it completed cost
studies to reflect how changes in different assumptions affect the
size of the estimate. While TSA officials noted challenges associated
with estimating costs related to performance, TSA expressed interest
in continuing to work with us to address this limitation. We will
continue to monitor TSA’s efforts to develop additional documentation
addressing this limitation.

Cost limitations: 5. TSA’s comparison of SPP and non-SPP airports’
costs was based on data from a single fiscal year: 2007. However, it
is unclear how representative the costs for a single year may be;
Status: Generally addressed;
Description of actions taken: TSA officials revised its cost
comparison to include data for multiple fiscal years.

Cost limitations: 6. TSA did not ensure that cost data collected were
reliable and did not prepare documentation of its costing methodology
called for in federal accounting standards;
Status: Partially addressed;
Description of actions taken: TSA officials provided us with
additional information and briefings on its methodology and
conclusions, and reported that its estimates were reviewed by an
independent group in DHS. However, TSA did not provide sufficient
evidence related to data reliability and its costing methodology.
Federal accounting standards require a reference guide documenting
instructions for managerial cost accounting procedures and
practices.[A] This documentation helps to ensure management’s approval
of the methodology and its consistent application over time, and
provides a basis for review of the methodology by agency officials and
independent evaluators as well as a tool for training. We will
continue to monitor TSA’s efforts to develop its guidance for
addressing this limitation.

Cost limitations: 7. Key assumptions and methods used were also not
documented in sufficient detail to justify the reasonableness of costs;
Status: Partially addressed;
Description of actions taken: TSA officials provided us with
additional information on its assumptions and methodology for its
original cost estimate. However, TSA did not provide sufficient
documentation to justify the reasonableness of key assumptions and
costing methods. We will continue monitoring TSA’s efforts to identify
and justify the assumptions and methods used in its cost estimates.

Performance limitations: 1. TSA’s study design provided no evidence
showing that the five performance measures are the most meaningful
indicators of performance or that they provide a comprehensive basis
for comparing the performance of SPP and non-SPP airports;
Status: Partially addressed;
Description of actions taken: TSA collects multiple performance
measures which may be used to measure and compare performance of
screeners between SPP and non-SPP airports, and reported implementing
additional metrics since the completion of our 2009 study. However, we
believe that the measures selected to compare performance should be
those most closely linked to differences between SPP and non-SPP
airports–specifically, differences that reflect greater management
flexibilities that may exist at SPP airports. For example, TSA stated
that it uses Threat Image Projection (TIP) scores to measure security
effectiveness at airports. However, it is unclear the degree to which
differences in TIP scores at SPP and non-SPP airports would be
attributable to the SPP. Thus, we will continue monitoring TSA’s
efforts to select measures that capture differences in performance
between SPP and non-SPP airports.

Performance limitations: 2. In comparing the performance of SPP and
non-SPP airports, TSA’s design did not control or otherwise account
for other possible factors such as airport configuration or size that
could contribute to performance differences between SPP and non-SPP
airports;
Status: Generally not addressed;
Description of actions taken: TSA officials stated that they manage
SPP airports with the same standards and performance metrics as non-
SPP airports. TSA also stated that one of the performance measures,
Threat Image Projection (TIP), is not affected by airport
configuration or size. However, TSA did not provide any analysis that
observed performance differences between SPP and non-SPP airports may
reflect factors unrelated to the Screening Partnership Program, such
as the possibility that SPP airports may be different than non-SPP
airports in configuration or size. We will continue to monitor TSA’s
efforts to develop additional documentation addressing this limitation.

Performance limitations: 3. TSA’s study design did not provide any
confidence levels for estimates related to screening performance,
consistent with generally accepted statistical practices. The study
also did not address the procedures needed to ensure data reliability;
Status: Generally not addressed;
Description of actions taken: TSA has not provided confidence levels
for estimates related to screening performance or addressed the
procedures needed to ensure data reliability. TSA noted that one of
its performance measures uses an algorithm that promotes data
reliability. However, TSA’s example does not directly address the
limitation–that TSA needs to develop confidence intervals on its
estimates of performance at SPP and non-SPP airports and also ensure
the reliability of all data related to screening performance. We will
continue to monitor TSA’s efforts to develop additional documentation
addressing this limitation.

Source: GAO analysis of TSA’s updated responses on steps taken to
address limitations.

[A] Federal Accounting Standards Advisory Board: Statement of Federal
Accounting Standards No. 4, Managerial Cost Accounting Standards and
Concepts (June 30, 2009).

Legend:

Generally addressed – TSA has taken key actions to address the design
limitation noted in our prior report, GAO-09-27R, Airport Screening
(January 2009).

Partially addressed – TSA has taken some key actions to address the
design limitation noted in our prior report, GAO-09-27R, Airport
Screening (January 2009), but TSA needs to complete additional steps
or provide additional documentation to fully address the design
limitations.

Generally not addressed – TSA has taken little or no significant
actions to address the design limitation noted in our prior report,
GAO-09-27R, Airport Screening (January 2009).

[End of table]

[End of section]

Footnotes:

[1] GAO, Aviation Security: TSA’s Cost and Performance Study of
Private-Sector Airport Screening, [hyperlink,
http://www.gao.gov/products/GAO-09-27R] (Washington, D.C.: Jan. 9,
2009).

[2] See Pub. L. No. 107-71, 115 Stat. 597 (2001).

[3] See Pub. L. No. 107-71, § 110, 115 Stat. at 614-16.

[4] See Pub. L. No. 107-71, § 108, 115 Stat. at 611-13 (codified as
amended at 49 U.S.C. §§ 44919-20). TSA established the SPP after
concluding a 2-year pilot program through which private screening
companies performed screening operations at five commercial airports.

[5] Statement of Federal Financial Accounting Standards No. 4,
Managerial Cost Accounting Concepts and Standards; GAO, Cost
Assessment Guide: Best Practices for Estimating and Managing Program
Costs, [hyperlink, http://www.gao.gov/products/GAO-07-1134SP]
(Washington, D.C.: July 2007); OMB Circular A-94, Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs; GAO,
Standards for Internal Control in the Federal Government, [hyperlink,
http://www.gao.gov/products/GAO-AIMD-00-21.3-1] (Washington, D.C.:
November 1999); and OMB Circular A-76, Performance of Commercial
Activities.

[6] For example, TSA stated that it refined its Federal Security
Director (FSD) staffing allocation models to more accurately account
for contractor management of the workforce, to adjust FSD staffing to
reflect the needs of airports, and to reduce redundancies. The
Staffing Allocation Model seeks, within certain TSA constraints, to
estimate the most efficient balance of screeners needed to ensure
security and minimize wait times. See GAO, Aviation Security: TSA’s
Staffing Allocation Model Is Useful for Allocating Staff among
Airports, but Its Assumptions Should Be Systematically Reassessed,
[hyperlink, http://www.gao.gov/products/GAO-07-299] (Washington, D.C.:
Feb. 28, 2007).

[7] As noted in OMB Circular A-94, estimates of costs are typically
uncertain because of imprecision in both underlying data and modeling
assumptions. Because such uncertainty is basic to many estimates, its
effects should be analyzed and reported. Useful information in such a
report would include the key sources of uncertainty; expected value
estimates of outcomes; and the sensitivity of results to important
sources of uncertainty.

[8] GAO, Aviation Security: TSA’s Cost and Performance Study of
Private-Sector Airport Screening, [hyperlink,
http://www.gao.gov/products/GAO-09-27R] (Washington, D.C.: Jan. 9,
2009).