Postol/Lewis Review of Army's
Study on Patriot Effectiveness

Congressman John Conyers Jr, Chairman
Committee on Government Relations
House of Representatives
Washington. D.C. 20515

September 8, 1992

Dear Mr. Chairman:

I am writing to report the results of the review you requested of the U.S. Army study Analysis of Video Tapes to Assess Patriot Effectiveness. In order to perform the requested review, my colleage, Dr. George M. Lewis, and I conducted a comprehensive study of all the currently available video evidence related to Patriot's performance. To our knowledge, this is the most complete review of available Desert Storm video done to date.

Our review included all 140 videocassettes made available to your committee and the U.S. Army by the Raytheon Company and a substantial body of additional video we obtained from ABC News and from WETA-TV in Washington D.C. In addition, we supplemented our analysis of video tapes with information from press, television, CENTCOM and Pentagon reports about Scud attacks. We believe that this effort has resulted in a new level of understanding of Patriot's Desert Storm performance.

All of the relevant video data we studied has been assembled on a single 45 minute video cassette. Enclosed are copies of our report and video cassettes for each member of the National Security and Legislation Subcommittee.

The body of video assembled on the enclosed video cassettes appears to contain data on at least 22 to 23 out of roughly 47 Desert Storm engagements. Of even greater significance, the video appears to include 17 to 18 out of roughly 30 engagements in Saudi Arabia. This indicates that there is a very substantial base of video information from which an assessment of Patriot's performance can be made. Our studies also suggest that there is likely a substantial body of unreviewed video that contains still more information about Patriot's Desert Storm performance.

We urge you to ask the Department of Defense to obtain and study this additional video for evidence of Patriot's Desert Storm performance. In our opinion, such a review is essential to responsible program management of Patriot and other follow-on Theater Missile Defense systems.

We found no convincing evidence in the video that any Scud warhead was destroyed by a Patriot. We have strong evidence that Patriots hit Scuds on two occasions, but in both cases the videos also show that the Scud warheads fell to the ground and exploded. These clips provide strong evidence that even when Patriots could hit Scuds they were still not able to destroy Scud warheads.

The Desert Storm press video is consistent with a significant body of publicly released high-speed video which is shown in slow motion on many sequences in the 140 Raytheon cassettes. These videos of carefully controlled Patriot tests show that even when Patriots were able to hit aircraft and ballistic missile targets, they almost always missed the front-ends of the targets. These publicly available videos of Patriot tests suggest that Patriot's ability to destroy ballistic missile warheads also could have been very low under test conditions. We suggest that your committee obtain the high-speed video and other data from the Army's Patriot test program and review these materials to deteimme how often, and under what conditions, Patriots were able to hit the front-end warhead sections of test targets. If the publicly available video is consistent with the video from other Patriot tests, the data may well show that Patriot had almost no ability to hit Scud warheads in Desert Storm. It is also possible that this data may reveal weaknesses in the Patriot system that will need to be addressed in the PAC 3 upgrade.

In order to better quantify information from the Desert Storm press video, we have made estimates of minimum miss distances for all cases where we could clearly observe Patriots missing Scuds. That data is summarized in tabular and graphical form in figures 8, 9, and 10 in our report and it is also attached to this letter. We found that the median minimum miss distance was roughly 600 meters. This is much larger than the press video minimum resolvable miss distance of 35 to 70 meters.

To achieve lethality against Scud targets, a system like the Patriot must routinely achieve miss distances of meters to tens of meters, not hundreds to thousands of meters as observed in the video. This result of the video review by itself indicates unambiguously that there was a serious system problem with Patriot during the Gulf War.

Our review of the U.S. Army report Analysis of Video Tapes to Assess Patriot Effectiveness leads us to conclude that it is seriously flawed both at a conceptual and technical level. More than two thirds of the descriptions of Patriot video sequences in the Army report contain important omissions, misleading information, or use poorly selected video sequences. These errors would mislead readers who are not familiar with the video sequences. In order to make it possible for subcommittee members and their staffs to directly review the accuracy of both our work and that in the Army report, we have provided you and the subcommittee members with video cassettes of all the video sequences analyzed in both our and the Army's report. A brief description of the issues raised by the Army report is provided in the Executive Summary of our enclosed report. The remainder of our report, and the accompanying video cassette, documents our findings in detail.

We recommend that you continue to seek information from the Department of Defense about Patriot's Desen Storm performance. We believe that useful additional data is likely available in the high-speed video from the Patriot test program. In addition, a truly comprehensive and objective study of available press video from Desert Storm still needs to be performed. It is essential that this study, unlike the flawed Army analysis, use raw unedited video, including sound tracks. Such video should have been obtained by the Army from the television networks and other sources. We hope that the materials provided herein are of use to you and your committee.

Sincerely yours,

Theodore A. Postol
Professor of Science,Technology,and National Security Policy
Program in Science, Technology, and Society and Defense and
Arms Control Studies [Massachusetts Institute of Technology]

INSIDE THE ARMY - October 5, 1992 - page 10

An Evaluation of the Army Report
"Analysis of Video Tapes to Assess Patriot Effectiveness"
Dated 31 March 1992

A Study Performed In Response to a Request by
Congressman John Conyers, Jr.
Chairman of the House Government Operations Committee

By George N. Lewis and Theodore A. Postol
Defense and Arms Control Studies Program
Massachusetts Institute of Technology September 1992

I. Executive Summary

This study was done in response to a request by Congressman John Conyers Jr., Chairman of the House Government Operations Committee. Its purpose is to review for the Chairman a U.S. Army study titled Analysis of Video Tapes to Assess Patriot Effectiveness, (Rev 1), 31 March 1992. The Army study, hereafter referred to as "WSMR", was performed by the Material Test and Evaluation Directorate at White Sands Missile Range in New Mexico for the Office of the Deputy Undersecretary of the Army -- Operations Research. WSMR's stated purpose was to "conduct an independent analysis of Patriot effectiveness against Scuds launched during Desert Storm."

WSMR was prepared prior to the April 7, 1992 House Govemment Operations Committee hearing on Patriot's Gulf War performance. During that hearing, both the General Accounting Office (GAO) and the Congressional Research Service (CRS) presented independent testimony that Army and Raytheon claims about Patriot's Desert Storm performance were not supported by the available evidence and analysis. However, shortly before the hearing the Army stated that it had done a new analysis of the available data and it issued revised figures. Although WSMR was issued one week before the hearing, its role, if any, in the deviation of these new figures is not publicly known.

After a careful and systematic review of the WSMR study and the data it used, we have reached the following conclusions:

  1. WSMR is seriously compromised by the inadequate video data it uses as the basis of its analysis.

    WSMR used only data provided by the prime contractor for Patriot, the Raytheon Company. According to WSMR, the Raytheon data was used exclusively in order "to assure [emphasis added] that the assessment remained independent.' It is inexplicable that the authors of a study which purports to be "independent would consciously use only the data provided by a prime contractor with a vested interest in the study's outcome.

    While the Raytheon video used by WSMR appears to be a representative compilation of sequences, almost all of it is derived from newscasts. Such newscast video is nearly always edited to meet programming time, subject and priority constraints. In consequence, much useful information of relevance to an analysis of Patriot's performance is removed from this video by broadcasters. WSMR should have supplemented the Raytheon data with raw unedited video from appropriate primary sources.

  2. Although WSMR reviewed and catalogued all 140 video tapes provided by Raytheon, its analysis was based on only 27 isolated "engagement sequences." We have not been able to identify any analytical criteria that explain why WSMR sometimes chooses isolated sequences that are less complete than those available in the data.

    WSMR's analytical methodology by its very nature provides an extremely incomplete and fragmented picture of events. This serious methodological shortcoming undoubtedly contributed to the many errors we found in WSMR. Some of these errors are discussed later in this summary.

    Unlike WSMR, our analysis is based on all the video data we could correlate with each engagement. This not only includes the Raytheon videos but also unedited video provided to us by ABC and WETA-TV in Washington D.C. We also used public statements issued by CENTCOM, the Pentagon, newspapers, television newscasts, and Saudi and Israeli sources. These statements provided additional valuable contextual and supplemental information. Even with our limited resources we believe that we have produced a far more complete and accurate assessment of Patriot/Scud engagements captured on news video than that of WSMR.

  3. Almost all of the emphasis of WSMR is to prove than video cannot be used to observe effects that no one has claimed can be seen. At the same time, WSMR makes essentially no effort to assess the far more meaningful question of what can be learned from video evidence.

    For instance, WSMR argues that it is not possible to conclusively determine from video "whether a Patriot intercepted or missed a Scud target." This WSMR statement is clearly incorrect. The videos analyzed by WSMR contain numerous clear misses at distances so large as to be completely unambiguous.

  4. WSMR uses a meaningless categorization of engagement sequences which gives a false impression that the majority of intercept attempts resulted in at least some damage to Scuds. In fact, the overwhelming majority of instances where Patriots can be seen in intercept attempts against Scuds instead show clear Patriot misses by hundreds of Meters or more.

    For example, WSMR defines a "Category A event" as one in which a Patriot detonates in the "proximity" of a Scud, followed by a change in the Scud appearance. WSMR's definition of "proximity" only requires that both the Patriot and Scud appear on the same video screen! This inexplicable WSMR definition means that Patriots and Scuds are in proximity even when separated by many kilometers. Events where Scuds are missed by Patriots followed by a change in appearance when they hit the ground and detonate are also included in this category.

  5. WSMR contains important omissions, misleading information, or poorly selected sequences in more than two thirds of the 24 "unique engagement sequences" it identifies, analyzes, and counts.

    These errors would mislead a reader who was not familiar with the video data. Some of these errors are discussed below.

  6. More than 30 percent of the 24 "unique engagements sequences" WSMR identifies, analyzes, and counts are not unique.

    For example, five events, which are described by WSMR as unique engagement sequences are in fact different views of the same intercept attempt in Israel on February 19, 1991.

  7. In the remaining 16 genuinely unique engagement sequences identified by WSMR there are at least 8 sequences where WSMR either does not have or does not report video

    INSIDE THE ARMY - October 5, 1992 - page 11

    evidence for ground explosions at the end of Patriot engagements or additional Patriot misses. Thus, 50 percent or more of the remaining WSMR descriptions of genuinely unique sequences fail to report significant hard evidence that indicates Patriot failed to destroy Scud warheads.

  8. Although WSMR discusses premature detonations of Patriots, it does not dismiss a substantial body of video evidence which shows Patriots diving into the ground in both Israel and Saudi Arabia. This is inexplicable given that WSMR's catalog of the videos (WSMR appendix D) shows that they properly identified some Patriots diving into the ground.

  9. WSMR fails to use even the most elementary physical reasoning to derive important miss distance information from the video tapes. WSMR incorrectly argues that the lack of certain detailed information, such as the focal length of the video cameras used to record engagements, makes measurements of miss distances possible. This clear implication is that no useful information on miss distances can be derived from video. In fact, minimum miss distances, which are unlikely to be in error by more than 30 to 40 percent of absolute measured values, are easily derived from real-time video.

    This measuranent is accomplished by using the real-time observed motion of the Scud in the video, the known approximate speed of the Scud, and the fact that the fireball from a Patriot intercept attempt is effectively stationary in space. Since the Army did not have recording devices on any of the Patriot fire units in Saudi Arabia, the hard data provided by these videos appears to provide a uniquely valuable record of Patriot's Desert Storm performance in Saudi Arabia.

  10. WSMR is an unsigned report and there is also no list of participants. Such a practice is both inappropriate and inadvisable for at least two reasons.

    First, it is not possible for those with questions or comments about a report to communicate with those who wrote it. This denies both the WSMR authors and others interested in a document's findings the benefit of useful intellectual exchange.

    Second, and in this case perhaps of greater importance, there is no accountability associated with a report that has been written using taxpayer's money.

We conclude that WSMR is seriously flawed both at a conceptual and technical level. In addition, it is written in a way that will mislead readers who are not almdy intimately familiar with the video evidence.

We recommend that a truly comprehensive and objective study of available video be performed. It is essential that this study, unlike the flawed WSMR analysis, use raw unedited video, including sound tracks. Such video should have been obtained by the Army from the television networks and other sources. Our own studies have greatly benefited from the limited number of unedited videos provided to us by ABC and WETA-TV Washington.

The table which follows this summary compiles our assessment of the 27 sequences considered by WSMR (two of these sequences were recognized by WSMR as not being unique and one other shows only a premature Patriot detonation). Our description and interpretation of each event are based on all the available public evidence, not just the video clips considered by WSMR. The summary at the end of each event lists Patriot detonations, with estimates of minimum miss distances (in meters), as well as Scud warhead ground explosions. Patriot detonations that occur within the resolution limits of the video are detonated as fireball overlaps. Several "Additional Events" at the end of the table describe engagements not analyzed by WSMR. A single video tape which contains the extracted video data used to construct this table has been provided to the Government Operations Committee to allow the Committee's staff and members to review the accuracy of the findings summarized herein.

We conclude that the body of video we have reviewed contains data on at least 22 to 23 out of roughly 47 Desert Storm engagements. Of even greater significance, the video appears to include 17 to 18 out of roughly 30 engagements in Saudi Arabia. This indicates that there is a very substantial base of video information from which an assessment of Patriot's performance can be made.

We have found no convincing evidence in the video that any Scud warhead was destroyed by a Patriot. We have strong evidence that Patriots hit Scuds an two occasions (in WSMR Events 8 and 13), but in both cases we found video evidence that the Scud warheads fell to the ground and exploded. These clips suggest that even when Patriots could hit Scuds they were still not able to destroy the Scud warheads. We also have several other clips where it is possible that Patriots hit Scuds without detonating their warheads. but the evidence in these clips is quite ambiguous (see, for example, Additional Event 3).

In addition, we have estimated minimum miss distances for all cases where we could clearly observe Patriot missing Scuds. We present our summarized findings in tabular and graphical form in figures 8, 9 and 1O. The median minimum miss distance was roughly 600 meters. This is much larger than the press video minimum resolvable miss distance of 35 to 70 meters. To achieve lethality against Scud targets, a system like the Patriot must routinely achieve miss distances of meters to tens of meters, not hundreds to thousands of meters as observed in the video. This result of the video review by itself indicates unambiguously that there was a serious problem with Patriot during the Gulf War.

Our review of the existing video indicates that there is likely a substantial body of raw video that has yet to be examined for additional information on Patriot's Desert Storm performance. Such video should be obtained by the Army from the television networks and other sources. A failure to obtain such video and review it for information on Patriot's Desert Storm performance would simply be a conscious and inexcusable act of negligence.

[Editor's note: The tables and charts referred to in the executive summary are not included in this repory.]

INSIDE THE ARMY - October 5, 1992 - page 12