A Review of the Postol and Lewis Evaluation of the White Sands Missile Range Evaluation of the Suitability of TV Video Tapes to Evaluate Patriot Performance During the Gulf War

INSIDE THE ARMY - November 16, 1992, pages 7-9

by Peter D. Zimmerman

For more than 18 months, since April 1991, MIT Professor Theodore Postol has been conducting a campaign to attempt to prove that the Patriot's performance in Desert Storm "may have been an almost total failure." The latest in these attempts is his "evaluation" of a report, prepared by the White Sands Missile Range (WSMR) Army Materiel Test and Evaluation Directorate, which examined the use of TV video tapes to assess Patriot performance in Desert Storm. The bottom line of WSMR's study was that "using the videos did not allow us to conclusively determine whether a Patriot intercepted or missed a Scud target, what the miss distance was, where on the Scud an intercept occurred, or what the effect was on the Scud warhead." Since these videos formed the cornerstone of Postol's claims of Patriot failure, he has now gone to great lengths to try to discredit the WSMR findings. However, he is not successful and in fact spends more time restating his own case, which Steve Hildreth, a senior analyst from the Congressional Research Service, previously described as "worthless," than finding any substantive fault with the WSMR analyses.

Before examining the flaws in Postol's latest evaluation, it is useful to examine the history of his prior "analyses," to put into context what he now claims. He became interested in Patriot's performance in the Gulf when what appeared to be Army claims of 90+ percent success rates seemed too good to be true. In this regard, some of the Army's wounds in the battle of Patriot success statistics were, in part, self inflicted. Early Army reports of Patriot performance, issued before any detailed analyses had been performed, centered around what the Army called "intercepts" - defined as being engagements where the Patriot detected and tracked the target, launched a missile, guided it to a continually updated intercept point and detonated the Patriot warhead. Although this definition did not necessarily mean that the enemy missile had been killed, the distinction was lost on many people, who tended to accept these reports using the ordinary interpretation of the word intercept - that the target was dead. The debate of the last 18 months might have been avoided had the Army simply used ordinary terminology to characterize what they knew, prior to the first statements that were approved on actual success rates in April 1991.

Postol's campaign has been marked by a continual shifting and change in his underlying arguments, as one by one his analyses have been discredited.

In April 1991, citing ground damage reports in an Israeli newspaper, Postol argued that Patriot failed because his calculations indicated that the level of ground damage, on a per TBM basis, increased after Patriot was deployed. This was subsequently shown to be based upon error in his counts of TBMs before and after Patriot was deployed and his failure to distinguish between superficial damage, such as broken windows, and major damage such as buildings destroyed. By winter of that year, Postol had virtually abandoned this line of argument.
In the summer and fall of 1991, Postol, in collaboration with an Israeli reporter, Reuven Pedatzur, was relying heavily on supposed statements from not-for-attribution Israeli experts and unnamed reports. Twice the Israeli embassy in Washington issued statements countering those claims. By early 1992, Postol had quietly stopped using Israeli reports. Pedatzur continued, however, until the April 1992 congressional hearing. After Pedatzur quoted some quantitative information allegedly provided to him by retired Israeli Air Force chief of staff MG Avihu Ben-Nun, a letter from Ben-Nun was read into the record, specifically denying Pedatzur's claims.
With ground damage and quotes from Israeli experts gone as arguments, Postol next turned to commercial TV video tapes taken during the war. In January 1992, on a network TV news program, Postol showed one of his video tapes of a Patriot-Scud engagement and said definitively, "This is a miss," by "hundreds of feet." Subsequently I, and other scientists familiar with video analysis, pointed out that the slow frame rate of commercial TV would not allow such a conclusion to be drawn, since the relative travel of the Scud and Patriot in between frames could be nearly 100 meters. A successful intercept that took place within that 100 meters would be totally missed by the TV camera. By the time of the April 1992 hearing, Postol's "hundreds of feet" had become "hundreds of meters." In addition, he acknowledged that "...you cannot determine with the space and time resolution of this kind of video camera whether or not the Patriot detonated at the right time and the right location [to do] damage to the target" and he admitted that his videos were "crude". In response to a congressman's question, "Would you be the first to say that your analysis is not a full analysis, that [it] is a very meager analysis," Postol's answer was "Absolutely, sir." Nevertheless, his conclusion about Patriot performance remained unshaken.
Postol's shift to "hundreds of meters" required some sort of reference yardstick if it was to withstand scrutiny. He conveniently found such a yardstick, at about the time of the April hearing, in the fireball created when the Patriot warhead detonated. He claimed "The [Patriot] fireball is 100 meters or more in size." Once again he was in error, this time by more than an order of magnitude. The Patriot fireball is in the order of eight to 10 meters in diameter, which can be established by simple physical principles and confirmed with an existing photograph of a Patriot fireball taken on the test range. The Patriot warhead would have to be equivalent to a nuclear yield of about two kilotons to create a 100 meter fireball.
So Postol needed yet another yardstick, or his "hundreds of meters" misses would become tens of meters, and thus subject to the imprecision of the TV cameras, to which he had already admitted. The much needed new yardstick appeared in his report on the White Sands Study. Using the fact that the fireball is relatively stationary in space, he calibrated the frame to frame movement of the Scud relative to the edge of the fireball. If he knew the speed of the Scud, the frame time of the camera and the geometry between the camera and the Scud trajectory, he could do what he claims. Unfortunately for his argument, he knows none of these things. The speed of the Scud at intercept altitude can differ by a factor of three the frame rate of the camera can differ by two times, and the unknown geometry of the camera can produce apparent differences on the two-dimensional TV screen of more than a factor of five because the TV cameras are often located near the targets of the Scuds and so can easily be looking nearly head on at the incoming missiles. The combination of these effects can introduce an error in his calculations of over 2,000 percent, not the 30 to 40 percent he claims. His report depends almost entirely on this new yardstick, which cannot reliably distinguish 50 meters from 1,000 meters.

Postol's efforts have shown a repeated pattern: attack and engage along one line of argument; see that argument defeated in technical "combat" with some independent scientist; retreat to find a new way to "demonstrate" his preconceived conclusion; and then sally forth once more, only to lose the next technical exchange and be forced to seek yet another line of attack using yet another theory. And when technical arguments fail altogether, Postol uses a totally different method of attack. He deconstructs his opponents' arguments into isolated phrases and disconnected clauses, and then misuses those out-of- context questions as if they were taken correctly from technical papers with which Postol disagrees.
The most recent example of this use of "Postolian logic" appeared in Inside the Army on October 5, 1992. It contained the executive summary of a report on the evaluation performed by Postol and his assistant, Dr. George Lewis, of WSMR's evaluation of video tape evidence. Postol and I were separately tasked by Congressman John Conyers, chairman of the congressional committee investigating Patriot's performance in the war, with performing a review of the WSMR report.
In the first paragraph of Postol's report he begins the subtle process of twisting the facts to suit his needs. He provides a partial quote from the task statement to WSMR to set them up for his criticisms which follow. Postol states "WSMR's stated purpose was to 'conduct an independent analysis of Patriot effectiveness against Scuds launched during Desert Storm.'" He then faults WSMR on a variety of counts for not doing the complete analysis which he claimed was their task. Left out from what he calls their stated job, however, is the rest of the WSMR task order, which significantly narrows the scope of their charge: "...to focus the analysis on assessing whether the video tapes are useful in determining the following: a. Intercept or miss; b. Miss distance; c. If intercept occurred, near what areas of the Scud the Patriot warhead detonated; d. Instances where Patriot caused the destruction, dudding or incineration of the Scud warhead; e. Continuity of video tapes." The difference is clearly significant, but ignored by Postol. Since Postol's video evidence was at issue and was found wanting by all independent experts, including WSMR, he conveniently shifted the terms of the discussion to something WSMR was never asked to do, and then criticized them for not doing it.
WSMR, upon request, was provided with copies of 140 videotapes Raytheon had purchased from the networks. These tapes contained every segment showing a Patriot-Scud engagement available on the market. In highlighting the fact that their analysis was independent, WSMR stated, "To assure that the assessment remained independent, we reviewed only the 140 video tapes shipped to WSMR by Raytheon...We did not review any analyses of the tapes that may have been conducted by Raytheon or the Project Manager [emphasis added]. The point clearly made was that WSMR did not want to influence their findings by studying what Raytheon or the Patriot Project had to say about the tapes.
But Postol quotes selectively and deceptively to make a totally invalid point. He says "WSMR used only data provided by the prime contractor [Postol's emphasis]....According to WSMR, the Raytheon data was used exclusively in order 'to assure [emphasis added by Postol] that the assessment remained independent.' It is inexplicable that the authors of a study which purports to be 'independent' would consciously use only the data provided by a prime contractor with a vested interest." Postol's incomplete WSMR quite and his twist on the point they were making is clearly designed to mislead.
In defining what WSMR categorized as the best video tapes to analyze (their category A), they include those sequences "which show [amongst other things]:...A presumed Patriot warhead detonation that occurred in the proximity (on the video screen) of the incoming object, presumably a Scud...[which] then experienced a flaring-up or explosion (within two seconds)." Note that what WSMR was saying was that on the TV screen, the patriot detonation and the presumed Scud appeared to be in proximity to one another when the Patriot detonation occurred. Postol deliberately twists this around and writes that "WSMR's definition of 'proximity' only required that both the Patriot and Scud appear on the same video screen! This inexplicable WSMR definition means that Patriots and Scuds are in 'proximity' even when separated by many kilometers." It is clear from what WSMR really wrote that their definition meant only that they were analyzing events as they appeared on a TV screen. Postol's omission of the direct quote purposely misleads whomever reads his report.
If any one of these "misunderstandings" were an isolated event, it could be allowed to pass as literary hyperbole. However, they are not isolated and rely on the reasonable assumption that the reader of Postol's paper will not undertake a line-by-line study of the WSMR report. Since Postol's report does not footnote its quotations, locating them in the original report to examine their context requires a certain degree of persistence. I must therefore reluctantly conclude that the examples I have cited form part of a deliberate pattern.
As part of this pattern, Postol uses other deceptive techniques. In his point numbered 10, Postol excoriates the WSMR report for being unsigned, which is, of course, the customary way government reports are issued. Postol then accuses WSMR of shying away from accountability and making it impossible for anyone to discuss the findings of the report with its authors. This is nonsense. With one phone call to the listed commercial telephone number for White Sands, I was connected to the director of the WSMR Test and Evaluation Directorate, who then unhesitatingly transferred me directly to the coauthors of the report. They, in turn, willingly discussed numerous points they had raised, many of which I agreed with, and a few of which I did not. Surely Postol could have been as creative in his use of AT&T or MCI.
In his report, Postol calculates a "median minimum miss distance," from all of his videos, of about 600 meters. From Postol's point of view this number is very significant, since it is well beyond the approximately 100 meter inescapable frame rate error, which is the principal reason why these videos cannot be used to categorize any close explosion as being either a hit or a miss. The problem is that Postol's median statistic is meaningless.
First, the median miss distance could not have been calculated correctly by Postol, since, as was shown above, he has no reliable yardstick. His estimate of the fireball diameter was off by a factor of 10 and his latest measure, using the stationary fireball edge, is uncertain by at least a factor of 20. Second, the median miss distance for all Patriot-Scud engagements shown on TV includes a number of very different cases - those that were assessed by the Army as having been successful (i.e., direct hits or near misses), those that were assessed as failures, those that were the second of two shots after the first had destroyed the target, those that were engaging debris (that might not have been hot enough to be visible on TV), etc. Clearly there were many Patriot missiles that did not destroy Scud warheads. The miss distances associated with these missiles are totally irrelevant and except for the effect of artificially coming up with a big number, a statistic that includes them has no practical meaning. Tom Lehrer, statistician, mathematician and humorist, who himself once taught at MIT, pointed out that there are useful statistics, such as the average age of a particular population, and useless statistics. Lehrer said that "the most useless statistic is the average telephone number." Postol's 600 meter median miss distance is akin to the average telephone number.
Finally, Postol winds up his executive summary by suggesting that it was wrong for WSMR to consider only the publicly available TV clips, because these videos had been edited by the networks. Ironically, this was an argument that I and others had raised against Postol's own use of these videos. He further suggests that the Army should obtain the "substantial body of raw video that has yet to be examined" from the networks. He references a "limited number of unedited videos provided to us by ABC." While Postol was able to obtain raw video from ABC-TV because he was working directly with its reporters and so was playing a journalistic role himself, the Army is not likely to be granted similar favors. Postol most likely knows this. I have personally consulted reporters and producers who now work or who have worked at ABC, CBS, NBC, PBS, and NPR about the policies of their networks on releasing outtakes to the government for the purpose of analysis. In each case the answer was not merely "no" but "hell no," sometimes embellished in less printable terms.
Postol has shifted his technical arguments five times, the last three of which depend upon the analysis of news video tapes. WSMR firmly concluded, as I did, that these videos could not be use to determine Patriot hits or misses, and that such use was simply wrong. When I reviewed WSMR's report I found some errors, particularly the double counting of the same engagements, but overall I felt that their findings were valid. Postol, on the other hand, could not allow the WSMR findings to stand. Having no solid scientific basis upon which to challenge their results, he turned, as he has done in the past, to using partial quotes, phrases out of context and a variety of other misleading tricks to "prove" them wrong. WSMR was not wrong, and Postol's misleading critique cannot remain unchallenged.
Peter D. Zimmerman, a physicist, is Senior Fellow for Arms Control and Verification at the Center for Strategic and International Studies. He testified on the use of video tapes to determine Patriot success at the April 7, 1992, hearing of the Legislation and National Security Subcommittee of the House Committee on Government Operations.