Firearms Tactical Institute
Reprinted with express written consent from the International Wound Ballistics Association (IWBA) and the author. Publication of this document on our web site does not constitute an endorsement of Firearms Tactical Institute by IWBA or the author.
Web Site Index and Navigation Center
Reprinted from Wound Ballistics Review; 4(2), 1999: 9-13.
Discrepancies in the Marshall & Sanow "Data Base": An Evaluation Over Time
By Maarten van Maanen
Over ten years ago, several gunwriters started publishing in the popular gun-press numbers and percentages on the actual effectiveness of handgun ammunition. The first really organized tabulation of the "actual performance of bullets on the street" was an article by Evan Marshall in the May 1987 premier issue of Petersen's Handguns purportedly tabulating on the real-life effectiveness of different types of ammunition.1 This article was immediately hailed by many as the final answer to all discussions about bullet effectiveness. The percentages given in this article were accepted at face-value by many readers and were only rarely criticized. In the November 1988 issue, the same magazine published an update2, this time listing both the numbers of cases as well as the number of "one-shot stops". In 1992, a book by Marshall and Ed Sanow was published by Paladin Press3 in which new numbers were listed, followed in 1996 by a second book.4 The Marshall & Sanow writings have received several very critical reviews.5-8 In these reviews it was pointed out that the numbers listed could not be true due to the fact that the data scatter normally found in such field-data was absent. Attempts to explain this effect always left me (and I suspect most other readers without a strong technical background) somewhat unsatisfied. Thinking about a less complex way to consider this issue led me to think about issues associated with the Marshall & Sanow data base, and most importantly, a very simple demonstration that the Marshall & Sanow data is not valid.
Issue 1 -- Data Base Ground Rules
The emphasis on "one-shot stops" and "one hit to the torso" incidents in the Marshall & Sanow books and article has always surprised me. What happened in all those cases wherein the first torso hit did not incapacitate the opponent? Were additional shots fired until the opponent went down? What would you do? Marshall makes a point of the fact that these cases were ignored, but why wasn't such a case included in the not-stopped group rather than simply being ignored completely? Marshall & Sanow consider a shooting with one torso hit, no stop, and subsequent misses a "one-shot stop" failure; but a shooting with one torso hit, no stop, and a subsequent second torso hit a case to be ignored. The most reasonable assignment of multiple hit shootings is as "one-shot stop" failures; the logic of Marshall & Sanow's failure to do this needs explanation, as it seems utterly irrational. This observation was made as early 19929, but no attempt has been made to correct the problem.
Issue 2 -- Evaluating the Shootings
The total of shooting cases given in References 2, 3 and 4 are summarized in Table 1. Reference 2 (1988) shows a total of 3024 cases, Reference 3 (1992) shows a total of 6136 cases, and Reference 4 (1996) shows a total of 20742 cases. So, the number of cases added to the Marshall & Sanow "data base" was 14606 between 1992 and 1996; an average of 10 cases per day 365 days per year. This does not even consider the cases that were analyzed and rejected for inclusion as a result of this analysis. Marshall claims that appropriate cases are "...extremely difficult to obtain. For every fifty shootings that I heard about, I might obtain enough data in ten to include in this study".10 Each day's work must then include not only Marshall & Sanow's claimed careful evaluation of all the necessary details for each of the 10 shootings accepted for the "data base," but also examination of each of 40 other shootings with enough care to judge them unacceptable. I'm not sure how much work it would be to track ten cases a day but just the reading, appropriate evaluation and filing of each individual case could easily make it a full day's work. Experts evaluating shootings for court testimony typically spend much more than one day in analysis of each case.
One must conclude that the care taken in evaluating the shootings that make up the Marshall & Sanow "data" is at best far less than the author's claims in the description of the process. One could easily conclude that time available for evaluating each case inherently makes the integrity of any evaluation process indistinguishable from simply making up the cases outright.
Table 1. Total Number of Source Cases
|.22 Long Rifle||3922|
|.45 Long Colt||123||199||200|
|Average Per Year||778||3652|
|Average Per Day||2||10|
A Simple Analysis Procedure
The setup of this evaluation is simple and uses the three publications by Marshall & Sanow already mentioned (References 2, 3 and 4). In each of these publications every specific type of ammunition is listed with the total number of "one-shot stops," from which a percentage is calculated. In each following publication the number of cases has increased, giving a new total and percentage. This means that if different totals are given in each publication, the increase in both number and "one-shot stops" can be calculated by simply subtracting. The same can then be done for the third publication. The first data set is the data up to 1988 from Reference 2, the second data set is the data from 1988 to 1992 (the 1992 data from Reference 3 minus the 1988 data from Reference 2), and the third data set is the data from 1992 to 1996 (the 1996 data from Reference 4 minus the 1992 data from Reference 3). Each data set then supplements the previous data sets without any overlapping or double counting of cases, and each data set has its own "one-shot stop" percentages. The sum of all three of these data sets for any caliber and ammunition type will give the total as published in 1996 (Reference 4). These calculations are not difficult and are both easily verified and understood. The data from the three References as well as the calculated data set values are given in Table 2. Types of ammunition which have been listed in only one Reference have been omitted. References 2 and 4 list both the 2 inch barrel .38 Special shootings and the 4 inch barrel .38 Special shootings explicitly for all loads. Reference 3 lists the 2 inch barrel .38 Special shootings and all .38 Special shootings for all loads; the 4 inch barrel loads were obtained by subtraction. With this exception, each and every number of shootings and number of "one-shot stops" in the table is directly quoted from the References. The percentages are computer calculated directly from these numbers to ensure accuracy; some of these percentages shown in the References have small errors.
Note that there are 9407 shootings in various loads and calibers in the above analysis. There are 608 shootings in these same calibers that are omitted because they involve types of ammunition which are listed in only one Reference. There are 10727 shootings in calibers first appearing in Reference 4. These subsets make up the total of 20742 shootings that appear in Reference 4.
Note: Adobe Acrobat Reader required to read Table 2
Simple Analysis Shows Misrepresentations
One of the striking differences in the Marshall & Sanow publications has always been that most ammunition seems to give better results over time. Since the second and third publications are averaged results over up to three data sets, any increase in percentage must be caused by the added data set, which must have an even higher percentage. This can be easily seen in Table 2. This result in and of itself is not necessarily a problem. A more serious concern is the fact that out of a total of 251 caliber/load combinations, 8 show a 100% stopping power percentage. What cannot be explained is the fact that another 16 caliber/load combinations show either a greater than 100% stopping power percentage or negative numbers. These negative numbers mean that fewer cases are reported in a later Reference, which is in total contradiction to everything that Marshall & Sanow have claimed in developing their "data base." If this "data base" is real, why were cases dropped and what was the criteria for selecting which ones to drop? Even more importantly from the standpoint of integrity, why has no mention of this manipulation of the "data base" ever been mentioned? Any kind of secret reevaluation of the results is a clear-cut violation of any form of doing research because it is one form of "fudging" the data.
Obviously, most of the caliber/load combinations which show a "one-shot stop" percentage of 100% do not seem realistic. The Winchester 9mm 115-grain +P+ load 1988 data shows a 79% rating while the second data set (1992 minus 1988) shows a full 100% effectiveness in 22 cases. Not only does this suggest that the effectiveness of this type of ammo suddenly increased 21%, it has become nothing short of perfect. An even more miraculous jump to perfection can be seen with the .380 ACP Federal 90-grain JHP, which has a sudden jump from 65% to 100% between the second and third data sets and a 41% difference between the first and third data sets. Anyone who has ever tested any of the .380 ACP JHP-types of ammo in gelatin will know how unimpressive the performance of these rounds is. So, if this unimpressive performance is good for 100% one-stop shots, then why are rounds like the .357 Magnum, .45 ACP or .44 Magnum not showing the same results?
But this is not all. Look at the .45 ACP CCI 200-grain JHP ammo. From 74% in 1988, the 1988 to 1992 data set shows 19 stops in 16 cases, a 119% effectiveness. This, of course, is simply impossible. Yet, Table 2 shows there are eight caliber/load combinations in which such impossible greater than 100% ratings have been produced by Marshall & Sanow during one of the data taking intervals.
These greater than 100% stopping percentage or negative numbers (showing mysterious disappearing shootings) are fairly described as misrepresentations because they demonstrate conclusively that the Marshall & Sanow "data base" is not as it has been claimed to be. Specifically:
Marshall & Sanow have claimed to have continuously collected their "data base" of shootings over time; this makes having fewer shootings in particular caliber and load combinations at later dates impossible, but eight such conditions exist in their "data base."
Marshall & Sanow have eight particular caliber and load combinations that show a completely impossible greater than 100% "one-shot stop" percentages in their "data base."
The simple analysis procedure I have used show a clear basis for claims of unreasonable characteristics in the Marshall & Sanow "data base" in earlier reviews.5-8 However, I believe that demonstrating the misrepresentations by the simple arithmetic method I have used adds an element that was missing in the previous reviews. This simple proof can be followed completely and understood by every reader, including myself and others who lack a strong technical background.
I believe the Marshall & Sanow "data base" is completely discredited by the impossible conditions shown to exist in it.
Marshall, EP: "The Lethal Truth About Handgun Stopping Power," Petersen's Handguns, 1987; 1(1); 32-37, 85.
Marshall, EP: "One-shot Stopping Power." Petersen's Handguns, 1988; 2(6): 24-29, 68-71.
Marshall, EP; Sanow EJ: Handgun Stopping Power: The Definitive Study. Paladin Press, Boulder, Colorado, 1992.
Marshall, EP; Sanow EJ: Street Stoppers: The Latest Handgun Stopping Power Street Results. Paladin Press, Boulder, Colorado, 1996.
Fackler, ML: "Marshall - Sanow Can't Beat Long Odds: Wound Wizards Tally Too Good to be True." Soldier of Fortune, January 1994, 64-65.
Roberts, GK; Wolberg, EJ: "Book Review: Handgun Stopping Power: The Definitive Study." AFTE Journal, 1992; 24(4): 383-387.
Fackler, ML: "Book Review: Street Stoppers: The Latest Handgun Stopping Power Street Results." Wound Ballistics Review, 1997; 3(1): 26-31.
MacPherson, D: "Sanow Strikes (Out) Again." Wound Ballistics Review, 1999; 3(1): 32-35.
Fackler, ML: "Police Handgun Ammunition Selection." Wound Ballistics Review, 1992; 1(3): 36.
Marshall, EP; Sanow, EJ: Handgun Stopping Power: The Definitive Study. Paladin Press, Boulder, Colorado, 1992: 43.
Delivering you informative multimedia essays about the "battlefield problem-solving" tactical aspects of armed self-defense.
Web Site Index and Navigation Center
© 2000 Firearms Tactical Institute. All Rights Reserved.
FirearmsTactical™, Salus In Periculo, and logo are trademarks of Firearms Tactical Institute.