Invalid Interpretation of Passing Sequence Data to Assess Team Performance in Football: Repairing the Tarnished Legacy of Charles Reep

1. INTRODUCTION

A paper on performance analysis in the Journal of Sports Sciences some years ago by Hughes and Franks [1] contained some erroneous and misleading statements. The conclusions reached have since been misinterpreted by an increasing number of authors as evidence that the work of Charles Reep, the founder of performance analysis in football, is fundamentally flawed. Furthermore, it is now claimed that Hughes and Franks provided proof that possession football is more effective than direct play. All this was based on a basic misunderstanding of how to interpret and assess the effectiveness of passing sequences of different lengths. It is time to revisit this paper and to look more closely at how the authors reached their conclusions, why some of these are themselves flawed and what might be a better way of assessing the effectiveness of different playing strategies.

2. BACKGROUND

The background to this controversy has its origins in a paper by Reep and Benjamin [2], the first to apply statistical methodology to performance data at football. The authors showed that the distribution of the number of completed passes in passing sequences closely followed the negative binomial distribution. Since this distribution assumes that events are occurring at random within a certain framework of probability, the implication was that chance played a large part in influencing the length of a passing sequence. A subsequent paper [3] showed that goal-scoring followed the same distribution suggesting that a stochastic element was also at work at determining the number of goals each team scored in a game.

Charles Reep was, by at least 20 years, the first person to devise a comprehensive notational system for recording on paper the in-game actions throughout a football match, a process he began in 1950. He went on to develop advantageous tactics and strategies of play based on these recordings. The career of this innovative and remarkable man is outlined in a tribute by Pollard [4]. A more detailed description of Reep’s work and influence is given by Larsen [5], but his account of Reep’s life should not be taken as definitive, as it contains many inaccuracies and questionable claims. Lyons [6] also gives a thorough account of Reep’s career in football.

Advocating a very direct style of play, Reep had considerable success working as a performance analyst with Wolverhampton Wanderers and other teams in England during the 1950s. However, it was not until the 1980s that debate over the relative merits of direct and possession football became more intense. This was following the astonishing success of a number of little-known teams that were employing performance analysts advocating an ultra direct style of play; notably Reep with Watford and then Neil Lanham with Wimbledon, Cambridge United and other teams. The media and most coaches were united in their determination to ridicule the idea that direct football was in any way superior. The Watford forwards were described as playing like “wild dogs hurling themselves at a brick wall” while the direct style was said to be leading football back into the dark ages. Charles Hughes, the Director of Coaching at the Football Association added flames to the fire in a book in which he also seemed to be endorsing a direct approach [7]. A good account of the controversy at this time is given by Pitt [8]. Nothing definite was resolved so that by the time Hughes and Franks [1] claimed to be shedding new light onto the debate of the effectiveness of the two contrasting styles of play, a revival of the issue was long overdue.

3. CRITIQUE OF PAPER BY HUGHES AND FRANKS

The main thrust of the paper was that a closer examination of data relating to the length of passing sequences and their subsequent outcomes showed that longer passing sequences were more likely to produce goals. Although not setting out specifically to assess the merits of direct and possession football, the authors made it clear that this is how their results could be interpreted. In the abstract to the paper, they used the terms “direct play” and “possession play” as proxies for shorter and longer passing sequences, respectively. In addition, phrases such as “short direct possessions” appear in the main text. The authors first claimed that passing sequence data relating to goals needed to be expressed as a ratio before a proper interpretation could be made. They did this by dividing the number of goals from a passing sequence of a certain length by the total number of such passing sequences observed, a process they called “normalizing”.

Before looking more closely at the reasoning for the authors’ conclusions, it is first necessary to accurately define what is meant by a “passing sequence” of a certain length. The authors’ definition cannot be correct; it raises more questions than it answers and is quite different from that consistently used by Reep since 1950. In essence, Hughes and Franks were saying that after a team had gained possession of the ball, an n-pass sequence consisted of n successful passes between members of the same team, followed by a pass which was “contacted by the opposition”; where n can be zero, or any positive integer greater than zero. In fact, there are all sorts of ways that a passing sequence can be terminated other than by an intercepted pass. For example the taking of a shot, losing possession from a tackle, being fouled, being offside, the ball going out of play for reasons other than an errant pass, thus leading to a throw-in, goal kick, corner etc. We do not know how these situations were treated by Hughes and Franks, only that it introduces an element of uncertainty into the validity of their analysis.

A much more serious problem with trying to make conclusions about the effectiveness of passing sequences of different length is that it has been clearly shown that the great majority of a team’s passing sequences in a game will be of four or fewer passes no matter what style of play is being adopted. With this in mind, when a team attempts to construct a passing move of, say, 5 or more passes it will usually be unsuccessful. In this paper, data from the 1990 and 1994 World Cups were used when most teams were playing their own version of possession football. Even so, fewer than 20% of passing sequences reached 5 passes. In other words, the 0, 1, 2, 3 and 4 pass sequences will include all the debris of these unsuccessful attempts at longer passing sequences. Thus even if it could be shown that multi-pass moves were more likely to produce goals and a team sought solely to construct such moves, we are left with the undeniable and inevitable fact that the vast majority of that team’s passing sequences will still be 4 or fewer passes. This means that simply using 0 – 4 pass sequences as a proxy for direct football is meaningless since a great many of such moves will represent failed possession football. It should also be noted that Reep had never fallen into this trap, using a completely different line of reasoning to reach his conclusions about direct play as we shall see later. Hughes and Franks were not overly critical of Reep and merely hinted at the notion that their “normalizing” process could be used to suggest that possession play was more effective than direct play. Nevertheless, this is how their analysis was interpreted by others, with Reep’s work coming under fire as being hopelessly flawed.

4. AFTERMATH

In an ambitious and interesting book on the history of football tactics, Wilson [9] was the first to interpret the results of the paper by Hughes and Franks as a weapon with which to discredit Reep, claiming to have identified “a startlingly obvious flaw” in Reep’s analysis of passing sequences. Reep is then said to employ “unsubtle parameters”, to make “a basic misinterpretation of figures” and is finally accused of being a party to “wrong-headed pseudo-intellectualism”. Quite a claim in view of the fact that Reep’s direct style of play was developed not from passing sequence analysis at all, but in a quite different way, while Wilson himself is being the uncritical disseminator of flawed conclusions. In a subsequent book, Wilson [10] again dismisses Reep on the same grounds as before, referring to the “basic flaw in his mathematics”, while in a newspaper article he derides Reep as “a fussy, rather pompous figure, unsubtle of thought, and intolerant of any criticism” [11].

Others were quick to follow the lead of Wilson in publicizing what they believed was proof that Reep’s conclusions were invalid. In another much-publicized book that claims to use a scientific approach to give insight and a better understanding of events at football, Anderson and Sally [12] acknowledge the contributions of Reep to performance analysis. However, they are quick to discredit him based on the alleged flaw in his work that they claim had been found by Hughes and Franks. He was “no analyst” and his “numbers weren’t wrong: unfortunately he didn’t analyze them deeply enough” is how Reep’s work is dismissed. After first describing Reep in a headline as “the most important soccer analyst you have never heard of”, McMahon [13] then quotes Wilson and others as having shown that “Reep’s primary thesis is inherently flawed”. Medeiros [14] reiterated the views of Hughes and Franks, also quoting at length from Anderson and Sally [12] in order to justify his rejection of Reep’s work as “too simplistic”. In another well-received book by Kuper and Szymanski [15] Reep’s “crude models” are said to be “based on extremely shaky numbers”, but no justification for these comments are offered, despite the book claiming to be applying “high-powered analytical tools”.

The popular website FiveThirtyEight has also been only too eager to denigrate Reep, regurgitating the passing sequence myth and describing his analysis as “fatally flawed” and his way of thinking “discredited” [16]. More was to follow from the same website with a blog absurdly titled “How one man’s bad math helped ruin decades of English soccer” [17]. Reep is said to have made “one big, glaring mistake that changed the course of English soccer for the worse”, after which the authors conclude that “Reep was a cautionary tale of the damage that can be done when stats go wrong”. Little could Hughes and Franks have known what they were starting, but more was to follow.

In a blogpost purporting to discuss the effectiveness of different playing styles at Euro 2016, Gerrard [18] states that Hughes and Franks “provide the definitive analytical critique of Reep’s conclusions”, going on to discuss “Reep’s fallacy” and finally concluding that “Spain’s success with tiki-taka football was not a statistical anomaly but just confirmation par excellence of Reep’s misinterpretation of the evidence”. Next, Johnson [19] simply dismisses Reep’s work as “flawed” due to “faulty deduction based on a lack of quality information”. In a book describing itself as “one of the most knowledgeable football books ever written”, Cox [20] first erroneously states that Reep worked with the Football Association Director of Coaching, Charles Hughes. He goes on to describe Reep’s figures as “generally misleading” and “sometimes illogical”. Finally, at least at the time of writing, another recent book, subtitled ‘a statistical journey through the history of football’, repeats the claim of a “fundamental flaw” in Reep’s thinking [21], yet another example of taking for granted the erroneous conclusions that have propagated from the original paper by Hughes and Franks [1].

5. QUANTIFYING EFFECTIVENESS OF PLAYING STRATEGIES

If data relating solely to the lengths of passing sequences cannot be used to assess the relative effectiveness of different styles of play and of different tactics, then what other approaches might be used? When Reep devised his shorthand system of recording every on-the-ball action in a game in the early 1950s, he soon had an enormous amount of information with which to start answering this question. His approach was initially to look at successful events such as shots and goals and then to go back and analyze all aspects of the sequences of play leading up to those events. Then he realized he was able to go one step further and provide the answers to such questions as “given possession of the ball in a certain situation and at a certain position on the field, what is the most effective strategy of proceeding”; all this based on the analysis of the precise record of events from tens of thousands of passing sequences. Thus the style of play he devised had nothing to do with an analysis simply of the length of a passing sequence, but everything to do with what actions occurred before, during and after each sequence. He was soon able to quantify the results of all sorts of actions and was thus able to make statements with great authority regarding the precise effectiveness of specific actions. For example, he found that with possession of the ball in midfield, a team was on average twice as likely to subsequently produce a shot, and hence a goal if they approached the attacking third of the field with a long forward pass rather than with a sequence of attempted short passes. He found that one particular type of centre was by far the most effective, from which a strategy was devised to make maximum use of this information. Eventually, he was able to formulate a style of play, right down to the finest detail, which incorporated all the results of his meticulous work.

One drawback of this approach was that it used only shots and goals as measures of success and it did not distinguish between other outcomes, some of which might be thought of as reasonably successful and others not. For example, a passing sequence that produces a corner is obviously more effective than one that ends with the opposition in control of the ball. A method to produce a solution to this problem was described by Pollard and Reep [22]. The effectiveness of each action, or strategy, on the field, was quantified by a measure called yield, representing the number of goals that a team would expect to score from 1000 such actions, minus the number it would expect to concede. Likewise, the effectiveness of a particular passing sequence could be quantified depending on its outcome. For example, if the sequence ended in a corner, then its yield was estimated to be 23 since from 1000 corners a team would expect to score 23 more goals than it conceded. The analysis also confirmed Reep’s earlier finding about the use of a long forward pass approaching the attacking third of the field. Its yield was found to be 23.1 compared with 11.1 for approaches using only short passes.

Lanham [23, 24] extended Reep’s approach to analyzing performance and devising playing strategies by developing a computerized system of his own as early as 1985. He reached much the same conclusions as Reep with respect to an overall direct style which he has adapted and extended to the findings of his own analyses. He is also able to easily quantify the effectiveness of any specific strategy. With many years of sustained success, Lanham has contributed to over a dozen championships and promotions while being quietly employed by a series of professional teams in England.

Research specifically aimed at quantifying and comparing the success of different tactics, strategies and styles of play at football have been slow to emerge. One such study by Tenga, Ronglan and Bahr [25] reached conclusions broadly consistent with what Reep and colleagues had been advocating for years. They found that ‘counterattacks’ were twice as effective as ‘elaborate attacks’ in producing both scoring opportunities and goals. They specifically stated that their terms ‘counterattacks’ and ‘elaborate attacks’ could be considered synonymous with ‘direct play’ and ‘possession play’. Furthermore, Tenga and colleagues were the first to critically assess the paper by Hughes and Franks [1]. They correctly suggested that it might be inappropriate to use the terms ‘short possessions’ and ‘long possessions’ interchangeably with ‘direct play’ and ‘possession play’ as had been done by Hughes and Franks [1], and subsequently many others, when attempting to interpret passing sequence data. In another study specifically designed to assess the effectiveness of possession football, and using data from the five main European domestic leagues, Collett [26] concluded that, after controlling for team quality and home advantage, “the effect of greater possession was consistently negative” or, put another way, “unproductive and superfluous passing was predictive of worse team outcomes”.

Mention should also be made of two recent and comprehensive papers which give reviews of the general topics of performance analysis and match analysis in football [27, 28]. However, neither found much to add to the narrower area of comparing the relative merits of specific strategies of play and no objective study could be identified that was specifically able to conclude that possession football was more effective than direct play.

Finally, the last 20 years or so has seen a dramatic increase in the production of match performance data, especially by commercial operations such as Opta, Match Analysis and others. However, the extent to which these companies have been able to quantify and compare the effectiveness of specific strategies is something they would probably want to share only with their clients. Rein and Memmert [29] discuss what the future might hold, but at the moment the literature remains as sparse as ever with regards to new methodology able to quantify and compare the relative success of different strategies, tactics and styles of play. Other developments include player tracking technology both on and off the ball, something that was provided with real-time access throughout the 2018 World Cup finals. Bornn, Cervone and Fernandez [30] describe the details and suggest that it is time for statisticians to make use of this new data source in a way that it can be applied to developing winning strategies. Meanwhile, the debate over the relative merits of direct and possession play continues. If the invalid interpretation of passing movement data that followed the paper by Hughes and Franks [1] are rightfully disregarded, no study has yet offered any solid quantifiable evidence to suggest that the style of play advocated by Charles Reep is not the most effective strategy.

CONCLUSION

Although there is nothing inherently wrong with the way in which Hughes and Franks normalized passing sequence data, it is difficult to see what, if anything, can be gleaned from this information with respect to practical applications on the field of play. What it certainly cannot do is shed any light whatsoever on the relative effectiveness of direct and possession football. Passing sequences with only a few numbers of completed passes may represent successful direct play or unsuccessful direct play, or failed possession play or attempts at any another style of play one might care to mention, either successful or unsuccessful. We simply don’t know, so using brief passing sequences as a proxy for direct football is clearly nonsensical. The way in which the paper by Hughes and Franks has been used by others to discredit Reep, while at the same time claiming definitive proof that direct football is less effective than keeping possession, is a salutary warning as to how easily false information can disseminate itself. Other much more appropriate ways of assessing the effectiveness of different strategies of play have been in existence for over 20 years, but have been largely ignored. The few studies that have specifically compared different football strategies have mostly suggested that a direct style leads to better match outcomes, lending strong support to the general conclusions of the unjustly maligned founder of football performance analysis, Charles Reep.

REFERENCES