The ISU "Code of Points" Computerized Judging System

Prepared by Sonia Bianchetti for the World Skating Federation

INTRODUCTION

The ISU could have fought the problem of cheating in judging by adopting lifetime bans for judges caught trying to fix competition results, strengthening the accountability and the assessment of the judges, and taking over from the ISU Members the right to nominate the judges for the ISU Championships. But instead, the ISU has merely adopted policies aimed at making judging completely anonymous. Every single judge could be cheating and the public will never know. Making the judging process secret and extremely complex does not help to restore the credibility in the sport -- on the contrary!

The problem is cheating, not the scoring system.

In this paper, we examine some features of the new system, which was adopted as a "project" by the ISU Congress in Kyoto in June 2002 and later, surprisingly, appeared in the 2002 edition of the ISU General Regulations as Rule 121 paragraph 3. In spring 2003, the ISU Council decided to implement the "project-not-rule", to use Mr. Cinquanta's own words, during the 2003-2004 season at the Nebelhorn Trophy, the six Grand Prix events and the Grand Prix final. ISU Communication No. 1207 gives all the details of the new judging system indicating the relative value of all the listed elements, their grades of execution, the various levels of difficulty of all the elements, and the new requirements for a well-balanced program.

1. It is totally dependent on technology
2. It introduces new officials
3. It adopts a radically new judging system called the "Code of Points" system
4. It uses the same secret random draw for the composition of the panels of judges, plus the same anonymous scoring used in the system referred to as "the interim system" during the past season

We shall deal with these items one at a time.

1. TECHNOLOGY

The new system is totally reliant on computer technology

Judges have a computer screen in front of them. There is a real-time video display so the judges do not miss parts of the programs when entering the information. The caller identifies the elements and they appear in the various boxes on the screen. The judges enter their assessments using the buttons placed under their video display, and the numbers appear on the display as well.

The system works well and it is the best part of the project. But it is not the most important! The real issue is not whether the judges can enter the information, but rather what information is going to be entered and how it will be used.

Another important matter is the reliability of the hardware and the software. The system is complex and expensive, to the point that the ISU cannot afford to follow its own rules that mandate the use of two separate computers to calculate and confirm the results during competitions.

On the other hand, experience in recent years has shown that anything may happen with computers: from system crashes, to producing results that are obviously incorrect, to producing two different sets of results, to uncorrected data entry errors.

If the technology fails under the current system, it does not take more than a few minutes to get the competition restarted. But under the new system, the consequences of a computer malfunction become much more serious because its implementation is so completely dependent on computer technology. What happens if there is a problem with a computer during a skater's performance that blocks the judges' touch-screen displays? Will the skater have to stop skating until the computer can be restarted? On the other hand, without the computer displays, how can the judges receive the caller's information about which elements have been executed, or enter their marks? Having the marks entered directly into the computer during the skater's performance without being written as well on the judges. private protocol raises the question: what will happen if a computer crash cancels part or all of the data?

Because of the "secret judging", the raw marks input by the judges will not be available for public inspection, and there will be no way for the public to verify that the computations are being carried out correctly by the scoring software. The secrecy and complexity of the new system will combine to make it impossible for anyone to verify that the judges are not cheating and that the scoring software is not full of bugs.

Another problem is the enormous cost of all the computer equipment and the technical specialists required to set it up and operate it. The ISU does not seem to be concerned about whether the results with the "Code of Points" system can be also calculated by hand or without touch-screen displays. Apart from the ISU, who can afford the costs of the system at ISU Championships and perhaps a few elite international events -- who else can do it?

There are hundreds of competitions held each year, internationally and nationally. Thousands of skaters participate in national competitions to qualify for their national championships and, ultimately, the ISU Championships and other events controlled by the ISU. It would be far too expensive and unlikely for national federations with very limited financial means to provide all of the high-tech equipment needed to use the "Code of Points" system at all of these events. It's quite probable that most of the organisers of less important international competitions could not support the cost of the use of the computerized system at all. The ISU experts assert that the results with the "Code of Points" system can also be calculated by hand, but this is a fairy tale that not even a child would believe!

As a consequence, the skaters will have to continue to compete under the old rules at certain competitions, and under the new system at other events. Is it fair to the skaters not to have the same rules and judging standards at all competitions? And what about the judges? Will there be two classes of judges? And how will their work be assessed? There is most likely no other sport in the world in which two totally incompatible judging and scoring systems are used to judge the same competitors.

Skating has always been a unique sport, but for its beauty and its artistry -- not for such stupidity!

2. NEW OFFICIALS

The judges still judge, but their role is more or less that of a clerk

The "caller" has a lot of responsibilities in the new judging system. This person identifies all the elements and establishes the level of difficulty of all the elements other than jumps, lifts, and throws, in all disciplines. The caller awards enough points to move skaters up and down several places. The caller can also award a bonus of two points for an innovative movement, which is enough to move a skater one place and thus decide who wins. The levels that he assigns to spins and steps, at his discretion, can have the same impact on the amount of points for the grade of execution that the skaters get from the judges. This new "judge" will make critical decisions that may determine the results. In ice dancing, all the elements are assigned levels, which gives the caller immense power in the final result of the competition. The caller must display extraordinary perfection of judgement in order not to commit errors that could jeopardise the results. This is much more than is required of an individual judge. One error of the caller can change the results several places, unlike the current system where an error by an individual judge has practically no impact on the overall result. The potential for error is very high. In a competition with 24 skaters, the caller will have to make approximately 500 decisions without errors. Is this possible? To award the bonus point, the caller must also have an immense knowledge of sport throughout the world and throughout history. Will this always be the case?

The caller is assisted in decision-making by the Technical Controller, who can be a member of the Technical Committee or an ISU or International Referee. This person should confirm the decisions of the caller on the identification of the performed elements and of their levels, eventual additional elements or wrong elements, and identification of the highlight bonus. In case of a discrepancy between the two, who has the final decision?

To prepare for the competition, the caller will attend all the practices. In effect, this person will be pre-judging the competition. Historically, the judges also tend to watch the practices. But the difference is that the judges are nine, with different personal opinions. The caller is one and this person's decision affects the whole panel.

Under the current judging system, there are nine judges watching each performance, and nine judges making independent decisions about the identification and difficulty of the elements. Even if one judge misses an element or gets it wrong, the other eight very likely will get it right and the overall result will not be affected. But under the new judging system, this is all reduced to two eyes and one person's opinion. From experience, we know how frequently even very qualified persons, employed as television commentators, commit errors in identifying jumps or miss some elements! Furthermore, callers can (and very likely will) be biased, in some cases due to a conflict of interest, and judges cannot override the callers.

So far, there are no published procedures for establishing the qualifications of the callers, evaluating their performance, or making them accountable for their work. The judges are trained for years and years; they have to attend international seminars and pass practical and written examinations. How are these callers selected? Who is checking their knowledge and their integrity? Who will control their work? Considering that callers have as much power as judges or more under the new system, they should be subject to the same procedures as the judges and not simply treated as paid consultants.

3. THE NEW CODE OF POINTS SYSTEM

• Each element is assigned a base value as indicated in the Scale of Value chart.

• Each judge identifies for each element one of the seven "grades of execution". Each grade has its own + or - numerical value as indicated in the Scale of Value chart.

• The Technical Specialist determines the name and the level of every element.

• The judges mark the elements according to their grade of execution.

• At the end of the program, each judge also marks overall criteria. These so-called Program Components are: Skating Skills, Transitions, Performance/Execution, Choreography, and Interpretation. These criteria range in value from 0 to 10 with 0.25-point increments.

• For a panel of 14 judges, only 9 judges are randomly and secretly drawn and then the two highest and the two lowest marks for each Element or Program Components are deleted. This means that only 5 of the 14 judges' marks are used.

• The marks of the short program and the marks for the free skating are added together and the highest total of points determines the final result.

The WSF has accepted an independent in-depth analysis of the "Code of Points" system by Ed Russell, a mathematical statistician, available at http://www.worldskating.org/news/russell.htm.

We examine some of these problems here. We welcome comments from all concerned.

A. The tables of elements are not complete

The tables of jumps for singles and pairs are not complete. While the six most common solo jumps (toe loop, Salchow, loop, flip, Lutz, and Axel) are included, others, such as the split Lutz or flip, the double toe walley, or the inside Axel, are totally ignored. Some of these jumps are rarely executed these days, but this does not mean they should be left out forever. It is evident that the full variety and complexity of skating was not taken into consideration in developing the point model.

What happens if a skater executes one of these elements? Does he get any credit at all? And, if these elements are considered as jumps and count as one jump within the maximum number of jumps permitted in a free program, why would a skater execute a split Lutz or a series of split jumps if this means they must take out a triple Axel or a triple Lutz? The list of spins is incomplete, as well. The tables list only upright, camel, sit, and combination spins. The flying spins comprise only the flying upright, flying camel, and flying sit spin. Combination spins that commence with a jump or that include a jump transition are totally ignored.

Giving credit only to elements specifically listed in the "Code of Points" gives the skaters practically no interest or motivation for inventing new and unusual elements in their programs. The caller has the discretion to give competitors credit for a 2-point "bonus" for an innovative movement, but what defines an "innovative" movement or element? Does an "innovative" element get a bonus only in the first competition where it is used? In several competitions? Or until other skaters copy it? When will the element be considered no longer innovative? And how will new and difficult elements not listed into the point system be accommodated? If a skater invents a very difficult innovative element, will he get credit for it? The value of this bonus may not be enough to encourage the skaters to attempt new elements that are particularly difficult. Moreover, it is not clear what kind of element could qualify for the "bonus", since the "Code of Points" system establishes new maximum limits on the number of jumps, spins, and step sequences permitted in a program.

B. The information about how elements other than jumps are evaluated is too vague

This is not fair to skaters because they are not given enough information to construct their programs in order to obtain maximum credit under the new judging system.

C. The point values for some elements do not make sense

i) Jump combinations

ii) Jump sequences

iii) Evaluation of jump combinations and sequences

Does this make much sense?

Furthermore, certain errors will turn a combination into a sequence with a significant reduction in points or the potential loss of an entire element. Other errors can turn a sequence into separate solo jumps, which can also lead to the loss of one jump element as an unmarked extra element. The evaluation of possible errors in combinations and sequences is not clear at all. What if the skater touches down with one hand? Two hands? What happens then?

iv) Spins and steps

In spin combinations, no guidance is given in case of errors, except for fewer numbers of revolutions or required positions. What happens when a spin combination is interrupted in the middle? Does it become two spins?

v) Pair skating

Among the elements for pairs, the two inside death spirals are given the same base values, as are the two outside death spirals. But it is generally considered that the forward inside death spiral is easier than the backward inside, and the forward outside death spiral is much more difficult than any of the others.

And again: how are side-by-side elements handled? If the man executes a triple toe loop and the woman only a double, what happens? If he falls and she lands correctly or he executes a spin and she falls down, what happens? In a spin combination, if he does two positions and six rotations, and she does three positions and ten rotations, what happens? It seems that the fact that there are two skaters on the ice to be looked at and judged has simply been forgotten.

vi) There is too much value given to failed jumps

Skaters who completely miss a quadruple toe loop will still get 5.0 points just for making the attempt, which is more than they will get for executing a clean triple toe loop, or performing the most difficult and well executed spin combination. Badly cheated jumps will now be treated as successful jumps of one less rotation -- another complete reversal of the current rules and practices. No other sport rewards failures. Does figure skating want to become unique with this innovation instead of being unique for its artistic side?

Question: do jumps of half a rotation count as jump attempts? What about throwing the free leg without leaving the ice? Can a jump be tried again if the skater has made less than a full turn in the air?

Surely all this will just encourage skaters to include jumps in their programs even if they have little hope of executing them cleanly or well. Under the current judging rules, it is fairly clearly understood that skaters deserve no or little credit respectively for jumps marred by a fall or other serious errors, such as landing an underrotated jump on two feet, or putting a hand down or stepping out to avoid falling.

In the new system, the message that the skaters will perceive is that all elements must be attempted no matter how badly, at least to earn the falling down points. It is always worth attempting the most difficult jump, even if they cannot land it, as long as it takes off with speed, is high, strong and looks good since the penalty for the grade of execution applied for a fall in jumps (-3) becomes smaller in percentage the more difficult the jump is.

Example: if a skater falls on a triple Axel he still earns 4.5 points, which represents 60% of the full value (7.5 - 3 = 4.5). If he falls on a triple toe loop he earns only 1.5 (4.5 - 3 = 1.5), which represents 34% of the full value. In the case of double jumps it is even worse: for falling on a double Salchow, base value 1.3, the penalty to be applied is -1, which represents 76.9%. In other words, the penalty for the same error varies according to the difficulty of the element: for a triple Axel 40%, for a triple toe loop 66%, and for a double Salchow 76.9%. And a fall at the end of a well executed step sequence, "level 3", is penalised with a deduction representing 60% of the base value. Does this make sense? Is it fair?

Rewarding skaters for failed jumps and pairs for failed lifts/throw jumps does not seem to be in the best interest of the sport. Television audiences will not enjoy watching all of the falls and failures. We experienced that in the past and it proved to be a disaster. The popularity of the sport went downhill. The rules had to be changed to discourage the competitors from attempting jumps they could not execute well just hoping to get some credit. The new system will set figure skating back 20 years. There is a real danger of the skaters attempting elements far beyond their individual standards of skating, thereby resulting in a greater risk of injuries and flawed performances.

D. The requirements for a well-balanced program are changed

ISU Communication No. 1207 changes the requirements of a well-balanced program limiting the number of "jump elements" (individual jumps, combinations, or sequences) to 8 for men and 7 for ladies. There is a maximum of only two combinations or sequences. A jump combination may consist of up to three jumps and the jump sequence may consist of any number of jumps, but only the value of the two most difficult jumps will be considered. A maximum of four spins and two-step sequences is permitted.

In pair skating, as well, the maximum number of elements allowed has been reduced. But while the number of lifts or jumps and death spirals has remained more or less the same, the number of spins (solo or in combination), step sequences and spiral sequences has been changed from a minimum of one to a maximum of one. As a consequence, the senior pair free program is dominated by lifts, throws and jumps. Spins, footwork and other movements are reduced to a minimum.

The free skating programs for singles and pairs and the free dance programs are turned into 14-element required programs. The long program will become more like an extended short program rather than a true free skating program.

Skaters will have little motivation to include single and double jumps as highlight elements in their programs. These jumps will count toward the maximum number of permitted jump elements while they will get few points compared to triple and quadruple jump attempts that have been totally failed.

The restrictions on jump combinations will limit skaters' creativity and with the devaluation of jumps in sequence, it is unlikely that any skaters would consider including multiple jump sequences in their program. Skaters will also have no motivation to include additional skating elements or moves that would add difficulty and variety to their performances but earn them no credit.

Finally, another problem connected with the change of the requirements for a well-balanced program is that during the next season, in some competitions, such as the Grand Prix events, the new "Code of Points" system will be implemented -- while at other competitions, including national championships and ISU Championships, the present judging and scoring system will be used and the competitors judged according to the existing ISU Regulations and technical standards. A program that is constructed to obtain maximum credit under the "Code of Points" system will not get maximum credit under the existing rules, and vice versa. This is unfair, especially for the best competitors, who have to cope with two different training programs to meet the technical requirements set forth in the current rules and the new ones.

E. The balance between the first and the second mark has been changed

When the ISU first introduced the five marks referred to as "Program Components", the first two were described as being part of the technical merit score and the other three as the presentation marks. But as a matter of fact, "Skating Skills" and "Transitions" contain criteria that are currently considered in the mark for technical merit. The new technical merit mark, according to the ISU, would consist of the element points plus these first two subjective marks. The new presentation mark would consist of the other three subjective marks. A scale factor would also be applied to each Program Component. This factor varies for each part of each event.

Since then, they have significantly changed the identity of the Program Components. They have decided that the first two subjective marks are no longer to be considered technical merit marks, but instead are now part of the presentation points. This does not make sense, because skating skills and transitions are clearly part of technical merit, according to both the current rules and to common sense. The three other subjective marks consist of the current presentation mark broken down into three pieces.

In the "Code of Points" system, all the factors currently being judged in the presentation mark will be limited to a maximum number of points which do not represent 50% of the total score, departing from the current presentation mark which makes up half of the current total mark. While the Program Components points are forever fixed, the element points differ depending on the event and the category (men, ladies, or pairs). The lack of balance between the first and the second mark will become even more extreme in the future as technical difficulties increase over current standards, while maximum presentation points remain fixed.

For a contemporary men's free skating program in which the skater executed the 11 most difficult jumps currently being done, and did them all with +3 quality, and did the most difficult spin and footwork elements listed with +3 quality, there is a maximum of about 229 points total.

Example:

Note: R = Loop jump, L= Lutz jump

The factors that make up the current technical merit mark (which under the current system represents 50% of the total mark) become 74% of the total points for the sample men's program above. Breaking it down further, 42.4% of the total points under the "Code of Points" system is from the 11 jumps, 8.3% for the spins, 5.6% for the steps and 17.5% for basic skating skills and connecting moves.

Another interesting thing to note is that in the same men's program as above, if all the jumps are perfectly executed with +3 quality, the percentage of the (current) presentation is 26% as in the table above. If the jumps have errors such as two-footed landings, they will still get the base mark after the quality point deductions, and the presentation becomes 29%. If the jumps have even more severe errors so the jumps points end up as the base mark -1 (after the quality points deductions), the presentation becomes 31%. In all three examples, the technical element points overwhelm the presentation points, which would in no case exceed approximately 30%.

In the ladies. event, since they do one fewer jump element (7 rather than 8) and so far cannot execute quadruple jumps, for the most difficult contemporary ladies. program, jumps can make up about 38% of the total points and spins 11%, while step sequences, skating skills and connecting moves can get similar percentages as for men.

Example:

Note: R = Loop jump, L= Lutz jump

Under the current system, if you have a skater who is weak on jumps but can execute good spins and footwork -- in other words, who actually skates and presents skilfully -- such a skater will generally get presentation marks exceeding the technical merit mark and therefore, for those skaters, the presentation will weigh more than 50% of their total. Such a skater today can still earn a relatively respectable placement. Under the "Code of Points" system, though, this is unlikely.

Consider, for example, the following middle-of-the-pack examples comparing skaters who can jump, but do little else, to skaters who can spin, skate and present, but cannot do many triples. The skaters are assumed to skate "clean" programs (no deduction and no error resulting in negative quality assessments). In these examples, the skaters who can skate are pretty soundly beaten by the skaters who can only jump. Another way of looking at these examples is that the skaters who are superior in about 60% of the skating skills evaluated lose to the skaters who can do little more than jump.

The following two tables are for men's free skating.

This table is for a skater who can jump well, but cannot spin or do footwork well, and presents at a mediocre level.

Compare the table above to the following table for a skater who can do great spins and footwork and can present at a championship level, though not a medalist's level.

Now make the same comparison for the ladies. The following two tables are for ladies. free skating.

This table is for a skater who can jump well, but cannot spin or do footwork well, and presents at a mediocre level.

Compare the table above to the following table for a skater who can do great spins and footwork and can present at a championship level, though not a medalist's level.

These examples highlight another important point. The question is not just whether the "Code of Points" system gets the winner "right". It's whether it gets all the places right. Every skater has the right to be correctly placed, just as much as the medallists do. Currently, skaters who can do everything well except the jumps can place respectably. Under the "Code of Points" system, they will tend to end up behind skaters who can jump but can't do anything else well.

F. The balance between the short program and the free skating

Under the "Code of Points" system, for singles and pairs events with a short program and free skate, the points from the short program can still make up about 1/3 of the total points. With a qualifying round, however, the qualifying round becomes 21% of the total points, the short program 26% and the free skate 53%, somewhat diluting the importance of the short program.

Here are some examples of short program point totals which can be compared to the free programs of top skaters described above. These tables illustrate how the balance of elements in the short programs compares to the elements in the free skating programs.

In this example, the short program is 33% of the total points for the event. With a qualifying round, the qualifying round would be 21.2%, the short program 25.9% and the free skate 52.9% of the total points.

In this example, the short program is 34% of the total points for the event. With a qualifying round, the qualifying round would be 20.9%, the short program 26.8% and the free skate 52.3% of the total points.

In this example, the short program is 34% of the total points for the event.

Beyond the weight for each event segment, a more serious consequence of abandoning Total Factored Placement in combining segments of events is that under the current system, a skater who places first in both the short program and final free skate, and second in the qualifying round, always wins. In the "Code of Points" system, a skater can win both the short program and final free skate, place second in the qualifying round, and lose. This is equivalent to a soccer team winning two out of three games in a tournament and losing the championship to the team that won only the one game.

G. The system has not been adequately tested

Last fall, the ISU conducted two so-called "tests" of the new system, one at Skate Canada and one at the Junior Grand Prix Final. As a result of these "tests", the system was modified, so the "Code of Points" being implemented now is quite different from the one that was used for these tests. In addition, the tests carried out last year were performed on programs constructed to meet the existing ISU regulations on well-balanced programs, not the new standards described in ISU Communication No. 1207. Despite being advertised in March as ready to use, the system continues to evolve and still has many unanswered questions. By the time it will be implemented in Oberstdorf, the version of the system to be used will be so different from last season's that it will be effectively untested when it is used during that competition. Is this fair to the skaters? And to the judges?

Furthermore, the results from last year's tests have never been made public for review or feedback by skating experts other than those appointed by the ISU. It can be asserted that the system has been "tested" in the sense that it has been proved that the judges can use the computer interface and the calculations can be made and can produce a result for a competition. But the real purpose of a test should rather be to determine whether the system produces the correct result under controlled conditions. Nobody in the skating community had the opportunity to compare the two judging systems to see whether the results are acceptable or, more importantly, reflect the actual skating. Nor has the ISU ever given enough consideration to how the adoption of the new system will affect the direction of the sport as a whole, or involved the skating community to decide what kind of sport they would like to see in the future. The ISU has strongly objected to testing the "Code of Points" system as a side system, in parallel with the current system, during the coming season. Why? Are they afraid that the "Code of Points" system might prove not satisfactory?

4. SECRET JUDGING AND THE RANDOM SECRET DRAW OF THE JUDGES

Secrecy has created an immense image problem during the last season. It was perceived by the skating world as a means to sweep the crooks under the carpet. The public reaction as expressed by fans, the media, many ISU member federations and even many ISU officials was overwhelmingly negative. The credibility of skating was badly affected. The ISU has been strongly criticized by the media. In particular, the lack of any identification of which judge gave which marks, and the subsequent lack of public accountability of the judges for their actions, have been referred to by the press as a public relations disaster for the ISU.

ISU President Cinquanta has tried to minimize these concerns by emphasizing all of the fantastic features of "his" new system and pushing its adoption as the only official system for all national and international events as soon as possible.

What Cinquanta does not say is that the "Code of Points" system will continue to utilize "secret judging", that there will still be a random secret draw to select the judges whose marks will count, that none of the marks will be identified by the judge who assigned them, and that there will still be no public accountability in the judging. In other words, all the negative points of the "interim system" will be carried on into the new system as well.

The belief that selecting only a limited number of judges' marks to count makes the results more accurate is wrong. From a statistical point of view, the results are more accurate when the marks of all 14 judges on the panel are used, even if some of the judges are biased or incompetent.

CONCLUSION

The WSF thanks Dr. George Rossano for providing an external review of the manuscript, and for his helpful and valuable input and comments.

 

© 2003-2005 World Skating Federation

http://www.worldskating.org/