2013 Championship: Einstein Scoring Error

Blog Date: 
Wednesday, May 15, 2013 - 10:04

Many people are aware there was a scoring error on the Einstein field, but not everyone knows the details.  I’d like to explain what happened and our plans to minimize the chance of this happening again. 

We knew that the automated scoring system in use had a design limitation which could result in disc count errors, so a manual count verification process was put in place at the start of the season to double check counts before scores were finalized. The goals over the driver stations had hinged bottoms to allow easy removal of discs. On Einstein, a field reset person would use a pole to push up on the hinge, allowing the discs to drop in to two separate bins, one on the left side, one on the right. The number of discs in each bin would be counted, written on a scratch piece of paper, totaled, and then the number of discs scored in autonomous would be subtracted from that total count before being transferred to the official score sheet for the match in the ‘teleop’ section. In Semi-Final 2-2 on Einstein, the field reset crew added 20 and 26 on the scratch paper and wrote ‘64’ instead of ‘46’, accidentally reversing the numbers. 10 discs had been scored in autonomous, so the official score sheet showed 10/54 for auto/teleop discs, rather than the correct totals of 10/36. 

The alliance captain pointed out the scoring error to the Chief Referee. We reviewed the information, spoke with the field reset crew, and found the scoring error. The field reset crew was very forthcoming about the mistake, and was obviously shaken that it had happened. We told the teams we were correcting the score, and would be going to a third match in that series. Dean came over to talk to the alliance that we erroneously believed had made it to the finals. I went over to the alliance as well, apologized, and showed them the sheets on which the error had been made.

The problem was not with the field reset team, but with the overall system we had in place, which exposed too many opportunities for error. In most cases, the manual disc verification system functioned well and resulted in accurate disc counts, but on Einstein and several other events, we know for certain it did not. At those events during which the errors were caught in time, they were corrected. However, I know there were events at which errors were not caught until it was too late to correct, and to those teams, I apologize. I’m certain this was a painful experience.   

Everyone makes mistakes, particularly under high-pressure situations, and I think it’s unlikely that we will be able to fully eliminate human judgment from every possible scoring or refereeing situation. Much of our ability to automate scoring depends on the game, some of which are relatively easy to score automatically, some realistically can’t be scored automatically at all. In any case, we remain committed to making sure that the scoring system in each FRC game is as accurate as possible. We’ll be working to reduce the opportunities for errors by employing more robust system checks and testing the system earlier and even more thoroughly, taking in to account any potential human factors and unlikely ‘corner cases’ that still seem to crop up with the number of matches we run at FRC events  - almost 7,000 in the 2013 season. The FRC staff, just like the FRC teams, want matches to be won or lost based on what actually happened on the field.

 

I’ll blog again soon.

Frank

Comments

I would like to recommend that if there's a field fault in elims, both alliances subject to the replay or rubber match should be granted at least the standard 6 minute break to regroup (and to the extent possible, the same amount of time if longer). Even if it requires a field timeout, it's poor practice to play a prepared alliance against an unprepared one, whether lacking technically or strategically.

Thank you for all the work that FIRST volunteers put into each and every season. This year I had an experience with scoring errors, but the amount of effort and dedication to this incredible organization is truly appreciated. The crazy field designs and intricate game play is difficult as a team but must be even more so as a volunteer. I would just like to thank all those people that make FIRST possible. Good luck to all the teams.

I cant even begin to imagine how disturbing this must have been to the drive team of the non-winning" team being told the error. The issue I have with this is the stress put on the kids being made to replay the match. As mentioned errors occurred & were not "caught" in other important and, probably, final/championship matchs. Once a score is posted it should stand. Look at sports when a penalty call is missed, a runner is called safe/out and the score is affected. Wins/losses are not changed or games replayed after the fact. Right or wrong once a match is decided the decision should stand.

In our case, no replays were required to correct the error. The score was simply corrected to what all officials agreed it should have been, which led to a third match in the series. To do otherwise would have led to the Championship results being called in to question.

In sports, bad decisions stand. But FIRST is not really a sporting event - it is to build scientists and engineers, who will build planes, bridges and buildings. Truth and ethics are needed in these professions. Think back to the Challenger disaster. Engineers reported problems with the boosters, but Thiokol hid it. The problem turned out to be catastrophic. Astronauts lost their lives. I applaud the admission of error on Einstein. It is heartbreaking to lose a robotics event at that high level. Better that than learn to hide errors which might result in the next Challenger disaster.

We try to teach our students to take a holistic system view of the robot, including the drive team interaction, and given the chance, try to test combinations of things to may be see the unexpected. We've had a couple of seasons where in hindsight, there was some combination in particular that we should have tested.

FWIW, you do a great job with the FLL score sheet the student refs used. It did a good job of minimizing mistakes in the scoring process itself.

KC

Luckily the scoring inconsistency was so large that people could figure out what had happened. If not the outcome of Einstein could have been drastically different.

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

All comments should embody the FIRST values of Gracious Professionalism® and will be moderated prior to posting. Thank you for helping to keep the conversation civil and productive.