DetEval - Evaluation software for object detection algorithms

NEW (23.7.2009)!!
Runs on windows or Linux
Choice between GUI or command line interface
Different object types
ROC plotting

DetEval is software for the evaluation of object detection algorithms. It reads XML files containing the detection results as well as the ground truth information and writes the evaluation output in XML or LaTeX format. Data for ROC curves in ASCII text format can be created, and PDF plots are supported. The criteria at which point an object is considered as detected are configurable, and graphs can be plotted showing detection performance with varying criteria.

Details of the evaluation algorithm can be found in the scientific paper published in the journal International Journal on Document Analysis and Recognition.

You need to cite the IJDAR paper in all publications which describe work for which you used the DetEval tool.

Table of contents

Introduction

The XML format and detection quality constraints

The XML format is the same for the detection results and for the ground truth. It is an extended version of the format developped for the ICDAR 2003 text detection competition organized by Simon Lucas and his team. Here is an example for a dataset containing the results on two images. I think the format is quite self explaining:

One rectangular bounding box is described by the tag "taggedRectangle". Its geometry is described by x,y,width and height, modelType is a application dependend numerical value which encodes the type or class of the object.

The result image and the groundtruth image must have the same name! However, if you decide to leave the directory path in the image names, the software can be configured to ignore the path or parts of it (see the manual for the command line interface). 

<?xml version="1.0" encoding="UTF-8"?>
<tagset>
  <image>
    <imageName>images/image1.jpg</imageName>
    <taggedRectangles>
      <taggedRectangle x="1276" y="900" width="193" height="61" modelType="17"  />
      <taggedRectangle x="348" y="844" width="197" height="105" modelType="3"  />
      <taggedRectangle x="776" y="812" width="281" height="165" modelType="42"  />
    </taggedRectangles>
  </image>
  <image>
    <imageName>images/image2.jpg</imageName>
    <taggedRectangles>
      <taggedRectangle x="1036" y="972" width="75" height="29" modelType="1"  />
      <taggedRectangle x="110" y="964" width="411" height="45" modelType="2"  />
      <taggedRectangle x="1126" y="966" width="441" height="59" modelType="3"  />
      <taggedRectangle x="604" y="964" width="349" height="53" modelType="2"  />
      <taggedRectangle x="262" y="766" width="595" height="67" modelType="17"  />
      <taggedRectangle x="948" y="766" width="461" height="71" modelType="36"  />
      <taggedRectangle x="366" y="318" width="57" height="29" modelType="14"  />
      <taggedRectangle x="598" y="844" width="411" height="89" modelType="11"  />
      <taggedRectangle x="828" y="388" width="181" height="45" modelType="11"  />
      <taggedRectangle x="1004" y="292" width="197" height="109" modelType="14"  />
      <taggedRectangle x="844" y="8" width="581" height="233" modelType="31"  />
    </taggedRectangles>
  </image>
</tagset>

The amount of overlap between detection bounding boxes and ground truth bounding boxes determines whether an object has been detected or not. More precisely, the evaluation results depend on a set of user defined thresholds, which will be used by all tools of this software. Plots can be created which give detection performance for different overlap requirements. The thresholds are described in more detail in the research paper (section 4.2, equation 10). The different tools can be parametrized with a single string containing the threshold values separated by commas:

1 Threshold tr (area recall) for one-to-one matches
2 Threshold tp (area precision) for one-to-one matches
3 Threshold tr (area recall) for one-to-many matches (splits)
4 Threshold tp (area precision) for one-to-many matches (splits)
5 Threshold tp (area precision) for many-to-one matches (merges)
6 Threshold tr (area recall) for many-to-one matches (merges)
7 A binary parameter determining whether an additional border verification should be done (not mentioned in the paper; should remain set to 0 if you don't know what that means).
8 An additional threshold on the difference of the centers of two matching rectangles (not mentioned in the paper).

Creating ROC plots

For a single detection file and its corresponding ground truth file, two values, recall and precision, can be calculated. They are defined as follows:

recall and precision

Intuitively, recall tells us how many of our objects have been detected, and precision gives us information on the amount of false alarms. Both are better if higher and should be close to 1 for perfect systems.

Usually, an object detection algorithm is supposed to be configurable with a detection parameter, which controls the amount of detection performed. More detection usually means more false alarms. The system can then be run for a large interval of parameters, which results in a range of pairs of recall and precision values. The ROC curve ("Receiver operation characteristics") gives precision on the y-axis for recall on the x-axis. It tells us the amount of false alarms we are forced to accept if we configure the system such that a required recall is produced.

Two versions

The tool comes in two versions, each coming with its own manual:

The GUI version: it is easier to use and sufficient for most cases. The command line version: it allows more control on the evaluation process and it is scriptable.

Download

File Download Remark
deteval-setup.exe http Windows binaries in a single-file installer package
deteval-bin-ubuntu-hardy.tgz http Linux binaries for Ubuntu Hardy Hedron (8.04) in a .tgz package
deteval-src-linux.tgz http Sources for Linux including compilation and install scripts.
deteval-src-windows.zip http Sources for Windows. No scripts and no support is given.

Installation

Windows binaries

A setup package is provided - just run the executable and follow the instructions. Requirement: the ruby programming language must be installed!

Linux binaries for Ubuntu hardy hedron

Download the .tgz file and extract the contents in the root directory. The files will automatically be placed into /usr/bin.

Assuming that it has been downloaded to /home/max/deteval-bin-ubuntu-hardy.tgz : 

cd /
sudo tar xvfz /home/max/deteval-bin-ubuntu-hardy.tgz

Linux sources

Download the .tgz file and extract the contents into some folder. This will create a new subfolder "deteval-linux". Go there: 

cd deteval-linux
./build.sh
sudo ./install.sh
That's it. All files are automatically installed into /usr/bin. They can be uninstalled with 
sudo ./uninstall.sh

Windows sources

The sources are provided as they are, feel free to create your own project files ...

Changes and bugfixes

Date Changes and bugfixes
21.07.2009 - Bugfixes for windows: use ruby for copying instead of shell commands. Put filenames into quotes instead of protecting spaces with backslashes. Also protect the groundtruth file
20.07.2009 - Bugfix crash when spaces in filenames
- Added --object-types option to evalplots
20.04.2009 - Added PDF export (--doplots=true option)

Requirements

Requirement: the ruby programming language must be installed, as well as libxml2-dev. The GUI requires the QT toolkit (not the command line tools). Plotting from the command line requires gnuplot as well as the epstopdf tool (not for the GUI). For ubuntu all packages are in the repository:

Necessary for the GUI version AND for the command line version: 

sudo apt-get install ruby 
sudo apt-get install libxml2 libxml2-dev

Necessary for the GUI version: 

sudo apt-get install libqt4-dev

Necessary for the command line version: 

sudo apt-get install gnuplot
sudo apt-get install texlive-extra-utils
For windows, the ruby language can be downloaded here.

Licence

Gnu Public License (GPL)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

Credits

DetEval was written by Christian Wolf, the windows port and the GUI have been written by Matthieu Bacconnier. Send comments, suggestions and bug reports to the follwing address:

christian.wolf (at) liris.cnrs.fr

You need to cite the IJDAR paper in all publications which describe work for which you used the DetEval tool.