Active Learning
The protocol is inspired in the protocol used in the PASCAL challenge on active learning, but taking the recognition rate instead of the ROC curve as the basis of the evaluation. The evaluation measure is the Area under the Learning Curve (ALC). In our case, a learning curve will plot the recognition rate in the test set, as a function of the number of labeled samples of the training set used to make the predictions.
In order to normalize the final global score and get a value between 0 and 1 we consider two baseline learning curves:
- The ideal learning curve, obtained when perfect recognition rate is achieved. It goes up vertically then follows a recognition rate of 1 horizontally. It has the maximum area “Amax”.
- The “lazy” learning curve, obtained by making random predictions (expected value of recognition rate: 0.1). It follows a straight horizontal line. We call its area “Arand”.
To obtain the final global score displayed we normalize the ALC as follows:
global_score = (ALC-Arand)/(Amax-Arand)
Points in the curve will be obtained at steps of 50 samples in the size of the learning set. We will interpolate linearly the curve between these points.
Online Learning
The basic measure of evaluation will be the number of prediction errors. We will use two complementary measures:
1. The total number of errors after all samples have been shown to the system
2. The AUC of the curve obtained by plotting the number of prediction errors against the number of samples shown to the system.