First experimental result on Revues.org corpus level 1 – Part III

Part III : Evaluation (CRF on the Revues.org learning dataset ver. 1)

After flattening the levels in tags, the tokens in a reference are represented as the left side of the following table. Then we construct the first version of learning dataset by picking each token as the input and its tag as the output label that is shown in the right side. Some closest tags are not good for labels but for features and some attributes are good for tags but not for attributes. But in this trial version of dataset, we do not count in detail this kind of problems.

Experimental Setting

From 737 references in xml format, several erroneous references are eliminated then we get 716 references. Among those, 500 randomly chosen instances corresponding about 70% of total references are used as learning data and the rest are used as test data. There are 31 labels in the learning set including <nolabel> which indicates token has no label: surname, namelink, forename, c, date, title, nolabel, abbr, biblscope, pubplace, publisher, hi, author, edition, extent, distributor, meeting, orgname, camera, emph, pb, name, sponsor, settlement, country, ref, genname, editor, ptr, region, lb.

Even at a glance, the tags <pb>, <ptr> and <lb> seem not suitable as the labels for the input tokens. Moreover, the tag <hi> which marks typographies would be better if it is replaced by its attribute. And it occurs very often inside of the tag <title>, which seems more suitable as the label.  But for the meantime, we put them aside to launch our first experiment.

Measures

We conduct the truth-based evaluation, which means that we compare the estimated labels of test references with the true labels of them. As the measures, we used the micro-averaged precision, which computes the global accuracy of the estimated result, and also the precision and the recall of each type of labels. For the micro-averaged precision, we count all the correctly estimated tokens regardless of the type of labels and divide it by the total number of estimated tokens. The precision of a type of label is the proportion of the correctly estimated tokens in all the tokens estimated as the label. The recall of a type of label is the proportion of the correctly estimated tokens in all the tokens having originally the label.

The experimental result is as follows:

Total accuracy (Micro Averaged Precision) : 85.34% (5315/6228 * 100)

***** Precision *****

Label                       # tokens correctly labeled              # estimated tokens                      # Precision

***** Recall *****

Label                           # tokens correctly labeled           # original tokens                 # Recall