Second experimental result on Revues.org corpus level 1 (dataset ver. 2)

In this experiment, we prepare a second version of dataset by modifying the first dataset, then learn and test a CRF model with this dataset. For the exact comparison with the previous experiment, we used the same reference instances of the learning data with the first experiment, and also those of the test.

Here we try to refine the quality of existing labels found in the previous dataset. We check out the co-occurring tags in the dataset ver. 1 to find which types of co-occurring tag (two or more) pairs need the re-labelling. For example, the <h> tag and the <title> tag co-occur more than 300 times in the tokenized data. Among them, the <hi> tags have been selected more than 200 times as labels because they were the nearest tags of given tokens. To prevent this incorrect label selection, we add another selection rule than

“Choose the closest tag for a token”,

which has been used for the labelling in the first version of dataset.

The newly added rule is

“Choose the upper tag if the closest tag of the token is one of <hi>, <abbr>, <pb>, <ptr> or <emph>”

by which we replace some inappropriate labels with the upper tags. We just leave current token if there is no upper tag. We go up the tag tree of the given token until finding an appropriate tag. If there is no appropriate token in the tag tree, we take <abbr> if it exists in the tree, or select the token’s top tag.

The experimental result with this newly prepared dataset is represented in the following tables. The overall accuracy (micro-averaged precision) is increased about 2.5% (85.34% -> 87.86%) compared to the first experiment.

Total accuracy (Micro Averaged Precision) : 87.86 % (5472/6228 * 100)

***** Precision *****

Label                       # tokens correctly labeled              # estimated tokens               # Precision

***** Recall *****

Label                       # tokens correctly labeled              # original tokens                   # Recall