Second experimental result on Revues.org corpus level 1 (dataset ver. 2)

In this experiment, we prepare a second version of dataset by modifying the first dataset, then learn and test a CRF model with this dataset. For the exact comparison with the previous experiment, we used the same reference instances of the learning data with the first experiment, and also those of the test.

Here we try to refine the quality of existing labels found in the previous dataset. We check out the co-occurring tags in the dataset ver. 1 to find which types of co-occurring tag (two or more) pairs need the re-labelling. For example, the <h> tag and the <title> tag co-occur more than 300 times in the tokenized data. Among them, the <hi> tags have been selected more than 200 times as labels because they were the nearest tags of given tokens. To prevent this incorrect label selection, we add another selection rule than

“Choose the closest tag for a token”,

which has been used for the labelling in the first version of dataset.

The newly added rule is

“Choose the upper tag if the closest tag of the token is one of <hi>, <abbr>, <pb>, <ptr> or <emph>”

by which we replace some inappropriate labels with the upper tags. We just leave current token if there is no upper tag. We go up the tag tree of the given token until finding an appropriate tag. If there is no appropriate token in the tag tree, we take <abbr> if it exists in the tree, or select the token’s top tag.

The experimental result with this newly prepared dataset is represented in the following tables. The overall accuracy (micro-averaged precision) is increased about 2.5% (85.34% -> 87.86%) compared to the first experiment.

Total accuracy (Micro Averaged Precision) : 87.86 % (5472/6228 * 100)

***** Precision *****

Label                       # tokens correctly labeled              # estimated tokens               # Precision

surname 311 345 90.1449275362
edition 11 13 84.6153846154
forename 268 301 89.0365448505
distributor 51 67 76.1194029851
biblscope 90 128 70.3125
settlement 1 1 100.0
lb 20 25 80.0
author 121 139 87.0503597122
orgname 18 39 46.1538461538
editor 3 3 100.0
meeting 7 13 53.8461538462
namelink 4 4 100.0
hi 13 23 56.5217391304
abbr 111 119 93.2773109244
extent 28 28 100.0
date 215 245 87.7551020408
publisher 190 236 80.5084745763
c 1436 1500 95.7333333333
title 2381 2760 86.268115942
nolabel 54 72 75.0
pubplace 139 167 83.2335329341

***** Recall *****

Label                       # tokens correctly labeled              # original tokens                   # Recall

surname 311 353 88.1019830028
edition 11 62 17.7419354839
sponsor 0 11 0.0
forename 268 319 84.012539185
distributor 51 90 56.6666666667
biblscope 90 138 65.2173913043
settlement 1 8 12.5
lb 20 36 55.5555555556
author 121 126 96.0317460317
orgname 18 42 42.8571428571
genname 0 1 0.0
pb 0 2 0.0
editor 3 25 12.0
meeting 7 69 10.1449275362
namelink 4 5 80.0
hi 13 27 48.1481481481
abbr 111 132 84.0909090909
extent 28 31 90.3225806452
date 215 257 83.6575875486
region 0 1 0.0
publisher 190 270 70.3703703704
c 1436 1478 97.1583220568
ref 0 8 0.0
name 0 1 0.0
country 0 8 0.0
nolabel 54 137 39.4160583942
title 2381 2437 97.702092737
pubplace 139 154 90.2597402597

Vous aimerez aussi...

Laisser un commentaire

Votre adresse e-mail ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search