Proper noun features III (corpus 1)

We extract seven different learning sets according to the defined strategies in the previous post. Several important fields are selected for the comparison including the surname, forename and place fields that are our main concern here.

First, we compare the baseline model and the strategy S1, which attaches the ‘namelist’ feature to both surname and forename found in author name list. We emphasize that surname and forename are searched in pairs in S1.

Compared to the baseline, S1 obtains a gain about 0.4 points in general accuracy. The three main fields improved a little especially the recall of surname and precision of place. The other fields also have some change, which is influenced by the added features.

Now we continue comparing other CRF models constructed by the strategies S2 and S3. Here we separate ‘surnamelist’ and ‘forenamelist’ features for their corresponding token. We also search them in pairs as in S1. In S2 we attach just the name-concerned features whereas in S3 we also find ‘placelist’.

S2 gives the same performance with S1 in general accuracy(87.43 vs. 87.46). However there are anyway some improvements in precision and recall of most fields. We assume that this separation between two name types induces the model to learn more intensively the name fields. On the other hand, the ‘placelist’ feature in S3 brings only a tiny difference in all fields. It is may because of the comparatively few number of tokens having ‘place’ label or the incompleteness of the used place lists.

Strategies S4 and S5 are designed to verify if searching surname and forename in pairs (as in S1, S2, S3) is effective or not. Therefore in S4 and S5, surname and forename are searched independently in the name list.

While the general accuracies do not change much, the detailed performance shows a somewhat different pattern with the previous strategies. In both S4 and S5, the improvement of recall of forename is noteworthy. But at the same time, the recall of surname rather decreases. It means that the separation brings a favorable influence to finding forename but not for surname. We also notice an interesting result in the place field, which is better revealed in terms of precision when applying ‘placelist’ feature (S5), but gives opposite result in recall. According to our analysis, when the ‘surnamelist’ and ‘forenamelist’ are scattered here and there in learning data, an incomplete ‘placelist’ feature can disturb modeling by estimating ‘place’ label to the tokens having ‘placelist’ feature. That is why the precision is more correct but the learned model can find less tokens, which have place label.

To see a pure effect of ‘placelist’ feature on accuracy, we add just ‘placelist’ feature in S6. Like the above diagnosis, the precision for ‘place’ field is much better than that of baseline (88.41 vs. 72.51) but its recall is worse as shown in the following table (85.80 vs. 90.53) :

These results lead us to some temporary conclusions. First, inserting proper noun features improves the performance but not much. Second, we can not guarantee that imposing different features to surname and forename will obtain a better annotation result. Third, ‘placelist’ feature works for the precision of place field but not for recall. Forth, the decision about searching in pair the ‘surnamelist’ and ‘forenamelist’ does not influence on general accuracy but it makes anyway some difference depending on fields. We assume that the independence assumption between two features will be more sensible to the quality of used external list than the pair assumption.

This synthesis inspires us to tackle the incompleteness of proper noun lists. And we treat this issue as a missing feature problem.