Upper bound of annotation with proper noun features

An idea, which naturally stuck us, is to try out learning a model with a complete proper noun feature set for each strategy. It will give us an objective upper bound on the annotation performance. And from the expected upper bounds, we can decide which strategy is preferable that the others.

 The question here is how to complete proper noun lists. We figure out this problem by artificially attaching proper noun features from the real labels. For example, if a token has ‘surname’ label we attach ‘surnamelist’ feature to the token. In this way we testify the upper bound of different strategies. In the following table, we compare the upper bound of S1 and S3.

The result well depicts the limit of ‘namelist’ feature. Even though S1 works better than the baseline and produces a similar performance with S2 and S3, the upper bounds of S1 and S3(or S2) are quite different. When we apply distinct artificial features for all three fields, these fields are almost perfectly predicted whereas there is no improvement in surname and forename for the artificial feature ‘namelist’. It justifies our decision to continue to separate the ‘surnamelist’ and ‘forenamelsit’. Moreover the perfect annotation accuracy motivates us to explore in detail the completeness of proper noun features.