Note Classification in Corpus level 2
Recall that we have several problems for the treatment of the corpus level 2 that come from the nature of footnotes at the end of articles. In this posting, we show our approach to tackle the first problem reported in the previous posting, by applying a machine learning method for classification.
To apply a CRF model on notes having both bibliographic and non-bibliographic parts, we may choose to use whole notes as learning instances regardless of the existence of bibliographic reference in each note. However this approach is easy of fail because the targeted notes’ format is too diverse to construct an effective CRF model, which learns internal patterns in token sequences.
The classification of notes into two different categories (bibliographic/non-bibliographic references) is therefore a reasonable pre-processing approach that gives a set of valid notes for reference annotation. We decide to use one of the most popular classification methods, Support Vector Machine (SVM) to classify the notes.