Indexing Inex SBS Corpus in Terrier

In this blog, we demonstrate how we can convert Inex Social Book Search corpus from xml format into Trec Collection format; this format is valid as input for Terrier (compatible with TRECCollection parser) where we can apply the DFR models for book retrieval.

In our experimentation, we consider that the content of all tags in xml file is important for indexing; therefore we consider the whole xml file as one document identified by its isbn. Thus, we just need two tags instead of all tags in xml, the isbn (named isbn) and the whole content (named text) following this format:

<book>
<isbn>123</isbn>
<text>the content of first book</text>
</book>
<book>
<isbn>124</isbn>
<text>the content of second book</text>
</book>

The following python code can make this conversion.

datadest="inex collection path"
for folds in os.listdir(datdir):
os.mkdir(datadest+folds)
trectxt=""
for files in os.listdir(datdir+folds):
if files.endswith(".xml"):
content= open(datdir+"/"+folds+"/"+files,'r').read()
soup = BeautifulSoup(content)
texts = soup.findAll("book")
for text in texts:
isbn =texts[0].findAll("isbn")[0].getText()
trectxt+="<book>\n<isbn>"+isbn+"</isbn>\n"
trectxt+="<text>"+' '.join(texts[0].findAll(text=True))+"</text>\n</book>\n"
f=open(datadest+folds+"/"+folds+".xml","w")
f.write(trectxt)
f.close()

Inex SBS corpus is composed of 2,8 million documents, distributed in 1100 folders, our script generates for each folder only one Trec formatted file which contains all xml files in this folder. In fact this processing is necessary for improving the execution time of Terrier indexing process.

We have also tested some DFR models, the InL2 model gives fair results in comparison with other models participating at Inex SBS 2014 workshop, where this model has classified the second w.r.t the measure nDCG@10 the official evaluation measure for the workshop.

The results of InL2 using all tags for indexing and retrieving processes :

nDCG@10

MRR

MAP

R@1000

0,128

0,236

0,101

0,441


Vous aimerez aussi...

1 réponse

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *