Indexing Inex SBS Corpus in Terrier

In this blog, we demonstrate how we can convert Inex Social Book Search corpus from xml format into Trec Collection format; this format is valid as input for Terrier (compatible with TRECCollection parser) where we can apply the DFR models for book retrieval.

In our experimentation, we consider that the content of all tags in xml file is important for indexing; therefore we consider the whole xml file as one document identified by its isbn. Thus, we just need two tags instead of all tags in xml, the isbn (named isbn) and the whole content (named text) following this format:

<book>
<isbn>123</isbn>
<text>the content of first book</text>
</book>
<book>
<isbn>124</isbn>
<text>the content of second book</text>
</book>

The following python code can make this conversion.

datadest="inex collection path"
for folds in os.listdir(datdir):
os.mkdir(datadest+folds)
trectxt=""
for files in os.listdir(datdir+folds):
if files.endswith(".xml"):
content= open(datdir+"/"+folds+"/"+files,'r').read()
soup = BeautifulSoup(content)
texts = soup.findAll("book")
for text in texts:
isbn =texts[0].findAll("isbn")[0].getText()
trectxt+="<book>\n<isbn>"+isbn+"</isbn>\n"
trectxt+="<text>"+' '.join(texts[0].findAll(text=True))+"</text>\n</book>\n"
f=open(datadest+folds+"/"+folds+".xml","w")
f.write(trectxt)
f.close()

Inex SBS corpus is composed of 2,8 million documents, distributed in 1100 folders, our script generates for each folder only one Trec formatted file which contains all xml files in this folder. In fact this processing is necessary for improving the execution time of Terrier indexing process.

We have also tested some DFR models, the InL2 model gives fair results in comparison with other models participating at Inex SBS 2014 workshop, where this model has classified the second w.r.t the measure nDCG@10 the official evaluation measure for the workshop.

The results of InL2 using all tags for indexing and retrieving processes :

nDCG@10

MRR

MAP

R@1000

0,128

0,236

0,101

0,441


Vous aimerez aussi...

3 réponses

  1. Irfan Ullah dit :

    Dear Sir
    I have tried this script
    It leaves so many empty rows, how to remove them. Did you use the resulting corpus as it is in your SBS experiments or removed those empty rows?

  2. Rocky Xanadul dit :

    Hi
    I am a Java programmer with almost no useful knowledge of Python. Could you please provide the code in Java.
    Also, is it possible to share your source code of using InL2 model in your SBS experiments, it would help me start quicker in experimenting with Terrier IR platform.
    Waiting for your kind replay
    –Rocky

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search