Wikipedia, bibliographies and reference annotation

[This post was co-written by OpenEdition’s Mathieu Orban de Xivry, Élodie Faath and Marin Dacos,]


The introduction of bibliographic references in the online encyclopaedia Wikipedia follows on from discussions about how it is used by its contributors. Readers are invited to take a look at the articles by Pierre-Carl Langlais and Gilles Sahut retracing the process by which the inclusion of references in Wikipedia has come to be seen as the norm (Langlais 2015; Sahut 2014).

With the aim of responding to its contributors’ need to indicate their sources1, Wikipedia has developed a syntax that makes it possible to add a reference to the article text either by using the visual editing interface or the wikicode interface2. These recommendations make it clear that the references must be described as completely as possible: author, title, publishers, date, language.

This post aims to provide an initial and reliable quantitative estimation of the bibliographic references (whether annotated or not) in the French Wikipedia corpus. The study covers a total of over 3 million references.

The bibliographic references taken into account are:

  • those found in the “Bibliography” zones
  • those found in the body text and which are indicated as such by a reference tag <ref>

This assessment is based on the application of an automatic reference annotator developed by OpenEdition (, currently deployed on the platforms and OpenEdition Books, to a large corpus such as that available on Wikipedia.


Data corpus

The corpus used3) is the French Wikipedia portal, available at this address: It covers a large swathe of the French Wikipedia corpus. However, protected articles, though present, are devoid of text.

The Wikipedia corpus is in an XML that is relatively difficult to manipulate, primarily because of the use of a templating language, wikicode4, within the document itself.

The document’s XML tags are poor: few tags, and little detail in the tags. The document is structured into sections (wikicode); the name of these sections depends on the language of the corpus.

Bibliography extraction tool

In order to extract the bibliographies, we used and enriched the Extract academic citations from Wikipedia utility developed by Aaron Halfaker ( Two extractors corresponding to the two types of bibliographic reference were added to this utility. The use of python generators developed by this utility makes it possible to process almost infinite volumes of text.

Extraction of bibliographic references

References were extracted at different levels of the article and we calculated the number of annotated and unannotated references in Wikipedia.

Extraction of bibliographies

We have taken different types of bibliographies into account:

  • bibliographies flagged as such: this is generally the only bibliography zone5
== Bibliographie ==

* {{Article|langue=fr|prénom1=AnneMarguerite|nom1=Fryba|titre=[[Maurice Grammont]], Antoine Meillet et l'institutionnalisation de la linguistique en France|périodique=Revue des langues romanes|numéro=105|année=2001|pages=503-517}}

* {{Ouvrage|langue=fr|titre=Meillet aujourd'hui|prénom1=Gabriel|nom1=Bergounioux|prénom2=Charles|nom2=de Lamberterie|lieu=Louvain-Paris|éditeur=Peeters|année=2006|isbn=978-9-042-91743-9}}
  • primary or secondary bibliographies: these zones correspond to a single section level as defined by wikicode6
=== Bibliographie primaire ===

*{{Ouvrage |langue=français |auteur1=Maurice Leblanc |titre=Arsène Lupin, gentleman-cambrioleur (1907) |sous-titre= |éditeur=Editions du groupe « Ebooks libres et gratuits » |collection= |lieu= |année=2004 |mois=janvier |volume= |tome= |pages totales=264 |passage= |isbn= |lire en ligne= |id=Leblanc1907 }}


=== Bibliographie secondaire ===

*{{Ouvrage|langue=français|auteur1=André-François Ruaud|titre=Les nombreuses vies d'Arsène Lupin|lieu=Paris|éditeur=Les moutons électriques|collection=La bibliothèque rouge|année=2008|pages totales=350|isbn=978-2-915793-59-8 | année première édition =2005|lire en ligne=|passage=}} {{plume}}

*''Europe'', revue littéraire mensuelle, août septembre 1979, {{numéro|604/605}} (numéro consacré à Maurice Leblanc et Arsène Lupin)

  • bibliographies composed of subparts: these different subparts correspond to a lower section level as defined by wikicode7
=== Bibliographie ===

==== Généralités ====

* {{Ouvrage |langue=fr |prénom1=Peter A.|nom1=Clayton|titre= Chronique des pharaons|sous-titre=L'histoire règne par règne des souverains et des


==== Hiéroglyphes ====

* {{Ouvrage |langue=fr |prénom1=Maria Carmela|nom1=Betrò|titre=Hiéroglyphes Sud|pages totales=986|isbn=978-2-7427-8922-1}}


==== Mythologie ====

* {{Ouvrage |langue=fr |prénom1=Aude|nom1=Gros de Beler|titre=La mythologie égyptienne|éditeur=Grand Livre du Mois|pages totales=133|lieu=Paris|année=1998|isbn=2-7028-1107-8}}

  • sources8: this is a frequent synonym for the bibliography. The rules are the same as those set out above.

Extraction of bibliographic references

After examining the extracted corpus, we detected references sorted according to three labels:

Unannotated references

"* Vincent Blanloeil: ''Une introduction moderne à l’algèbre linéaire''. Ellipses, 2012."

Partially annotated references9

"* [[Donald Knuth|Donald E. Knuth]], ''[[The Art of Computer Programming]]'', Addison Wesley, 2011,{isbn|978-0321751041}}"

Fully annotated references10

" {{Ouvrage|prénom1=Thierry|nom1= Bardinet|lien auteur1=|titre=Les papyrus médicaux de l'Égypte pharaonique| lieu=Paris| éditeur=Fayard| année= 1995|pages totales=591|isbn=2-213-59280-2}}"

It should be noted that there are different “Models” [modèles], like, all of which are taken into account.

Extraction of bibliographies in reference cues

These references are in the <ref> tags throughout the text11.

Unannotated references

 "* Vincent Blanloeil: ''Une introduction moderne à l’algèbre linéaire''. Ellipses, 2012."

Partially annotated references

Jacques Fontaine, Nécrologie, ''Annuaire de l'association des anciens élèves de l'École normale supérieure'', 1981, p. 89.

Il a dédié à sa mère son ''Histoire littéraire des grandes invasions germaniques'' [ Riulbic, Bic Basin Council {{en}}]

Fully annotated references
A posteriori processing was conducted to remove annotations that are well structured but correspond to web links: {{lien web|lang=en|url=… »}}. In this particular instance we are not interested in these structured links, which nevertheless represent almost 900 000 references.


Identified biases

Among the fully annotated references, there are false positives such as: * {{ Autorité }}. Here we identified 7 527 false positives. However, it is possible that other marginal false positives may have been (wrongly) taken into account.


Out of a total of 4 015 496 processed pages, 870 274 Wikipedia pages were exploitable. Many pages are mere revisions or protected articles, etc., and do not contain texts. Furthermore, we are only analyzing pages that contain a reference or bibliographic zone.

We observe that the proportion of unannotated references in the bibliographic zones is 53.8% and that this rises to 77.4% for references in the body text. The vast majority of references in the French Wikipedia corpus are not at all annotated.

A third level of label (partially annotated references) was integrated so as to document the complexity of bibliographic conventions12, which, despite theoretically making it possible to cleanly annotate bibliographic references, in reality leave too great a degree of flexibility and often introduce ambiguity into the annotations of these references. This label can be applied to 10% of the references found in the bibliographic zones and 3% of the references found in the body text. It would be worth pursuing this study in greater depth so as to more accurately assess the notion of “partially annotated”.


Kim, Young-Min, Patrice Bellot, Elodie Faath, and Marin Dacos. 2011. “Automatic Annotation of Bibliographical References in Digital Humanities Books, Articles and Blogs”. In Proceedings of the 4th ACM Workshop on Online Books, Complementary Social Media and Crowdsourcing, 41–48. BooksOnline ’11. New York, NY, USA: ACM. DOI:10.1145/2064058.2064068.

Kim, Young-Min. 2016. “From Digital Library Citation Parsing to Wikipedia Reference Analysis”. Billet. OpenEdition Lab. Consulted on 13 April.

Langlais, Pierre-Carl. 2015. “{{Référence nécessaire}}: L’émergence d’une norme wikipédienne (2003–2009)”. In Wikipédia, objet scientifique non identifié, ed. Lionel Barbe, Louise Merzeau, and Valérie Schafer, 77–90. Sciences humaines et sociales. Nanterre: Presses universitaires de Paris Nanterre. DOI: 10.4000/books.pupo.4106

Sahut, Gilles. 2014. “’Citez vos sources’ : archéologie d’une règle au cœur du savoir wikipédien (2002-2008)”. Études de communication, no. 42 (July): 97–110. DOI: 10.4000/edc.5721


  1. []
  2. []
  3. frwiki-20161120-pages-meta-current.xml (27G and  434 280 888 lines []
  4. []
  5. []
  6. []
  7. []
  8. []
  9. []
  10. []
  11. []
  12. []

Vous aimerez aussi...

1 réponse

  1. 22/05/2017

    […] Mathieu Orban de Xivry, Élodie Faath and Marin Dacos, « Wikipedia, bibliographies and reference annotation », Carnet de recherche OpenEdition Lab,  22/05/2017. URL: […]

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *