Annotation

(Source: F-LOB manual, original version)

The purpose of any coding system is to produce an ASCII text that maintains as much of the information of the original text. Instead of using the rather complicated coding system of the LOB corpus, we used a simplified version of SGML-based mark-up codes that were drawn up for the coding of the International Corpus of English (ICE) subcorpora (see Nelson 1996 for details). For example, all types of typeface change, like underlined, bold, italics, etc. were subsumed under one general typeface-change code. The mark-up codes are enclosed in angular brackets. They consist of an opening tag (e.g. <quote_>) and a closing tag (e.g. <quote/>). If a feature extends only over one word a vertical stroke is used (e.g. <quote|>). A list of all the mark-up codes used in the Frown corpus is included in the manual.

In addition to those mark-up tags that help to represent the microstructure of the original (i.e. those indicating a typeface change or the beginning of a new paragraph), the ICE mark-up includes codes that help to interpret rather than represent the original text (i.e. the marking of non-English text or transliterations of Greek or Hebrew text).

In order to ensure that the corpus text would be as ‘readable’ as possible, the use of mark-up symbols was kept to a minimum. In particular, we tried to avoid the use of double codes. If a non-English word in the original text was set in italics it was only coded as non-English (<foreign_>word<foreign/>) and not as (<tf_><foreign_>word<foreign/><tf/>).

The main stages in the production of the POS-tagged versions of the Brown corpora were

1. Conversion of corpus markup
2. Tokenization
3. Initial tag assignment
4. Tag selection (disambiguation)
5. Idiomtagging
6. Template Tagger (1)
7. Template Tagger (2)
8. Post-editing

These stages of tagging and post-editing are described in more detail in the manual for the POS-tagged Brown corpora.

The table below summarizes the stages in the development of the four core Brown corpora.

References

Nelson, Gerald. 1996. "Markup Systems." In Greenbaum, Sidney, ed. Comparing English Worldwide - The International Corpus of English. Oxford: Clarendon. pp. 36-53.