Conversion of the CEEC-400 into XML

A Manual to Accompany the XML Edition

2020

Lassi Saario
Research Unit for Variation, Contacts and Change in English (VARIENG)
Faculty of Arts
University of Helsinki

1. Starting point
   1. Family tree of CEEC-400
   2. Why the mess?
2. Pre-processing
   1. File division
   2. Ordering of letters
   3. Character encoding
   4. Parameter coding
   5. Text-level coding
   6. Custom codes
   7. Reversion of normalisation
3. XML conversion
   1. Structure of the corpus
   2. Parameter coding
   3. Text-level coding
      1. Textual structure
      2. Special characters
      3. Page numbers
      4. Headings
      5. Emendations
      6. Comments
      7. Type changes
      8. Foreign language
      9. Superscripts
4. Post-processing
5. Endpoint
6. References

CEEC-400 is a cover term for a family of corpora: the original Corpus of Early English Correspondence (CEEC), the CEEC Extension (CEECE), the CEEC Supplement (CEECSU) and their various versions. Together they cover a time span of almost 400 years from 1402 to 1800. The corpora are based on published editions of letters, which were sampled and digitised by a team of compilers at the University of Helsinki.

The original CEEC-400 was written in a custom version of the ancient COCOA (Word COunt and COncordance on Atlas) format, based on that of the Helsinki Corpus of English Texts. Different collections were often encoded by different people, mostly research assistants, interpreting instructions written by someone else. As there was no validator to check their output, the codes were exposed to errors and inconsistencies, the amount of which multiplied with each version of the corpus.

In the autumn of 2018, we were granted funding by the Faculty of Arts to convert CEEC-400 into XML so that it could be imported into more modern platforms (such as CQPweb). We soon found out that the shortcomings of the original coding would have to be detected and corrected before the conversion could take place. This is a documentation of the correction and conversion project, carried out by Lassi Saario under the supervision of Tanja Säily and Samuli Kaislaniemi.

The documentation is arranged in chronological order. The chapters 1–2 on the background of the project and pre-processing are mostly intended for internal use within VARIENG, whereas the one about the XML conversion might be more of interest to the general public.

Starting point

Family tree of CEEC-400

The CEEC corpora have a large family tree, the branches of which had grown quite far apart from each other by the time the conversion was about to start. Here’s a brief inventory of all the different corpus versions there were at that point.

First, there were the three separate basic corpora, CEEC, CEECE and CEECSU, stored on our network drive (the ‘P drive’). Each letter collection was stored in its own text file, preceded by some metadata about the collection and followed by the letters, which in these versions were lacking letter IDs or ‘L-lines’ such as <L ALLEN_001>, where ALLEN_001 is the identifier of the first letter in the Allen collection.

Second, there were two published subsets of CEEC: the CEEC Sampler (CEECS) and the Parsed CEEC (PCEEC), the latter of which was provided in three formats: plain text, tagged and parsed. The PCEEC was also stored on the P drive and had L-lines for all the letters in it, so that the letters could be unambiguously referenced in the separate metadata file.

Third, there were the normalised versions of CEEC, CEECE and CEECSU, known by the cover term SCEEC (short for Standardised-spelling Corpora of Early English Correspondence). We will call them CEEC-norm, CEECE-norm and CEECSU-norm. Unlike their non-normalised counterparts, the normalised collections did include L-lines; however, they only included a subset of the non-normalised collections. They were stored on the P drive in both tagged and untagged formats: the tagged versions included the original variants inside XML-like tags, whereas the untagged ones had no such tags.

Fourth, there was the brand new POS-Tagged CEECE (TCEECE) that we had just finished prior to this project (see the TCEECE manual). TCEECE was based on CEECE-norm that had been further normalised, converted into XML and annotated by a part-of-speech tagger.

Finally, the basic corpora CEEC, CEECE and CEECSU as well as PCEEC had also been imported into CEECer, a web-based search engine with associated metadata about the letters and their writers. As opposed to the P drive versions, all the letters in CEECer had L-lines, but the header sections that preceded each collection on P drive had been lost in the importing process.

All in all, there were various versions of the CEEC corpora that were more or less related, but not in a straightforward way. Some changes had been made to some versions that had not been synced with the parallel versions. The same letter might appear in a slightly different form and even bear a different identifier in different corpus versions. Not even the collections could be consistently individuated across the different versions. There were also many deviations from the expected COCOA coding here and there. This mess had to be sorted out before the XML conversion could begin.

Why the mess?

The history of the L-line and the letter ID plays a crucial role in understanding the genesis of unsynced corpus versions. The original P drive versions of CEEC, CEECE and CEECSU did not involve any L-lines at all. The L-line was first introduced in the PCEEC, which for copyright reasons did not include all of the letters in the CEEC. When CEEC, CEECE, CEECSU and PCEEC were imported into CEECer, L-lines were added to all letters, which resulted in some ‘half’ IDs (e.g. BROWNE_043.5) and other peculiarities. The normalised versions were based on the P drive versions, however, so the L-lines were added again to the normalised versions, which resulted in more errors.

Another source of errors is the person ID that appears in the ‘Q-line’ (such as <Q A 1497? T RFOX>, where RFOX is the person ID of Richard Fox, the author of the letter). When the corpora were imported into CEECer, it was discovered that different collections (particularly those in different subcorpora) might employ the same person ID for different persons, as the person ID had been derived directly from the person’s first and last name. These IDs were corrected in the CEECer metadata but not in the P drive corpora.

Pre-processing

We decided to merge the P drive and CEECer versions of CEEC, CEECE and CEECSU and the tagged versions of CEEC-norm, CEECE-norm and CEECSU-norm into one master corpus that would be kept in a Git repository where the version history could be tracked and controlled automatically. We wanted to rid the corpus of all known errata and ensure that the letter identifiers be consistent across those parallel corpora that remained. It would be the new master corpus that would then be converted into XML. What follows is a reconstruction of the pre-conversion process.

In the version history of our GitLab project, the steps were actually taken in a different order than what is presented here. Now that the process is complete, it is easy to see that the actual order was far from optimal. That is why we present here an alternative version history where the changes are made in a more logical order that is easier to follow and understand. The end product is, nevertheless, exactly the same master corpus as in the actual Git repository.

File division

We began by aligning the basic collections with their normalised versions and determining for each collection the file that would be included in the new master corpus. Whenever a collection had both a non-normalised and a (tagged) normalised version, we preferred the normalised one, as the non-normalised version could be reverted from it automatically, and the normalised one also contained L-lines which the non-normalised one did not. For each row in the table below, if the ‘final file’ column is empty, it means the final file is the normalised one—unless that is empty too, in which case the final file is the non-normalised one. The collections without a normalised file are mostly from the 15th century, which represents Late Middle English and has so far been deemed too challenging to normalise.

On the non-normalised side, the biggest collections had once been split into smaller files because of a restriction on the file size imposed by the WordCruncher application. The respective normalised collections had been merged into one file (with one exception). Even those non-normalised collections that did not have normalised counterparts were now merged into one file, so that there would be only one file for each collection.

Ordering of letters

We spotted two collections where the ordering of letters did not correspond to PCEEC and CEECer. In the other collection, it was not only letter order but also letter IDs that contradicted.

Character encoding

The encoding was unified to UTF-8 throughout the corpus.

The ‘old line’ refers to the line number as it was in the corpus after the changes to file division and letter order (see the two previous sections), and the ‘new line’ refers to the line number as it is in the final version of the corrected corpus.

Parameter coding

The standard COCOA parameters are documented in e.g. the CEECS manual and also below. Deviations from the standard had to be corrected so that the coding could be automatically converted into XML. At this stage, we also made the necessary corrections to the IDs of letters, persons and collections.

In addition to the changes above, L-lines were added to those collections that lacked them.

Text-level coding

The standard COCOA text-level codes are documented in e.g. the CEECS manual and also below. Deviations from the standard had to be corrected so that the coding could be automatically converted into XML. There were also some code instances that were not incorrect as such but still had to be changed to meet the stricter requirements of XML.

Custom codes

Unlike other collections in CEEC-400, the Bacon and Willoughby collections in the CEEC involved custom codes that had not been converted into COCOA text-level coding:

The custom codes were converted into COCOA codes. The conversion was carried out partly automatically and partly manually. We omit the complete conversion tables and only give a few examples so the reader will get the idea:

Reversion of normalisation

The pre-conversion process resulted in a new hybrid corpus of normalised and non-normalised collections. The normalised collections were yet to be reverted to their non-normalised versions before the corpus would be converted into XML. The reversion was performed by ‘VardStripper’, a Java application written by Lassi Saario. In what follows, the ‘original CEEC-400’ will refer to the corpus as it was at this point, after the reversion and before the conversion.

XML conversion

The XML schema of CEEC-400 has long roots. Before we got funding for the XML conversion of the entire CEEC-400, we had already converted a version of the CEECE-norm as part of the POS tagging project that resulted in the TCEECE. The XML schema of CEEC-400 was based on that of the TCEECE, which had in turn been based on that of the Helsinki Corpus, which had again been based on the TEI standard.

The conversion from COCOA into XML was performed by our own ‘XmlConverter’, a Java application written by Lassi Saario. The resulting XML files were validated against the Document Type Definition by XmlStarlet (version 1.6.1), a command line XML toolkit developed by Mikhail Grushinskiy.

Structure of the corpus

The original corpus was divided into text files, one for each letter collection. We decided to preserve this division. Each COCOA-encoded collection file was converted into an XML-encoded file of the same name.

In the original corpus, each collection was preceded by a header followed by the individual letters. Each letter was likewise preceded by a header followed by the contents. The XML version follows the same overall structure, illustrated below.

Each XML document begins with the same two lines. The first line specifies the XML version and the character encoding. The second line defines the document type by a reference to an external DTD file. The entities are given an internal declaration as well, since omitting it would cause errors on some browsers which do not support external DTDs.

Each document has teiCollection as its root element. It is made up of a teiHeader element, containing header information about the collection, and a series of TEI elements, representing the individual letters. Each TEI element is likewise made up of a teiHeader element which includes header information about the letter, and a text element which includes the actual contents of the letter.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE teiCollection SYSTEM "../CEEC.dtd" [
	<!ENTITY ETH "&#208;">
	<!ENTITY eth "&#240;">
	<!ENTITY YOGH "&#540;">
	<!ENTITY yogh "&#541;">
	<!ENTITY THORN "&#222;">
	<!ENTITY thorn "&#254;">
	<!ENTITY pound "&#163;">
]>
<teiCollection xml:id="FFOX">
	<teiHeader>...</teiHeader>
	<TEI xml:id="FOX_001">
		<teiHeader>...</teiHeader>
		<text type="letter" xml:lang="eng">...</text>
	</TEI>
	<TEI xml:id="FOX_002">
		<teiHeader>...</teiHeader>
		<text type="letter" xml:lang="eng">...</text>
	</TEI>
	...
</teiCollection>

Parameter coding

In the original CEEC-400, each file is preceded by the identifier of the collection (same as the file name), followed by source information:

<B FFOX>

[^LETTERS OF RICHARD FOX 1486-1527. EDITED BY P. S. AND H. M.
ALLEN. OXFORD: CLARENDON PRESS. 1929.^]

In the XML conversion, the identifier is put in the xml:id attribute of the teiCollection opening tag, and the source information is included in a titleStmt element in the teiHeader:

<teiCollection xml:id="FFOX">
	<teiHeader>
		<fileDesc>
			<titleStmt>LETTERS OF RICHARD FOX 1486-1527. EDITED BY P. S. AND H. M. ALLEN. OXFORD: CLARENDON PRESS. 1929.</titleStmt>
		</fileDesc>
	</teiHeader>

A letter header in the original CEEC-400 consists of an L-line, a Q-line, an X-line and a P-line:

<L FOX_001>
<Q A 1497? T RFOX>
<X RICHARD FOX>
<P 17>

• The L-line gives the letter identifier.
• The Q-line specifies the authenticity of the letter, the year of writing, the relationship between the writer of the letter and the addressee, and the identifier of the writer, respectively.
• The X-line contains the name of the writer in full. (In some collections there is an A-line instead of an X-line, but the content is the same nevertheless.)
• The P-line includes the number of the page on which the letter begins in the source edition. Similar lines appear amidst the body whenever the page changes.

The contents of the lines are included in the XML header as follows:

<TEI xml:id="FOX_001">
<!-- from the L-line -->
	<teiHeader>
		<fileDesc>
			<titleStmt>
				<title key="A 1497? T RFOX"></title>
				<!-- from the Q-line -->
				<author key="RICHARD FOX"></author>
				<!-- from the X- (or A-) line -->
			</titleStmt>
		</fileDesc>
	</teiHeader>

The P-line is included in the XML body along with the other P-lines. See the section on page numbers.

The S-lines like <S SAMPLE 1> that sometimes occur between letters mark samples taken from different source editions in the original corpus. They have been converted into XML comments like <!-- SAMPLE 1 -->.

Text-level coding

Textual structure

A letter body in the original CEEC-400 is divided into lines, the maximum length of which is limited to 65 characters. Some of them are P-lines that annotate page breaks; for the rest there is no fixed format. Paragraphs and sentences flow rather freely from one line to another along with code brackets for headings, emendations etc. See the example below.

[} [\705 FROM MARY HOWE TO MR RANKING IN COOPERSALE (THEYDON 
GARNON), 13 JANUARY 1731\] }]
Jenaw 13 day 1731
Mr ranking this is to let you know that the doxtor have done 
what he can for me but my iees are never the better but rather
wors i ame to be discharge=d= next wandsday i 
hope you will be so kind as to send me word how i must come home
by next wandsday morning so with humble sarvis to you and your 
good wife
   sir I hope you will exquese me in wrighting of a letter but i
did not know no other way So i rest your humble sarvant
   mary how patient in
[\CONTINUED CROSSWISE IN LEFT-HAND MARGIN\] peter ward

Unfortunately, the line and page divisions that are so explicit in the original CEEC-400 are irrelevant for the purposes of linguistic research, especially as they do not reflect those of the original manuscript. Much more relevant is the paragraph division, which is also much more implicit. Lines that start paragraphs are usually indented with three whitespaces, but not always: sometimes the only clue of the line starting a paragraph is the previous line being shorter than usual. Matters are further complicated by the fact that P-lines sometimes appear in the middle of a paragraph and sometimes between paragraphs. Code brackets often appear in the middle of a paragraph, but sometimes they continue across paragraph breaks, and sometimes they even seem to form paragraphs of their own.

To recognise such delicate divisions may be easy for a human eye, but it is far from easy for a computer. We wanted to try it anyway. The rule of thumb that we gave to our converter is that a line starts a new paragraph if it is indented or if the previous line is shorter than 40 characters. Lines that were recognised to form a paragraph were then merged and, given that the paragraph in question was indeed a proper paragraph (as opposed to a single P-line or a bunch of code), put inside a p element. Code bracket sequences that continued across paragraph or page breaks were split at the breaking point in order to guarantee the sanity of the element tree.

Special characters

Grave accent symbols that annotated accents (not only grave but acute ones and circumflexes as well) and tildes that annotated abbreviations in the original CEEC-400 remain in the XML edition. Certain special characters have been converted into XML entities according to the following table.

Page numbers

Page changes were annotated as P-lines, e.g. <P 45>, in the original CEEC-400. They are converted into pb elements, the n attribute of which contains the page number, e.g. <pb n="45">. Note that pb elements may appear inside as well as outside p elements.

Headings

Headings, annotated with the code [}...}] in the original CEEC-400, are annotated with the code <head>...</head> in the XML edition.

Note that most headings have been added by either editors or compilers, i.e. they are double-coded such as

[} [\98. TO FANNY BURNEY\] }]

where the inner brackets stand for the editor’s or compiler’s remark. The double-coding is preserved in the XML conversion so that the given example is converted into

<head> <note resp="editor" value="98. TO FANNY BURNEY" /> </head>

See the section on comments for more information.

Emendations

Emendations are annotated with the code [{...{] in the original CEEC-400. These have been converted into supplied elements in the XML version. When the emendation consists of complete words, we simply put the content of the brackets in between the XML tags. In e.g. the following passage,

I turned so sick that I [{could{] hardly speak

the [{could{] is converted into

<supplied>could</supplied>

The case is a bit trickier when the emendation contains partial words, as in w[{ife and son{]. These kinds of emendations we have converted so that in between the XML tags there is the final amended expression, while the original code is included in an orig attribute:

<supplied range="1,10" orig="w[{ife and son{]">wife and son</supplied>

The ‘range’ attribute specifies the extension of the emendation. The expression in between the XML tags is indexed so that the first character has the index 0, the second has the index 1 etc. The first number in the range value is the index of the first character in the range, and the second number is the index of the first character not in the range. Note that whitespaces do not count as characters here. When there are several ranges in the same expression, they are delimited by a semicolon:

<supplied range="7,13;15,16" orig="felysch[{yp of Ho{]ll[{a{]ndars">felyschyp of Hollandars</supplied>

Comments

The original version of CEEC-400 contains two types of comments added to the body text. Comments by compilers of the corpus are annotated with the code [^...^], e.g. [^LIST OF NAMES OMITTED^]. Comments by editors of source editions are annotated with the code [\...\], e.g. [\TORN\].

In the TEI XML edition of the Helsinki Corpus, both codes are converted into a note element. The author of the comment is specified by a resp attribute which points to his/her name in the document header. For our purposes, however, it is sufficient to separate the editors’ comments from the compilers’ and not specify the individual commentator. The attribute is simply given the value compiler for compilers’ comments and editor for editors’ comments.

Comments, whether they are written by editors or compilers, are actually used for two different purposes. One is a ‘proper’ comment, such as the two previous examples. The other is more like an emendation, as in

an order [\was made\] at his Lordship's instance

The difference between the two kinds is that a proper comment is a comment about the surrounding text, whereas an emendation-like comment is more like a part of the text. They are rather easily distinguished by the fact that a proper comment usually involves several consecutive upper case letters while an emendation-like comment does not. This holds true even when a proper comment contains text that is parallel to the preceding text, as in

it will [\be DELETED\] come safe hither

When the brackets contain a proper comment, we put their contents in an attribute inside an XML tag:

<note resp="compiler" value="LIST OF NAMES OMITTED" />
<note resp="editor" value="be DELETED" />

When the brackets are used to annotate emendations, we follow the same principle as with the [{...{] code explained above. Emendations of complete words are encoded as

<note resp="editor">was made</note>

whereas emendations that contain partial words are encoded as

<note resp="editor" range="3,7" orig="Jan[\uary\]">January</note>

Type changes

Changes of typeface in the printed source editions were annotated as (^...^) in the original corpus. In the XML edition, they are annotated as <hi rend="type">...</hi>. (This usually corresponds to an underlined passage in the original letter.)

When the change of typeface concerns partial words as in Theo(^log^), the original coding is preserved in the orig attribute of the hi element as in

<hi rend="type" range="4,7" orig="Theo(^log^)">Theolog</hi>

Foreign language

Passages in foreign language were annotated with the code (\...\) in the original CEEC-400. In the XML edition, they are annotated with the code <foreign>...</foreign>.

Superscripts

=vi=
w=ch=
p=r=ferm=t=

In the XML corpus, they are annotated as

<hi rend="sup">vi</hi>
<hi rend="sup" range="1,3" orig="w=ch=">wch</hi>
<hi rend="sup" range="1,2;6,7" orig="p=r=ferm=t=">prfermt</hi>

Post-processing

There were two abbreviations in the Dodsley collection where the superscript code had been used to encode a superscript inside another superscript. These instances had to be hard coded into the converter in order for them to be converted correctly, or else the converter would have taken them to consist of two consecutive superscripts:

Endpoint

At the time of writing this manual, we are currently adapting the new XML corpus for our own CQPweb server in collaboration with Lancaster University. Once the remaining (15th-century) collections have been normalised, we will consider the possibility of converting the normalised CEEC-400 into an even more comprehensive XML version that would provide access to both the original and normalised variants.

References

For more information on the CEEC corpora, see the front page.