How fluent are advanced German learners of English (perceived to be)? Corpus findings vs. native-speaker perception [1] [2]

Sandra Götz
Justus Liebig University, Giessen

Abstract

1. Introduction

Two of the central descriptors of oral performance in the teaching and assessment of foreign languages are accuracy and fluency (see for example the Common European Framework of Reference for Languages, Council of Europe 2001). While both concepts reflect rather “complex and fuzzy notions that involve numerous factors and that could be seen to (at least) partially overlap” (Gilquin & De Cock 2011: 145), the analysis of annotated learner corpora have allowed for more systematic analyses of these two features. The mainstream linguistic approach to quantifying accuracy in learner language is to assess the number, type or proportion of “errors”, i.e. deviations from a given native target norm (cf. Ringbom 1987) (cf. Section 2). Likewise, there are various approaches towards the description and assessment of fluency. Here, linguists mainly focus on fluency in the “narrow sense” (cf. Lennon 1990) and analyze temporal variables of speech production. This focus on speed in speech production is generally accepted as being the best indicator of a learner’s fluency (cf. e.g. Lennon 1990; Riggenbach 1991; Chambers 1997; Gut 2009). Recent studies have proven one’s natural supposition that temporal variables as well as the number/type of errors correlate with native speaker assessments of a learner’s overall oral proficiency (e.g. Lennon 1990; Riggenbach 1991; Cucchiarini et al. 2002). However, these studies focused on a low number of raters (a max. of 12 raters in Riggenbach 1991) and did not take into consideration other variables that might be equally – or maybe even more – important for establishing a perception of oral proficiency in advanced learners’ speech on the part of the listeners, such as accent, idiomaticity, lexical diversity, register, sentence structure, intonation, or pragmatic features.

In the present paper, after taking a brief look at the theoretical background of the concepts of accuracy and fluency in spoken learner language, I will investigate some of these other variables that might be equally relevant for the perception of a learners’ degree of overall oral proficiency. I will then report on a study that takes into consideration a combination of these approaches: First, I will present the findings of a previous quantitative analysis of the error-tagged version of the c. 87,000-word German component of the Louvain International Database of Spoken English Interlanguage (LINDSEI-GE) as compared to the Louvain Corpus of Native English Conversation (LOCNEC) (cf. Brand & Götz 2011). Secondly, based on the quantitative findings of this corpus analysis, five learners that represent certain prototypical accuracy and fluency learner types are rated by fifty native speakers with regard to (1) their overall oral proficiency, and (2) six other perceptive fluency variables, in order to investigate possible correlations between the overall ratings and any of the additional variables. I will conclude this paper with a discussion of some of the limitations of the present study and of potential follow-up studies for future research.

2. Central descriptors of overall proficiency in spoken learner language

2.1 Accuracy and temporal fluency

Accuracy is one of the central descriptors of oral proficiency in foreign language teaching, but it is not a simple concept. Accuracy can be quantified by the number, type and proportion of errors a speaker commits, where an error can be very broadly defined as a deviation from a given language norm (cf. Ringbom 1987). While it seems difficult enough to establish norms for accuracy in written language due to the great deal of variation in English lexis and grammar (cf. Gilquin and De Cock 2011), it is even more challenging in the case of spontaneous spoken language, where additional cognitive planning pressure might have a negative impact on the accuracy of a speaker’s output and increase the number of performance-based errors. Corder (1974) therefore suggests the distinction between mistakes, i.e. slips of the tongue or flaws in performance and errors, i.e. failures in competence and thus systemic faults. For a conceptual differentiation of accuracy in native vs. learner language, Gilquin and De Cock (2011: 142) suggest that mistakes are shared by native and nonnative speakers alike, while only learner language is characterized by true errors, which “result from incomplete knowledge of the language and normally display some systematicity”. Also, it is crucial to take into consideration major differences in listeners’ perception of accuracy in native and nonnative speech: In conversations between native speakers, speakers and listeners tend not to be very concerned about the formal correctness of the output, but focus rather on the content of the utterance (cf. Chafe 1980). Therefore, in native speech, instances of grammatical or lexical ill-formedness usually go unnoticed by listeners or at the worst are perceived to be slips of the tongue rather than errors. Previous studies (e.g. Lennon 1990; Riggenbach 1991; Herbst 1992; Cucchiarini et al. 2002) have shown that native listeners attach greater importance to accuracy in their perception of the overall proficiency of recognizably nonnative speakers than of native speakers. Therefore, any ill-formed items in learner language – including performance mistakes – seem to have a much more negative impact on overall proficiency ratings in nonnative speech. As there are most certainly some types of errors that are perceived more negatively than others, future large-scale empirical studies will have to investigate and quantify such differences. Even without these fully developed distinctions, I consider the perception of errors to be important as a variable and have included it in this study. Since the annotated learner corpus that was used in the present study distinguishes only between different types of functional error categories without added values for possible differences in their perception, accuracy in learner language was measured simply by calculating the number of errors in the speakers’ output (including performance mistakes) without further distinctions. [3]

Temporal fluency refers to fluency in the “narrow sense” (Lennon 1990), i.e. it focuses on automaticity and speed in speech production and is therefore “measurable in a series of quantifiable phenomena” (Gut 2009: 79), such as the number and position of filled and unfilled pauses, speech rate, the mean length of runs, the phonation/time ratio, etc. (e.g. Lennon 1990; Riggenbach 1991; Chambers 1997; Gut 2009). Similar to errors, previous studies have shown that dysfluencies, too, are hardly noticed in native speech (cf. Gilquin & De Cock 2011). Again, in learner language, native-speaker ratings of overall language proficiency also correlated with the learners’ performances according to these temporal fluency variables (e.g. Lennon 1990; Riggenbach 1991; Chambers 1997; Gut 2009). Therefore, they will also be taken into account in the present paper. Methodologically, as a recent study by Osborne (2011) has shown that learners’ performance in different temporal fluency variables clearly influence each other, I will follow his suggestion and merge them into one value, i.e. an overall temporal fluency score (see Section 4).

2.2 Additional variables of perceptive fluency

The major question the present paper addresses is to test whether other factors, apart from temporal fluency and accuracy, have an influence on the perception of an overall oral proficiency. To this end, I will consider fluency as a holistic phenomenon broadly synonymous with language mastery and native-like performance on all levels, which is in line with Lennon’s (1990) concept of fluency in the “broad sense”. Therefore, it might be worth investigating whether some additional variables also contribute to the listeners’ perception of learners’ fluency.

Owing to the fact that learner language is often characterized by usages that are not grammatically wrong, but ‘sound odd’ to a native speaker’s ear, one additional variable to be included is idiomaticity. Acquiring a nativelike level of idiomaticity is very challenging even for advanced learners, because there are no clear-cut rules of what is idiomatic and what is not, as it is largely usage-based and culture-bound. The difficulty of this variable for learners is pointed out by Erman, who exemplifies this by referring to collocations: “a non-native speaker may be able to decode but unable to encode a collocation, and even if the non-native speaker encodes it correctly she may not know that it is conventional” (Erman 2007: 33). Additionally, idiomaticity can be established on various other linguistic levels, for example on the lexical, lexico-grammatical or syntactic level (cf. Fernando 1996). Idiomaticity has also been equated with a nativelike use of formulaic sequences (cf. Pawley and Syder 1983). In this context, Pawley and Syder (1983: 194) suggest that learners need to “learn a means for knowing which of the well-formed sentences are nativelike – a way of distinguishing those usages that are normal or unmarked from those that are unnatural or highly marked” if they want to dissolve the “puzzle of nativelike selection”. It seems quite difficult to measure the degree of idiomaticity quantitatively in learner output, although some methodological suggestions have been made for some features of idiomaticity. These include, for example, Nesselhauf’s (2005) study on the use of collocations, or Wulff’s (2008) study on the use of “Verb Noun Phrase”-constructions. In order to grasp a holistic impression of idiomaticity in learner language, it appears to be a very appropriate variable for the native-speaker rating.

Another related variable to consider in spoken learner English is the learners’ ability to know and use the appropriate register for the communicative situation. While the use of complex syntactic structures in academic discourse may contribute to a high level of perceptive fluency, it would appear inappropriate in an informal conversation and would render the learners’ speech too formal and non-nativelike. It has often been claimed that learners, even at an advanced level of proficiency, mix registers and lack a nativelike “text-type sensitivity” (Lorenz 1999: 64). They may be considered as “too chatty” (cf. Gilquin & Paquot 2008) in their writing and too “bookish” and “pedantic” (cf. Cutting 2006) in their speech. The characteristics of different spoken genres have been described thoroughly by different researchers (among others McKelvie 1998; Biber et al. 1999; Thornbury & Slade 2006). The greatest challenge for foreign language teaching (FLT) is to provide learners with a large enough stock of different structures they can use in different modes and registers along with a good sense of a “critical register awareness” (Gilquin & Paquot 2008), in order to learn how to use the most nativelike structure for the individual communicative situation they may encounter.

Another variable which has not gained much attention in spoken learner language analysis is sentence structure. While a large-scale study by Borin and Prütz (1996) has shown that the written sentence structure produced by learners of English with various L1s is – unsurprisingly – significantly influenced by interference from their L1, this has not yet been investigated for spoken output. While L1-interference will most probably also hold true for spoken language, more importantly, learners have also been shown to have little syntactic variation in their output, which manifests itself mostly through their avoidance of complex structures and a high proportion of “facilitation strategies” (Bygate 1987). Although this variable is probably not noticed as quickly as their accent or their level of accuracy, the learner’s overall performance may certainly affect the fluency perception of the listener.

Additionally, non-native speech is very frequently characterized by a foreign accent and if this accent is very ‘strong’, a native speaker of the target language would even be able to pinpoint the L1 of the non-native speaker in no time, because it is, naturally, mostly shaped by L1 interference. In a very broad sense, accent refers to “deviations in pronunciation of non-native speech compared to the norms of native speech” (Gut 2009: 253). Also, the influence of intonation and accent on the perception of fluency account for areas that are “very under-researched at the moment” (Chambers 1997: 540) and should thus attract more attention. One exception is Herbst (1992), who conducted a study that showed a tendency for native speakers’ perception of learners’ overall performances as well as their perception of other communicative variables to be highly dependent on the learners’ accents.

In a similar vein, intonation also “contributes to making you sound foreign, and may quite possibly lead to your being misunderstood by other speakers” (Wells 2006: 2f.), so it might well have an impact on a native speaker’s perception of the overall oral proficiency of a learner. Although there have been made some valuable suggestions how to implement the teaching of intonation into the FLT-context at schools or at university, clear-cut and fine-grained teaching models of nativelike intonation patterns have not yet been implemented in the German curricula (cf. Götz 1986; Esser 1992: 23ff., Chapman 2007). Whether or not a learner acquires even the basic intonational features of controlled speech therefore seems to a certain extent arbitrary. Because of the fact that they form an important feature of language use, as they are not only characteristic of the naturalness of the sound and rhythmicity of a language, but they can also carry clear communicative functions. Hence, they may lead to a perceived foreign-soundingness or loss in the perception of overall proficiency if they are not used adequately.

The last variable I would like to include in this study is pragmatic features because

in terms of language learning, the area of pragmatics is perhaps one of the most difficult areas in that one is generally unaware of this aspect of language and may be equally unaware of the negative perceptions native speakers have of them. (Gass & Selinker 1994: 184)

The difficulty is caused again by the lack of explicit pragmatic rules, so learners mostly transfer the pragmatic conventions of their mother tongue, which may be inappropriate or not nativelike. Studies have shown that the speech of even highly advanced learners of English displays a great amount of inappropriate language use, especially when it comes to pragmalinguistic features, such as an inappropriate realization of a speech act, e.g. “the non-native speaker intends a request, but because of inappropriate directness or modification, the native speaker interprets it as a command” (Cutting 2008: 67). This inappropriate use may lead to pragmatic failure and, as a consequence, to a decrease in the overall perception of fluency (cf. House 1996: 227f.).

3. Research question, database and methodology

The key question of this study is whether native speakers’ perception of the overall oral proficiency of advanced German learners of English is dependent on any of the variables mentioned above (see Section 2), or, in other words, whether any of these variables is significantly correlated with native speakers’ perception of the learners’ overall proficiency.

The database for this study is the error-tagged version of the German component of the Louvain International Database of Spoken English Interlanguage (LINDSEI-GE). [4] LINDSEI consists of several national subcorpora containing data gathered from learners with different mother-tongue backgrounds such as Bulgarian, French, German, Italian, Japanese, Swedish and Spanish and is the first large-scale multinational corpus of advanced spoken learner English (cf. Gilquin et al. 2010). There is also a comparable corpus of native speakers of English, the Louvain Corpus of Native English Conversation, LOCNEC (cf. De Cock 2003), which provides a basis for the comparison of interlanguage and native language. The German component of LINDSEI (LINDSEI-GE) was compiled and transcribed at the University of Giessen (cf. Brand & Kämmerer 2006) and includes, like the other components of the corpus, fifty interviews conducted in well-defined settings to guarantee the collection of comparable data, e.g. duration of the interviews, restricted set of topics and comparable proficiency levels. In the second phase of the project, the German component was tagged for errors (cf. Kämmerer 2009a) according to a refined version of the Error Tagging Manual 1.2 (Dagneaux et al. 2005).

I took three steps in the present analysis. First, according to the findings gathered from a previous study on fluency and accuracy (cf. Brand & Götz 2011), I investigated five learners that represent specific learner types that are represented in LINDSEI-GE. In a second step, these five learners were rated by native speakers with regard to their perception of the learners’ overall proficiency and the six perceptive variables mentioned above (see Section 2). The third step was then to calculate the correlations of the overall ratings with each of the variables in order to find out if any of these had a significant impact on the overall ratings.

4. Findings

4.1 Selection of the five learner reference types

Since the native-speaker ratings of the perceptive fluency variables of all fifty learners in LINDSEI-GE would not have been possible, I chose five learners that represented specific accuracy and fluency profiles and thus seemed the most promising candidates for finding possible correlations according to the data analysis. I adapted the findings from a previous study (cf. Brand & Götz 2011) and chose

1. GE024, the least accurate one, i.e. the learner with the highest number of errors in LINDSEI-GE (with 3.38 errors phw and 58 errors in total),
2. GE027, the most accurate one, i.e. the one with the lowest number of errors (0.4 errors phw / 6 errors in total),
3. GE001, the learner with the most fluent performance across three temporal variables, viz. speech rate and number of filled and unfilled pauses,
4. GE041, a learner with a comparatively low temporal fluency and who spoke – from my point of view – with a strong German accent and intonation,
5. GE028, with an average error rate and temporal fluency performance, who is supposed to serve as a control learner.

In order to calculate the correlations between the overall communicative competence of the learners and the different variables, it is most helpful to have only one value for each variable. While for accuracy this value can easily be set as the number of errors per one hundred words (phw), for temporal fluency several temporal fluency variables need to be combined in order to obtain one overall temporal fluency score for each learner (cf. also Brand & Götz 2011). Before this could be done, each learner’s performance in each of the variables was set in relation to the native-speaker mean. That is, for each of the variables the NS mean was set as equal to 100% and the learners’ performances were each converted into a percentage of the NS mean by calculating learner performance (LP) divided by NS mean multiplied by 100 (LP / NS mean *100). This calculation allows for the following: If learners perform better than the NS mean for a particular variable, they are able to exceed 100% for this variable. I did this in order to take into account the fact that the learners may not be equally adept in the use of different kinds of fluency variables and, as a result, may use one variable much more frequently than another to establish their spoken fluency. In other words, the learners may achieve a very poor performance in one fluency variable and only score, say, 10% of the NS mean, but give an extremely good performance in other variables where they may score, say, 140% and 120% respectively, and thus may establish their overall fluency performance through different means and arrive at a total fluency score of 90% altogether.

Also, it needs to be taken into consideration that while for some variables a high value is considered to be more fluent (e.g. the higher the speech rate, the more fluent), for other variables a low value means a high degree of fluency (e.g. the fewer unfilled pauses the more fluent). Therefore, two different formulas are necessary to set the learners’ scores for each variable in the appropriate ratio to the NS mean: (1) If a high value shows a higher degree of fluency (e.g. for speech rate), I calculated learner performance divided by NS mean multiplied by 100 (LP / NS mean * 100) and (2) if a low value stands for a high degree of fluency, I turned it around to NS mean divided by learner performance multiplied by 100 (NS mean / LP * 100).

The overall temporal fluency score was then obtained by combining the means of each variable under scrutiny. The five selected learners’ performances for temporal fluency and accuracy are summarized in Table 1. Note that temporal fluency and accuracy are not correlated.

4.2 Native speaker ratings [5]

In an online survey, fifty native speakers of English were asked to listen to the sound files of the interviews that were carried out with the five selected learners. The majority of 66% of the raters are native speakers of Australian English, 12% are half Australian- half other English native speakers and 20% of the raters are New Zealand, English, American and Scottish English speakers. The raters are staff and Ph.D students from the faculties of Arts and Human Sciences of Macquarie University Sydney, consisting of linguists and non-linguists with a certain degree of ‘language awareness’, so as to grasp the possible perceptive differences of trained vs. untrained raters. After listening to each of the interviews, the raters were asked to judge the overall oral proficiency of the learners from 1 (“sounds like an absolute beginner”) to 10 (“sounds like a native speaker”). Then, they were asked to listen to the sound file again and rate the other six perceptive fluency variables idiomaticity, register, sentence structure, accent, intonation and pragmatic features from 1 (“sounds like an absolute beginner”) to 10 (“sounds like a native speaker”). All these variables were briefly explained in the questionnaire for clarification and to avoid ambiguity. They were able to listen to the sound files as many times as they wished before moving on to the next learner. Also, there was on option in the online survey to go back to the ratings of a learner, listen to the sound file again and change their ratings in case a rater changed their judgement in the course of the survey. For each learner there was an optional text field for comments on certain features and there was a section on feedback on the complete survey at the end of the survey.

The means of these fifty ratings (with a mean inter-rater reliability of 89%, calculated through a mean intraclass correlation of ICC=0.89) are shown in Table 2.

The overall ratings are highlighted (in green) on the left and the fluency score and error rates have been added to the table (in grey) on the right. The rest of the columns display the means of the fifty native speaker ratings for the six perceptive variables idiomaticity, register, sentence structure, intonation, pragmatic features.

GE028 gets the highest overall rating of 7.96, but has an average performance in both her fluency and her accuracy performance. She does not get the highest ratings for any of the investigated perceptive variables, but shows a comparatively good performance across all the investigated variables. GE027 gets the lowest overall ratings (6.80), is the most accurate speaker, but receives the lowest scores for idiomaticity (6.58), accent (6.08) and intonation (6.74). The temporally most fluent learner, GE001, has a medium overall rating of 7.70 and gets the highest ratings for register (8.08) and sentence structure (8.14) as well as for pragmatic features (7.84).

In the light of these diverse findings, it seems particularly interesting to investigate if any of the variables is more influential than the others for the native speakers’ overall perception of overall proficiency of the learners. Therefore, the correlations of the each of the variables with the overall ratings are shown in Table 3.

Of the eight investigated variables, the lowest correlation – and thus the variable that shows the least impact on the overall ratings – is attested for accuracy in errors phw (with r=0.560 p>0.05). Temporal fluency has a higher correlation (r=0.778), but it is not significant, either (p>0.05). Concerning the investigated perceptive fluency variables, only two of the variables have significant correlations, namely accent (r=0.916 and p<0.05) and pragmatic features (r=0.926 and p<0.05), both highlighted in orange. The correlations for the other variables are not statistically significant (for all of them p>0.05).

The findings of the correlations are particularly interesting, because there is a great focus on accuracy as one of the key variables in the teaching and assessment in EFL in Germany, even at an advanced level. However, these findings suggest that, since the learners are already highly advanced (as shown, for example, by the low number of their errors), once a certain level of proficiency has been exceeded, maybe other and more subtle factors and variables of speech become more important to a native speakers’ perception of overall oral proficiency.

As far as pragmatic features are concerned, the significant impact might be due to fact that there is little gradation in performance: That is, there are not too many different options for a nativelike realization of a certain speech act in a communicative situation, which can, of course, increase the probability of the learners to choose the wrong option. This is in contrast to sentence structure, for example, where it may be easier to disguise one’s deficiencies through avoidance of difficult structures (cf. e.g. Liao & Fukuya 2004). The same applies to a speaker’s accent which they cannot hide or avoid and which is, obviously, present at all times while the speaker is talking. It is easily noticeable, and also by default particularly influenced by the learners’ L1 (cf. Section 2) and may therefore be more noticeable than other features and thus have a strong impact.

Accordingly, more focused instruction especially in these two areas, pragmatic features and accent, may be worthwhile and highly beneficial for language practice courses at university level. Suggestions for advanced learners’ improvement of these variables have been recently made among others by, e.g. Yates (2005) for accent reduction or by Cutting (2008) for how pragmatic features may be taught in EFL classes from very early stages onwards.

5. Conclusion and outlook

I would like to conclude this paper by mentioning some caveats to this study and giving an outlook on possible future research.

First, of course, the number of rated learners in this study is low. Although they represent certain learner types as suggested by the selection in LINDSEI-GE, these may have been arbitrary. Nevertheless, it might suggest one possible way of how perceptive fluency can be investigated, although there are likely to be more diverse findings for more rated learners chosen at random. Secondly, one could question the raters’ ability to objectively distinguish between the variables: The questionnaire clearly indicated that different ratings for different variables may be very likely and may also be very likely to differ from their overall ratings. However, the raters may still have tried to ‘equal out’ their overall judgements by rating the perceptive variables according to their overall ratings. Also, since the online survey was anonymous, I did not have the opportunity to verify if all of the raters were native speakers of English (despite the questionnaire’s clear definition of who is considered a native speaker in the context of the study), or if some raters were judging randomly and somewhat superficially.

More importantly, however, especially features like accent or pragmatics seem particularly prone to culture-specific conventions and despite the geographical diversity of the raters, the majority of 66% (N=33) labelled themselves as Australian English native speakers; maybe there would be different results for a majority of American English, British English, Irish English, New Zealand English, etc. raters. It could be worthwhile performing a similar study with an equal percentage of raters of several different native Englishes to see if the ratings differed significantly.

It needs to be pointed out that I have investigated only a small number of variables and there may very well be other variables which have an equal or even a stronger impact on native speakers’ perception of learners’ overall oral proficiency, for example learners’ use of formulaic language or their use of certain performance phenomena that render their speech more natural, like discourse markers. These are features which future studies should take into account.

Another important aspect for future studies might be the different effects of the investigated variables on raters when different proficiency levels come into play. For example, if learners are perceived to be nativelike in their overall proficiency (e.g. because they speak without an accent and perfectly accurately), possible pragmatic errors may be perceived as much more face-threatening than if the same error is committed by a speaker who is clearly a beginner and in whose case certain deviances and errors may therefore be expected by the raters. Such an ‘inter-variable correlations’ test at different proficiency levels may produce rewarding results.

Notes

[1] I am very grateful to the native speakers who took part in the online survey, the CECL-team at Louvain-la-Neuve for providing me with the transcribed version of LOCNEC, Johannes Herrmann for excellent statistical support, Rosemary Bock for proofreading this manuscript and Joybrato Mukherjee, Magnus Huber and an anonymous reviewer for various valuable comments on an earlier version of this paper. Of course, all remaining errors or infelicities are my responsibility alone.

[2] Note that major aspects of theory, methodology, corpus analysis and native-speaker ratings, which are presented only summarily in this paper, emanated from a larger study on fluency in native and nonnative speech, which is described in more detail in Götz (2013).

[3] Note, however, that this approach also includes all mistakes and infelicities of the learners’ output as errors, as a pilot study by Kämmerer (2009b) has shown.

[4] This study is based on a previous version of LINDSEI-GE, which may differ slightly from the final version included on the LINDSEI CD-ROM (cf. Gilquin et al. 2010).

[5] The ethical aspects of this study were approved by the Macquarie University Ethics Review Committee (Human Research, Reference No: HE03MAY2008-D05765). If you have any complaints or reservations about any ethical aspect of this research, you may contact the Ethics Review Committee through its Secretary (Tel: +61 (0)2 9850 7854; Email: ethics@mq.edu.au). Any complaint you make will be treated in confidence and investigated, and you will be informed of the outcome.

Sources

Louvain International Database of Spoken English Interlanguage (LINDSEI): http://www.uclouvain.be/en-cecl-lindsei.html

References

Biber, Douglas, Stig Johansson, Geoffrey Leech & Susan Conrad. 1999. Longman Grammar of Spoken and Written English. Harlow: Pearson Education.

Borin, Lars & Klas Prütz. 1996. “New wine in old skins? A corpus investigation of L1 syntactic transfer in learner language”. Corpora and Language Learners, ed. by Guy Aston, Silvia Bernadini & Dominic Stewart, 67–88. Amsterdam: John Benjamins.

Brand, Christiane & Sandra Götz. 2011. “Fluency versus accuracy in advanced spoken learner language: A multi-method approach”. International Journal of Corpus Linguistics 16(2): 255–275.

Brand, Christiane & Susanne Kämmerer. 2006. “The Louvain International Database of Spoken English Interlanguage (LINDSEI): Compiling the German Component”. Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods, ed. by Sabine Braun, Kurt Kohn & Joybrato Mukherjee, 127–140. Frankfurt am Main: Peter Lang.

Bygate, Martin. 1987. Speaking. Oxford: Oxford University Press.

Chafe, Wallace L. 1980. The Pear Stories: Cognitive, Cultural and Linguistic Aspects of Narrative Production. Norwood: Ablex.

Chambers, Frances. 1997. “What do we mean by fluency?”. System 25(4): 535–544.

Chapman, Mark. 2007. “Theory and practice of teaching discourse intonation”. English Language Teaching Journal 61(6): 3–11.

Corder, Stephen Pit. 1974. “Error Analysis”. Techniques in Applied Linguistics, ed. by John Patrick Brierley Allen & Stephen Pit Corder, 122–154. London & New York: Oxford University Press.

Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press.

Cucchiarini, Catia, Helmer Strik & Lou Boves. 2002. “Quantitative assessment of second language learners' fluency: Comparisons between read and spontaneous speech”. Journal of the Acoustical Society of America 111(6): 2862–2873.

Cutting, Joan. 2006. "Spoken grammar: Vague language and EAP". Spoken English, TESOL and applied linguistics. Challenges for Theory and Praxis, ed. by Rebecca Hughes, 159–181. Houndmills: Palgrave Macmillan.

Cutting, Joan. 2008. Pragmatics and Discourse: A Resource Book for Students. Second Edition. London: Routledge.

Dagneaux Estelle, Sylviane Granger, Fanny Meunier, Jennifer Thewissen, Sharon Denness & JoAnne Neff. 2005. Error Tagging Manual 1.2. Louvain-La-Neuve: Centre for English Corpus Linguistics.

De Cock, Sylvie. 2003. Recurrent Sequences of Words in Native Speaker and Advanced Learner Spoken and Written English. Ph.D dissertation, Université catholique de Louvain.

Erman, Brit. 2007. “Cognitive processes as evidence of the idiom principle”. International Journal of Corpus Linguistics 12(1): 25–53.

Esser, Jürgen. 1992. Englische Prosodie. Eine Einführung. Third Edition. Tübingen: Narr.

Fernando, Chitra. 1996. Idioms and Idiomaticity. Oxford: Oxford University Press.

Gass, Susan M. & Larry Selinker. 1994. Second Language Acquisition: An Introductory Course. Hillsdale, NJ: Lawrence Erlbaum Associates.

Gilquin, Gaëtanelle & Sylvie De Cock. 2011. “Errors and disfluencies in spoken corpora: Setting the scene”. International Journal of Corpus Linguistics 16(2): 141–172.

Gilquin, Gaëtanelle, Sylvie De Cock & Sylviane Granger. 2010. Louvain International Database of Spoken Learner Interlanguage: Handbook and CD-ROM. Louvain-la-Neuve: Presses universitaires de Louvain.

Gilquin Gaëtanelle & Magali Paquot. 2008. “Too chatty: Learner academic writing and register variation”. English Text Construction 1(1): 41–61.

Götz, Dieter. 1986. “Intonation an der Hochschule”. Die Neuen Sprachen 85(5/6): 465–477.

Götz, Sandra. 2013. Fluency in Native and Nonnative English Speech. Amsterdam & Philadelphia: John Benjamins.

Gut, Ulrike. 2009. Non-Native Speech. A Corpus-based Analysis of Phonological and Phonetic Properties of L2 English and German. Frankfurt am Main: Peter Lang.

Herbst, Thomas. 1992. “Pro-Nunciation: Zur Bedeutung einer guten Aussprache in der Fremdsprache”. Die Neueren Sprachen 91(1): 2–18.

House, Juliane. 1996. “Developing pragmatic fluency in English as a foreign language”, Studies in Second Language Acquisition 18(2): 225–252.

Kämmerer, Susanne. 2009a. “Error-tagging spoken features of (learner) language: The UCL Error Editor ‘revised’”. Paper presented at the 30th annual conference of the International Computer Archive of Modern and Medieval English (ICAME30), Lancaster, May 2009.

Kämmerer, Susanne. 2009b. “Error annotation from the learner's perspective: Learner-centred identification and tagging of errors as a new annotation level”. Paper presented at the 30th annual conference of the International Computer Archive of Modern and Medieval English (ICAME30), Lancaster, May 2009.

Lennon, Paul. 1990. “Investigating fluency in EFL: A quantitative approach”. Language Learning 40(3): 387–417.

Liao, Yan & Yoshinori J. Fukuya. 2004. “Avoidance of phrasal verbs: The case of Chinese learners of English”. Language Learning 54(2): 193–226.

Lorenz, Gunther. 1999. “Learning to cohere: Causal links in native vs. non-native argumentative writing”. Coherence in Spoken and Written Discourse. How to Create it and how to Describe it, ed. by Wolfram Bublitz, Uta Lenk & Eija Ventola, 55–75. Amsterdam & Philadelphia: John Benjamins.

McKelvie, David. 1998. “The syntax of disfluency in spontaneous spoken language”. Corpus Linguistics: Reading in a widening discipline, ed. by Geoffrey Sampson & Diana McCarthy, 404–420. London & New York: Continuum.

Nesselhauf, Nadja. 2005. Collocations in a Learner Corpus. Amsterdam: John Benjamins.

Osborne, John. 2011. “Fluency, complexity and informativeness in native and non-native speech”. International Journal of Corpus Linguistics 16(2): 276–298.

Pawley, Andrew & Frances H. Syder. 1983. “Two puzzles for linguistic theory: Native-like selection and native-like fluency”. Language and Communication, ed. by Jack C. Richards & Richard Schmidt, 191–226. London: Longman.

Riggenbach, Heidi. 1991. “Toward an understanding of fluency: A microanalysis of nonnative speaker conversations”. Discourse Processes 14(4): 423–441.

Ringbom, Hakan. 1987. The role of the First Language in Foreign Language Learning. Clevedon & Philadelphia: Multilingual Matters.

Thornbury, Scott & Diana Slade. 2006. Conversation: From Description to Pedagogy. Cambridge: Cambridge University Press.

Wells, John Christopher. 2006. English Intonation: An Introduction. Cambridge: Cambridge University Press.

Wulff, Stefanie. 2008. Rethinking Idiomaticity: A Usage-Based Approach. London & New York, NY: Continuum.

Yates, Joan. 2005. Pronounce it Perfectly in English. Second Edition. Hauppauge, NY: Barron’s Educational Series.