Pittsburgh learner corpus & complexity checker

I am always looking for true learner texts to see how the complexity checker will perform.  EnglishGrammar.Pro is very lucky to have access to a very large and clearly documented learner corpus.

PELIC is based on data collected from students at the English Language Institute (ELI) at the University of Pittsburgh from 2005-2012…. The IEP data include written and spoken production from writing, grammar, reading, and speaking classes.

A note on levels from https://github.com/ELI-Data-Mining-Group/PELIC-dataset

Levels of proficiency in the dataset range from Level 2 (approximately equal to the Common European Framework (CEFR) A2) to Level 5 (CEFR B2/C1).

We can try breaking that up a little more, but exact boundaries are never that accurate.

2 = A2, (3=B1, 4=B2, 5=B2C1)

For this demonstration of the complexity checker, I have randomly selected texts at each level.  They are not handpicked to show how well it works.  The only extra selection time was at A2, to look for longer text entries since most level 2 were not much more than a sentence.  Also, it should be remembered that just because a student is in a certain level class, it does not mean they can perform to it continuously.  We all have bad days and good days in areas outside of language too!

---

Level 2 – A2

If I get homesick,the only one way is to cry than you will much better! And maybe you can talk with your friend, thay can help you a lot.
I never get homesick in here. i am very
happy everyday.

Even though there are many errors in the writing and therefore errors in the automatic tagging and highlighting, the complexity checker marks the level accurately at A2:

---

Level 3 – B1

My favorite restaurant is The Rose Tea, it is in Shaydeside. I like to go there, becasue they have delicious food and drinks. Their food and drink are all Chinese food or Taiwan food. I like to drink a bubble milk tea in Taiwan. Although a bubble milk tea is very expensive here, I still like to drink a cup of bubble milk tea every week. I just find three restaurants to sell this drink in Pittsburgh, and The Rose Tea sell bubble milk tea is more delicious than others. So I like to go there and drink my favorit drinks. Their food also is delicious, especially drumstick rice. You can choose what kind of method you want to cook it. I recommand you choose fry in oil.

The strong point of the complexity checker is that it highlights all the language that goes into the final prediction.  Here you can see a suggestion of either B1 or C1, which at first seems a very wide prediction.  If you look at what might make up the ‘advanced’ language you can see only 1 word ‘bubble’ in the C1 blue.  This is repeated 4 times, and also the B1 ‘rose’ in red two times.  Firstly, this student has not shown usage of  the words ‘bubble’ or ‘rose’ as stand alone concepts.  They are merely names the Taiwanese see.  Therefore, the lower value of B1 is the best choice for a complexity level.  This matches the student’s class level too.

---

Level 4 – B2

To boil water to 95 cent-degree. If you have water from fountand, it will be great.
Putting some loose tea in a tea pot. The quantity of the tea bases on the kind of tea. For example, if you use woolong tea, you only put a forth of the pot.
Pour down the hot water into the pot.
Then, pour out the water from the pot.(Only when you renew tea.)
Pour donw the hot again.
Wait around 60 minutes.
You can pour out the tea into your tea cups.
Don’t drink it first.
You should smell the flavor of the tea and drink
it little and little. That will make you know the value of the tea.

The procedure described here seems to be from a student that is not performing at the top of the level they are in.  (What would be even better is that these texts had grades given by the teachers).  Even though the class is level 4, there is not much evidence of it, either because of the task, or the student’s performance or something else.   Much of the green B2 complexity highlight rests on the phrasal verb combinations ‘pour out|down’.  B1 still looks like an accurate prediction by the checker.

---

Level 5 – B2/C1

When I was in Germany, I met a friend who was from America. His name was Devon. He was very funny guy. when I met him first, his voice was very deep. He had brown hair color, and brave and blue eyes. Also he was very tall and looked very strong. In addition, he looked pained because he had a big bruiseon his left eye. So I scared.
However, after we knew well each other, his voice changed melodious and his eyes changed smiling. he always smiled and he always wanted to know about my country. I realized I was wrong when I met him first. so now I’m very happy to know him and after this semester, I’m going to Oragon to meet him, my friend Bulldog!!

As with the previous text where the student appears to be performing much lower than the level they are in, this text at the highest level the corpus records does not appear to demonstrate at a more advanced level.  The text is longer, but reading through it as a human would (looking for meaning), there are moments where communicating the intended meaning is a little awkward.  A2 might be a little harsh to say to the student as a teacher in the class.  But the ‘complexity checker’ is cold hearted and only shows ‘when I met’ as some of its highest noticed but repeated structures.  The ‘B2 – C2’ “so I scared” is obviously highlighted incorrectly due to low accuracy.   I believe that B1 is demonstrated here if the communicative purpose of the text has been met.  As a computer reads it, I believe it is accurately marked at A2.

EDIT:

I added two words that are not in the EVP to the C2 vocabulary list.  And surprise surprise!!!  The checker now suggests B2 as the upper prediction.  This is another reason I require real data to test the complexity checker.  It brings up issues that the EVP data set alone cannot provide all the necessary words.

 

Finally, it should be remembered that complexity is only a small part of level prediction.  Language is used to communicate meaning.  As teachers, students and researchers, however, we are looking not only for accuracy, but also for what is not there.  What language would you attempt to introduce to these students?

---

---

17/11/2020

What else can we do with this corpus?

Well, in my wildest dreams, Cambridge would employ me to verify that the CLC EGP EVP data matches up with other large learner corpora.

To this end, and for convenience or simplicity, I will check the last grammar point that I linked up to the complexity checker.  It is a simple search for ‘by itself’, a B2 complexity marker according to the EGP.  Here I list every example in the Pittsburgh corpus.  From the longitudinal perspective that Ben suggested in the comments below, I found by accident that there is a student that is recycling a short cluster too, along their interlanguage journey.

The main conclusion for this grammar point is that although there are a couple of examples starting at B1, it is at B2 and above that there are many more reliable examples.

The two sentences at B1 are quite amazing to be honest.  The first one doesn’t seem to be from an intermediate student.  It sounds copied from an online advertisement.  Possibly from:

However, the second example shows that for me at least, that using intuition of a student’s level to answer something that they have read about at first glance, does require a tool.  I would have overlooked the level of the second sentence at B1, but when I see ‘earth can’t be created by itself and no one else’ etc. can be pieced together by a student, maybe that complexity checker is not going wrong.  To have a modal verb + Passive + preposition + reflexive pronoun + more…. I believe that even though the accuracy of the sentence is not spot on, the complexity is quite off the chart in that one cluster especially considering this is a reading class.  Again, when my students produce these kinds of language clusters, I trust in my friend google.  The first result returns some study on the same topic as this blog.

But, a shortening of the key phrase which is too similar to a native speaker’s ability for a B1 student to piece together, we find:

Now, language is clustered, and many believe that is how we learn it.  Maybe the student actually produced this language.  But if this is not under test conditions, then we cannot rely on the data that comes from it.  I know as a teacher myself just how hard it is to collate an online portfolio of a student’s work.  In the digital world, it becomes very hard to distinguish what has been copied from elsewhere.  At the end of the day, the students need to be engaged in learning, and if some utilize the technology of our day, good for them.  Everyone has strategies to complete tasks.  I don’t believe that copying or reusing learnt phrases is such a bad thing, it just may interfere for our research here.

Level 3 – B1

The $999 refrigerator has electronic touch temperature control, so if the refrigerator needs to lower its temperature, it changes by itself because it has a sensor.

Yes , i think . Because the earth can’t be created by itself and no one else aple to create the earth. *(Note: this is from a reading class)

Level 4 – B2

Prevention has never solved anything by itself.

If this participation can provide, this problem will have solved by itself.

In my country, rice a served a main dish by itself.

The solution has to by fighting the negativity, because the political issue will not get fixed by itself, while no one even noticing that there is a problem.

But at the same time I believe that we can all write about the same idea but every person will look at it from a different angle, precisely from his own perspective, so even if many people discuss the same topic, every opinion can be valuable and unique by itself. (fj2 duplicate for Arabic student, listed as Version 1)

Personally, when I write, I constantly ask myself can I do it? Will readers like what I wrote? So, even if many people discuss the same topic, every opinion can be valuable and unique by itself. (fj2 V1)

Level 5 – B2/C1

The beauty of nature, however, can’t teach anything by itself.

Consequently, death penalty should be abolished because, human can change by itself.  *(Note: this is a second draft)

Finally, while the Internet has a considerable effect for spreading misinformation, it also has a strong ability to correct inaccurate information by itself.

Many teachers did not understand that every child is a unique individual by itself and has different capabilities. (fj2 reading class)

Finally, I have not looked at question prompts for the texts or what lead up to the production of any of these texts may be.  The students may actually be working with an open book.  Key phrases might then be manipulated from the stimulus.

---

---

20/11/2020

* This post was a brief introduction to the corpus and some ideas were briefly tested.  As I work on my grammar research I will be pulling examples from it, but not updating this post here anymore.  To see what the future use will look like, here is another EGP grammar point, that has been expanded into iWeb corpus and then laced with PELIC examples https://englishgrammar.pro/indefinite-pronoun-relative-clause-focus/