Kotoba Rollers 2018: A reflection of the second term

More details of the framework (including worksheets) can be found here, but as an overview, the semester follows this structure:

Research games and choose one to playGo through the KR lesson cycle with Game 1Research games once more and choose one to playGo through the KR lesson cycle with Game 2Complete one of the 5 Final Projects

To save you scrolling (just control+F the following weeks) key parts of this post are:

  • [Week 1 and 2] Promoting collaborative learning as students work in pairs to research games.
  • [Week 5] An update to self-transcription analysis with more structured error-mining.
  • [Week 6] The introduction of a rudimentary speaking assessment.
  • [Week 8 and 9] More explicit instruction and goal-setting.
  • [Week 12 – 14] Final Projects: Relationship between students’ level of consent and the quality of their work.

Week 1: Game Research

The first week back, I introduced the syllabus to students and we got straight into researching games to play.

Teaching POINT: Why make videos at the end of the semester?

I introduced classes to the work of previous student groups, such as the One Night Werewolf Gameplay video or the Secret Hitler “How to Play” video. After that I asked the class what the importance or benefits of making videos like this might be.

Whiteboard featuring reasons for doing the Final Projects

We came up with the following list:

  • To help the next generation of students understand how to play the game.
  • So that students can see how much or what they learned this semester.
  • They can learn digital literacy skills by making the video.
  • The teacher can see what they learned and use that to grade them.

The main point being the first: helping the next generation of students learn how to play. This is perhaps the biggest benefit of their creations. The ‘how to play’ videos on YouTube are generally created by native English speakers and are therefore difficult to understand. Student created videos on the other hand will be much easier to understand because the creators’ English ability and pronunciation will be much similar to the viewers.

I also linked the whole cycle to the concept of mastery in gaming and how players start out as learners of the game, then progress to a state where they are competent players, then master players and finally, how some gamers come together to become creators of content for other players (phrased as “designers” in Squire, 2011, p.43). I am trying to create the same culture in my teaching context, where students progress in a similar fashion to a point where they give back to the community, helping future students. This may be considered a kind of semi-closed affinity space, where learners have a genuine (albeit limited) audience for their re-designed products (New London Group, 2000).

Game Research: From individuals to pairs

The original plan for this part of the lesson is for students to research four games by themselves and then present their findings to other group members in the following class. However, this is proving to be much more demanding than I imagined. So, I am adding in more scaffolding materials, and, as is a recurring theme in my iterative design of KR, reducing the workload to something manageable.

Students were told to research games in pairs instead of individually so that they can discuss what they found, and immediately check their understanding with a peer, thus providing them with a socially-informed way of creating knowledge through dialogue, and as learners work together in the zone of proximal development. Additional benefits of working in pairs or “small groups” instead of individually:

  1. Feedback regarding their comprehension of game rules
  2. Language support of a peer to start considering how to present their findings in English.
  3. Increased responsibility based on the creation of individual roles (see Murphy & Dornyei, 2003, particularly in Chapter 7).
  4. An interlocutor to practice their presentations with and further understand where their interlanguage/proficiency fails them.

Watching other groups’ videos

As part of this class, students get the opportunity to view the videos that former students created as part of their final projects. This is the first time the videos have been seen in a truly authentic way by their peers. What I mean by this is that when I view the videos and grade them, I already know the content and so I’m not watching videos as a way to learn the game that they are introducing. In the Game Research (Week 1) class, students use the videos for the authentic purpose of learning how to play the game. This point hadn’t occurred to me until today.

I asked one student what he thought of a video he was watching and his response was that it was easy to understand the rules, but not the gameplay (I assume he was probably watching a gameplay video rather than a rules explanation video for this point). My own reflection here is that it may be worth getting my classes to review each others’ creations with a peer assessment rubric. This is certainly on my mind, but I have no concrete ideas for implementation yet. I will add this to my KR to-do list.

Practice presenting

Students were also asked to present what they had found so far to the rest of the class as a way to start noticing the gaps in their interlanguage. In other words, by trying to present (articulating verbally) the content that they have written down, they may find that a difference in what they can comprehend and what they can articulate. As a concrete example, one student started to read out details of his game and was unable to verbalise the hyphen in ‘3 – 5 players.’ I was nearby and this able to instruct him in how to do this. Without engaging in English through such experiences, they will not notice any problems, and certainly not improve their competence in speaking. This is therefore an example of forced output as a way of noticing the gap between their proficiency of various English skills (see Swain & Lapkin, 1995).

Week 2: Game Research Presentations

The second week of the semester is where students present the details of their game to others. This is an output focused class, where the communicative goal is to transmit their knowledge of a game to another person orally. The non-linguistic goal is to learn as much as possible about the different games that I have on my game list in order to make an informed decision regarding which game they will choose to play.

I made the following analogy regarding communicating in the L2:

Language is a tool for transferring information from your brain to somebody else’s brain. We can consider language a LAN cable, transferring this information. Now, you have a very high-speed LAN cable that you use daily: Japanese. However, as this class can be considered a game, like golf, (and making reference yet again to Suit’s (2014) definition of a game) we have to use inefficient means in order to get the goal of “receiving credit at the end of the course.” In this class, the inefficient means of getting that goal in the low-speed LAN cable of… yes… English.

Unfortunately, the analogy mentions nothing of WHY students should do anything in English other than to not fail the class, which is the elephant in the room regarding KR right now. That being: What are the wider social, real-world goals of participating in my class other than just “practice using English in a safe environment for 100-minutes a week.” But alas… the abyss is deep.

Some further observations:

Better in pairs

As mentioned previously: Students work much better in pairs as opposed to doing this activity individually. It’s pretty obvious why if we consider the social nature of learning and language use. Having a partner to research with has the following benefits:

Reduce slacking

The social pressure of having to pull one’s own weight during the research phases is helpful in keeping each other responsible for producing quality output.

Check comprehension

Students are able to check their understanding of the subject matter (in this case, mostly: rules) with their partner. Thus, they can fill the gap in each others comprehension or test hypotheses with each other about a certain point. This leads to promoting confidence in the subject. Although I have no empirical data, I found that students working together seemed much more confident that they understood how a particular game works in comparison to students that researched individually (last year).

Provide moral support

Being able to check comprehension with their partner boosted confidence, but just having another person “on one’s side” or able to “back them up” as moral support seemed to boost confidence also.

Students have very little confidence in their English

They can write down a bunch of sentences and practice talking but actually, the presentation is just reading English sentences verbatim and then talking at long length in JP after. It’s like they have to get through the English as quick as possible, not check any meanings with their interlocutor so that they can get to the JP explanation instead.

I had to explicitly state that they were to do the activity in English, to use English as the form of communication, not just something to be rushed through.

Lack of research skills

I noticed that the general trend in students research habits was:

  • Learn rules in JP → try and translate JP into English → write that on the worksheet

When instead, a much more logical process (for me at least??) would be to do the following:

  • Learn rules in JP → find an EN version of the rulebook to see how to phrase things in English → Paraphrase what you read in EN → write that on the worksheet

Do they just have an aversion to even looking at EN text?!

I updated the KR textbook to present this way of researching more explicitly.

Japan’s excruciating lack of EN ability

I have a Chinese foreign exchange student in my class using EN as his third language. He absolutely nailed the presentation phase doing it all in very fluent English. He was also working individually..! The JP students in the same class, that have had formal EN lessons for the last 6 years, stutter and fail for the most part. Yes, I may be over exaggerating, but it prompts the thoughts:

  • Why is he the exception?
  • Why is speaking English not “normal” in Japan?

I hear students laugh about their terrible English ability and just end there. There’s no shame in it at all. It’s like this country isn’t supposed to be able to speak Japanese. The notion that “That’s the norm, so why change it?” Infuriating.

Their written work does not match up with their oral output

For instance, the game research guide worksheet: They have some great work on their worksheets, they’ve thought about how to express themselves, but when they go around to speak to people they just rush through just or do it in JP instead.

Explicit instruction

In one of my slightly higher level classes, we slowed things down a lot. I made it explicit that pairs were to present details of one game ONLY in English, and we took time to think about how to present:

  • What key elements of the game are unique and worth talking about?
    • Make a list of three things that are core/unique/interesting to this game.
  • Who will say what?
  • How can do can we check comprehension?
    • Shadowing..!
    • Asking questions as a listener.

I also demoed the sequence with a student that is not higher in proficiency, but may be higher in confidence. He actually shadows naturally instead of just nodding or staring blankly. The demo helped others see what it means to be a good listener.

It went OK. I could see that some pairs didn’t really get across exactly what they wanted, and some groups didn’t understand fully, but so what…! I just wanted them to get across a general feel for what the game is and what it’s about. That’s a good start! ??

Week 5: Analysis — Serious error mining!

With some classes, I’m using the experimental analysis doc. This is a much more rigorous self- and other-analysis doc which was designed after rereading the literature on self-transcription. I’m not using it with all classes yet to see how it goes with the more advanced (in terms of study habits) classes first. Essentially, I realised that I hadn’t been very thorough with the transcription and self error correction with the original ‘Analyse’ class and am working on correcting that, as well as to see what the potential is for more rigorous academic work.

Some observations:

Students quickly grasped the differences between the error types in my introduction (morphological/lexical/syntactic). They did this easier than I expected.

Thanks to some suggestions from Peter, I added a page on the worksheet for students to practice coding errors before they looked at their own. The errors are all based on things that I have observed in class such as missing articles, verbs conjugated incorrectly, and words in the wrong order. These initial errors are helping students to find the same mistakes in their own output. They worked hard on the worksheet to find errors in their own transcription but, of course, they were not able to find all of the errors they made which is common for self-transcription activities (Stillwell, et al., 2010). We used the following additional process to allow for peer-review. This, again, sets up the potential for socially-informed, collaborative learning:

  • Find, underline and code your own errors.
  • Pass to the left and check over a partner’s errors (Rechecking)
  • Pass back.
  • Discuss errors as a group.
  • Tally up all errors in the group
  • Write out common errors for each category (Morphological/Lexical/Syntactic).
  • I go around the class and point out errors as best I can also.

The lesson has a lot of content, from learning about errors, coding practice, finding their own errors, translating JP → English, and mining YouTube videos for new phrases and expressions, however, the students don’t seem to tire. I, therefore, think this class was the right level of “difficulty,” appropriate to university-level students’ intellect.

Some photos of a student’s worksheet:

Week 6: Replay — Speaking assessment introduced

I realised that I haven’t been assessing students on the main skill this class is supposed to develop: speaking. They have a written test at the end of the year (which I decide zero of the content for) which accounts for 50% of their grade. The other 50% I’m directly responsible for giving and up until now I have been basing this grade on the quality of written work they give to me. So, I am going to grade/evaluate the students speaking skill as they perform during gameplay in the Replay class. The (simple) rubric is below:

poorbelow averagesatisfactory excellent

For these topics:

  • Fluency (how much they spoke)
  • Accuracy (how accurately they spoke)
  • Volume (how confidently they spoke)
  • Use of game-specific terms
    • Did they refer to items by the correct keyword or just generically?
    • How well do they know the game in terms of:
      • The language required to play well
      • How to play in general
  • Cooperation (Did they only consider themselves or did they speak about/with others?)

I used the rubric above to grade students during gameplay. I was surprised by how much students could use English in the second gameplay. Essentially, it proved to me that the work we did up to this point has helped them realise their potential. It also means that unfortunately:

Unless I assign X, they won’t do X

Simply: I feel that because this class was “graded” students tried harder than they would if it was not. From a gaming perspective, they will “game the system” and find the shortest way to get their course credit. In my class, this means that unless they are specifically told to use English, some of them won’t. It also shows how “gamified” education is in general, or at least, how extrinsic motivation is the only way I can get some students to engage with the course content.

Another way of framing it is that they have no purpose for using English other than for a grade. There is no authentic audience or interlocutor they are communicating with, only classmates.

Week 7: Reanalysis

Impossible to compare the two transcriptions

I had hoped that it would be possible to compare their performance during the two gameplay sessions by tallying their errors in the first session and the second session and seeing if there was a decrease between the two sessions. However, I didn’t (foolishly) consider the fact that the two play sessions would be different in length. So, by merely comparing the total number of errors between the two sessions, although some figures went down, others went up. What really needed to be done was for students to consider how many errors were made as a percentage of the total number of utterances. For example:

Play SessionNumber of
utterances
Lexical
errors as % of
utterances
Syntactic
errors as % of
utterances
Morphological errors as % of
utterances
110010 (10%)2 (2%)5 (5%)
250030 (6%)1 (0.2%)5 (1%)
Difference+ 400%4% decrease1.8% decrease4% decrease

The above makes sense to me. If I’m being over simple here or there is a better way of showing improvement over the two sessions, please let me know.

Week 8: Game research — Explicit instruction again

In a word: The power of expectations.

Students completed the following steps in today’s class:

  1. Research a game.
  2. Tell your partner about the game you researched and work as a pair to prepare for a class-wide speaking activity where you introduce your games to others.
  3. Tell other pairs about the games you researched.

I proposed for them to search in Japanese first, then look for an English version of the rule book for them to see how the same is said in English, then for them to write on their worksheet some keywords, verbs of the game, and 3 interesting points as a way to quickly introduce the game

The first class:

Parts 1, 2 and 3 were done all in Japanese despite me telling them that the aim of the class was to use their English to explain something to others.

When I got close to a pair, the speaker would stop and try to say something in English then revert to Japanese straight away. Some didn’t even bother doing that.

I had to tell them at the end of the class that I was a bit disappointed in their effort and asked, “What exactly did you do in English today? Nothing, right? And what class is this? Your Japanese class? I don’t think so….”

Why though? Some ideas:

  1. This is a monolingual class. I get it. For them, speaking English to each other is just totally unnatural. But that is the goal of the class. It’s also totally unnatural for them to be programming ifs fors and other such loops into a computer, but they learn that language…!

What I _could_ do in this instance is to actually ALLOW the speaking to be in Japanese but make the first part explicitly searching for English sources only. So in a way, this would be hitting all the right buttons in terms of Lambert’s (2011) needs analysis which showed that one of the main skills for university graduates is to be able to translate and describe L2 sources into Japanese.

  1. They want to explain the WHOLE game to their partner. Not just the interesting parts. Sometimes those interesting parts don’t make sense until you know the whole rules.
  2. They just can’t be bothered to try.
  3. I have allowed the class to fail at this task by not promoting “English only” activities. About that….

The following class:

I made it VERY VERY explicit that I wanted the third part of the class done in English. I told them that the first two activities were designed specifically to aid them in that third activity, and that they definitely could do it if they only tried. I also made a point of telling them that it didn’t matter that they don’t tell their partners ALL of the rules, as long as they get a gist of the game.

And it worked.

The third stage of this class was done all in English with students REALLY trying to explain things like how the assassin works in Resistance: Avalon.

The power of explicit instruction (and threats?) ladies and gentlemen!

WEEK 9: Learn — MORE explicit instruction

Reading the rule book in order to learn the game. So, an input activity, mostly reading, but some groups did make reference to the previous students’ videos and YouTube.

The main problem with the class is that there is a massive difference in effort between groups. It’s not consistent at all. Some groups doing almost every single part of the class in Japanese, others using a mix of English and Japanese and others still that predominantly use English.

Why is there such variety?

  • Am I not strict enough?
  • Is there not enough departmentalisation of time? In other words time-boxing, or pomodoros or something else to explicitly allot time to DOING THINGS IN ENGLISH?
  • Have I allowed for a class that doesn’t feel like they need to speak English?
  • Are they just lazy?
  • Is it because I’m not grading them or testing them on their work in looking through rulebooks and watching YouTube?

Other groups COULDN’T do the learning activities in Japanese because their game doesn’t have a Japanese rule book or sites which explain how to play the game in Japanese. They were, by default, forced to do the activity in English. Also, because they had played the game before in English they also carried out the test play in English. Why are they the exception though? What motivated them to do that?

Again, for the majority of groups today the old adage reared its head again:

If it is not graded or required, they will not do it.

Seriously.

Update

I gave an extremely explicit talk about the learning goals of the class during my classes today (Thursday, first classes after the disappointing Tuesday class).

I took 10 – 15 minutes to outline what the goal of the class was and decided that actually the goal could be separated into two subcategories, which we then brainstormed. Learners lists were very very poor. Most of the students wrote “speak English” as the English skill for the day which absolutely blows my mind. In this class, they have to read the rulebook, watch rules explanation videos, and discuss the game rules with their group members… The goal of THIS CLASS is definitely NOT speaking. I can’t quite understand why they wrote that.

I created the following list:

Goals related to the game

  • Learn the rules
  • How to play
  • Individual character abilities
  • Keywords

English/academic goals

  • Collaboration
  • Cooperation / effective communication
  • Smartphone usage (searching for word meanings)
  • Reading an English text
  • Listening/watching an English video
  • Interpreting the text
  • Translating the text
  • Learn new
    • Vocab
    • Grammar

Regarding these skills, I mentioned that some (like collaboration and cooperation) are clearly not limited to English language ability but are considered 21st-century skills, or more concretely: the skills that anybody entering the workplace should possess. The English-related skills I tried to show examples of how they may transfer into work requirements placed on them in later life, such as translating an English document for other colleagues at work.

Upon making the goals of the class explicit like this, I asked them what they need (as a prerequisite) in order to complete the activities in italics above. I think my question was too obvious for them. They genuinely didn’t know how to answer. Like I’d asked them a trick question. I asked them in Japanese too. Multiple times. Some answers I got:

  • やる気 (willingness to try)
  • A dictionary
  • A mobile phone

No one mentioned the obvious:

  • English input/text

You need something written in English to be able to practice reading in English, right?

They laughed. But there was a reason I did the whole speech at the start of the class: because some students, groups and even classes do not even bother to use an English text as their input for rule learning. We know this. I’ve talked about the above. And I let these students know it, too.

You may laugh, but I’ve been in classes where people go from Japanese rulebook to speaking about it in Japanese to then test playing in Japanese. So they don’t actually learn any English at all.

The spiel worked for my lowest level class this afternoon. They all spent over 45 mins each reading through their chosen game’s rule book in English, making notes in Japanese on how to play (interpreting) and jotting down the meaning of words they did not understand.

Following that, I instructed them to watch a YouTube video which explained how to play their game before they finally compared notes with their other group members. A summary of activities then being:

  1. My speech (15 mins)
  2. Reading and note-taking individually (45 mins)
  3. Video watching individually (10 mins)
  4. Group conversation, rule checking, and set up (20 mins)

What did I learn?

That I need to communicate why we are doing class activities. What the goals are and how they link to possible further life skills.

Update 2: New critical analysis question added

I’m finding it hard to push students to even get proficient enough to do things like play codenames in English. It’s a struggle because there are multiple different student “types” if you will. Borrowing from Bartle’s player types, just thinking up some typical students for my classroom:

  • Killers
    • Grade Hunters — are willing to do what is necessary to get a grade, but nothing more.
    • Game Hunters — just want to win the game (and not even think about using English, but at least they are engaging…).
  • Socializers
    • Jesters — want to play and have fun. Generally well-liked students but joke around at the expense of English language development.
    • Anti-socializers (those students who really struggle to interact with their classmates in Japanese, let alone English)
  • Explorers
    • Starry-eyed Steves — Those that will go the extra mile to learn all the ins and outs of a game and help others play it well. This usually inspires them to work hard on their English, too.
    • Serious Sallys — Students that are too serious for gaming and wish we were doing TOEIC test practice instead
  • Achievers
    • The leechers — who are happy to let other people learn the rules, analyze worksheets and do homework, as long as they also get an achievement

In terms of group dynamics:

Some groups recognize that the game they are playing is incredibly fun, and if they use as much English as possible, can really push themselves to learn new expressions and make themselves understood in the L2. Other groups are much much less engaged. They are happy to just come to class and not have to look at a textbook. A real range of abilities and level of engagement.

To try and raise that bar regarding what I expect, I have been experimenting (again) with some more “academic-style” questioning during this phase which they are to answer in writing (i.e. not speaking, their weakest skill). The question was simply for students to reflect on their learning of game rules and give preference for learning via the rulebook or via instructional video and to give a reason why. I expected most students to glance over the question and not really pay it much attention, but I saw some great answers (of course I saw poor ones too…).

Really contemplating changing some things up with KR for next year… Maybe just play one game and do a range of different analysis projects.

Week 10: Play

The second playthrough. This class was also another instance of the speaking test which I outlined above.

I am still curious as to why students don’t refer to their worksheets from previous classes during play. We have done three weeks’ worth of classes up to this point to help them to prepare for this “test” play… Some possible reasons:

  1. They don’t have the time to check their worksheets because the pace of the game is too fast.
  2. It doesn’t occur to them that they are allowed to check their worksheets (based on the knowledge that other classes forbid the use of calculators during tests, they may think the use of worksheets during their play sessions is not allowed)
  3. They don’t really care how well they speak English during play as long as they are having fun.
  4. They are not concerned with how well they speak English because they are pretty sure they can pass the class without working hard.

Honestly, there could be students in the class that have one or more of those opinions plus others I haven’t thought of. However, unless I do some formal surveying, I can’t claim to have a concrete answer.

Week 11: Analyze

Focusing explicitly on errors during play is possibly too myopic in terms of the kinds of analysis they could do.

Concrete example:

The Sheriff of Nottingham group had no idea about the mythos behind the game. They played it well, they used lots of English, they learnt some new expressions, they had fun, but they didn’t dig into the background or theme of the game. This was a perfect opportunity to get them to do some research about the game in more depth. They had heard of Robin Hood but had no idea that he was being alluded to in the game they were playing. They also didn’t know that Nottingham was a real place…!

Now, not every game lends itself to such follow-up discussion work. Or, maybe they do… Interesting food for thought and a direction that KR could go in the future.

Week 12 – 14: Final Projects — Consent and engagement issues

As always, no group decided to do a presentation about how to play their game to other students. The ‘live’ nature of the project terrifies students.

The Sheriff of Nottingham team were interested in remixing the game into a Japanese context. They started playing with the idea of re-theming it so that mead was saké and bread was rice. That could be a project for them next year if I have them in my class again.

Use of Google Docs for written reports

Game Reviews were created with Google Docs this year which was a much better system. Some key takeaways:

  • I set up a specific time mid-way into their doc creations to leave comments and feedback which I could not do without using this technology
  • Transcription groups could work together at the same time in class on different parts of the doc: a few students writing the spoken part, a few others writing grammar guides.

Video editing software:

I recommended Videoleap as a good, more robust alternative to iMovie for some groups, and they made great use of the additional features. I also saw groups talking amongst each other regarding editing techniques and software. “Tips” were also given by one group that had finished to a group that was still in the process of editing. This reminded me of apprenticing and LPP to a certain extent.

One group, in particular, surprised me with the way that they created a super short, comprehensive guide to playing One Night Ultimate Werewolf:

As part of creating their video, they are instructed to watch and analyse a “professional” video on YouTube, which just so happens to be the same game (link). This analysis stage is designed to get learners thinking about their own video project, specifically regarding the sequence that they should introduce elements of their game. I tell groups to not get concerned about the quality of the “professional” video, and that I don’t expect their work to be of the same standard, but the group above created something that is incredibly simple to understand (for future students) and is on par with the video they analysed in my opinion.

Storyboarding and transfer of skills from other classes

One group started but did not finish using storyboards for their project — showing that they drew upon the skills they had acquired in another context. In sum, they were transferring those skills to the English classroom.

On consent and willingness to engage

I experimented with a new consent form asking for permission to post their work on youtube publicly. A few groups have been kind enough to allow me to post their work publicly, and as you might imagine, those that did are the groups that put a lot of effort into their projects and are proud of their work. These groups are aware of the genuine, authenticity of their audience and planned to create something of value from the beginning of their project such as the group that created storyboards.

My thoughts going forward regarding consent and student work is to continue to provide the option to opt out of public sharing but remove the option to opt out of sharing with other students. It clear to me that there is a direct relationship between the quality of a group’s production and their willingness for it to be shared with others. Simply: the more effort students have put into a project, the more willing they are to share it with future students and the public. The opposite is also true: students that put in no effort don’t want others to see their shoddy work. Therefore, it seems that by allowing students to remove the genuine audience (by opting out) allows them to slack on the work. Removing the option to opt out could inspire learners to work harder.

It’s an interesting side thought that failing the class doesn’t seem to be an indicator of improved motivation to create a high-quality product. It’s as if they are already sure they will pass the class regardless of the quality of the work they do.

Issues with worksheets

Some students (50%??) ignored the worksheet for the most part and just went ahead and made their video, making up the content on the fly. Upon completing the video they then tried to fill in the worksheet. In effect then, attempting to do all the planning work after recording. A useless task, which is clearly not the way I intended the worksheet to be used.

They seem to look at the worksheet to get a feel for what I’m asking for, in broad strokes, and then go about making the video themselves. It’s not entirely bad, but I feel like they are wasting the opportunity to plan their project beforehand.


If you made it this far, thanks for reading. I really appreciate it.
Comments are very much welcome.
?

References

  • Dörnyei, Z., & Murphey, T. (2003). Group Dynamics in the Language Classroom. Cambridge University Press.
  • Lambert, C. (2010). A task-based needs analysis: Putting principles into practice. Language Teaching Research, 14(1), 99–112. http://doi.org/10.1177/1362168809346520
  • New London Group. (2000) A pedagogy of multiliteracies: Designing social futures. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures (pp.9-37). New York: Routledge.
  • Squire, K. (2011). Video Games and Learning: Teaching and Participatory Culture in the Digital Age. Technology, Education–Connections (the TEC Series). Teachers College Press. 1234 Amsterdam Avenue, New York, NY 10027.
  • Stillwell, C., Curabba, B., Alexander, K., Kidd, A., Kim, E., Stone, P., & Wyle, C. (2010). Students transcribing tasks: noticing fluency, accuracy, and complexity. ELT Journal, 64(4), 445–455. http://doi.org/10.1093/elt/ccp081
  • Suits, B. (2014). The Grasshopper: Games, Life and Utopia. Broadview Press.
  • Swain, M., & Lapkin, S. (1995). Problems in output and the cognitive processes they generate: A step towards second language learning. Applied Linguistics, 16(3), 371-391.

Extras

As a teacher at TDU, the class I am in charge of is an “English communication” class with a focus on speaking and listening skills. All I’m really told to do is improve their speaking ability. It’s an incredibly vague, CLT-inspired goal such that ‘student talk time is paramount.’ My teacher training and general reading of SLA literature brainwashed or at least influenced me to have those very same thoughts: “I should have the students speaking as much as possible,” or I’m happy if students interacted in English during class. My introduction to colleagues such as Jonathan deHaan and Peter Hourdequin and subsequently the work of New London Group, Multiliteracies, Connected Learning and other progressive approaches to education really blew my mind and popped my CLT-bubble. It made me realise that I’ve been focusing too much on my own goals as a teacher-researcher interested in the potential of a pedagogical sound game-based language learning environment. And, as a part of this, I have been ignoring or not thinking about the goal of the class outside of “getting students to speak English.” Of course, KR has come a long way since its initial iteration, but learner goals are largely ignored still.

As my current framework is not designed in advance to teach any specific skill, that would suggest that I could improve it by either:

  1. providing students with English education that is more appropriate for when they graduate (i.e.focused more on their major field of study)
  2. teach English with a focus on academic/thinking/global participation skills.

I can’t see the students at my university having any need for English in their lives post-graduation, and so it may be time to consider teaching other, more useful skills through the medium of English…

Kotoba Rollers 2018: A reflection of the first term

? Two Rooms reflection

Two rooms and a boom is the first game we play. Here are some small tweaks to the general model I’ve made, why I made those changes and reflections on what the changes brought about.

Model updates

Grammar presentations

During the first analysis session, students find errors in their transcriptions, discuss how it could be corrected and make notes on grammar, phrases or vocabulary that they will be able to use in the next play session. The presentations went well but have been lacking a ‘test’ element. The “test” element may be considered a quiz where each group introduces the grammar/phrases/vocabulary that they researched as well as four or five questions to check other groups’ comprehension.

I wanted to do this activity so that there is a need for students to listen to each others’ presentations. Additionally, as they are all playing the same game, the lexical items or grammar that each group researches will have relevance to all other groups. Finally, by grading themselves and giving themselves a score I am able to use it towards their final grade which hopefully inspires them to be responsible for what they do in the classroom.

Here is a sample worksheet for the grammar presentation and quiz

I only did this activity with one class. It went well, but they are still only focusing on super low-level grammar. “Let’s…” “Do you want to…” etc. Where is the deeper dive?

Let’s play video → SUSD video

I didn’t show my Minecraft-based let’s play video, instead opting for SU&SD’s play through. As a part of this, I made students answer six questions based on the video such as:

  • How does Quinns ask the leader what colour he is?
  • What do players say when giving information to their teammates?

As a result of watching the YouTube video, I saw those expressions come out in the subsequent play session, but only in limited, isolated instances. Not all classes or students seemed to internalise the expressions they had been exposed to. In fact, it was the vast majority of students that used what they had seen/heard the previous week. This calls for additional viewings before the following play session to refresh their memories.

Playfulness

I witnessed a fantastic episode of playfulness from one student who was given the demon character card. The card states that the player must lie. As a result, the player didn’t just lie about things related to the game but ‘meta-lied’ about non-game related topics. One example of this is in his utterance: “I have a girlfriend,” poking fun at himself.

Such episodes were again isolated though. I feel that for my context the ‘leader’ character really is a paramount, essential, locus of attention role which prevents or inhibits other players from ‘going about their business’ of gathering information individually. However, whether it’s the games fault that students aren’t more creative is difficult to say.

(Some, no… Most) students are very passive during Two Rooms play

For instance, the Gambler is an easy character to win with. They just need to know who the president and bomber are before the end of the game. If they have this information, they can accurately guess which team has one. However, without making this point explicit, and promoting the gambler mid-game, the person who gets the gambler role just tends to stand around and wait for information to come to them. Why is that? It has to be a mixture of these elements:

  • Lack of understanding
  • Lack of language skills and confidence
  • Lack of interest

Or, as mentioned above, the leader character is such an alpha role, it demands attention from all other players.

I can’t figure out how to teach these students to work as an individual during the game. Could this also relate to cultural differences? Maybe… A tough question to solve.

Post-play character research observations

After playing Two Rooms in the first play session, all students are given the homework of researching a few of the (oodles of) extra characters in the official Character Guide. The task in the following class is to introduce those characters to other students and choose some of the characters to add to our game. However, some students just copied down the text that was written next to characters without trying to understand the meaning!

Perfect! /s

If they did not go to the effort of understanding the unique features of new characters, they were not able to explain this to other classmates. In other words, they were not interpreting the text in a way that either they could understand or their interlocutor. They were just reading the text verbatim and hoping their interlocutor could understand.

This is one of the best examples of how to structure “homework” in my opinion.

The work that students have to do at home is an essential part of the following class and essential for progressing through the model (i.e. play → analysis, debriefing, and learn some more about the game → replay). If they do not do the homework, they cannot participate in the following class. In contrast, in a previous teaching position homework was generally comprised of completing reading activities. These reading activities typically appear at the end of textbook chapters as a way to “check understanding” of the chapter’s content. But there is no following activity… No necessity to complete it other than “you’ll get an F if you don’t…” No connection to the wider curricular aims. That, to me, is an example of ineffectual, boring, generic homework.

So, to backtrack a little, I have students talking amongst each other about characters they have researched. And, among those students are those that haven’t done the homework and have no idea what the “Angel” does or how the “Hot Potato” works. I need to improve this part of the model to be able to identify and punish those that did not do the homework properly. Yes, they already get a sort of punishment by looking foolish in front of peers, but it should have an effect on their final grade, which, at the moment, it doesn’t. This activity should be assessed more aggressively though. Maybe get each student to explain their character to me, and then I grill them on the meaning of the new rules. However, they respond to my questioning determines their score for that particular class.

Post-play reanalysis and discussion observations

At the end of the play, analyze and replay cycle I have asked students to discuss the results of their second play session. In other words, asking them to consider the following question — “Did you use the grammar points or expressions that you researched between play sessions?”

This is a valuable activity as they will be doing the same cycle three more times this year, and so it can inform them on how to conduct the “analysis” session with more focus, more accurately identify errors, or improve efficiency for in future cycles (i.e. so that they research grammar or expressions that they will actually use in subsequent play sessions, or at least think about how to prepare for subsequent play sessions in future cycles).

However, the overwhelming result is that, no, they didn’t use any of the phrases that they looked up during the first analysis session…

One thing they didn’t pick up on was the lack of deep analysis they did between play sessions. For example, some groups looking at how to translate a single, isolated, very specific sentence or word into English. However, just because it came up in the first session, doesn’t mean that it will appear in the second session.

For example, one group looked up:
言ったらスパイにバレる → translated to → The spy will find out if I say.

Then, in the second play session, we didn’t use the spy character, and so they didn’t use that phrase.

They need to look more broadly at their output and find structures or patterns that can be utilized in the next play session instead. Situations change. They need to do deeper analysis.

Reasons that were found or put forth for the lack of usage of the words they researched during the analysis stage:

  • Game time limit too short
  • Not enough time to think
  • Too many characters causing cognitive overload
  • No practice of language before class
  • The situation changed so didn’t need certain items. (as described above)
  • That they were lazy
  • ⭐ They can make themselves understood without using long sentences. Words are enough.

Another new addition to the post play reanalysis class was for students to use a modification of the “5 whys” method of exploration (I just asked students to ask “Why?” three times instead). In other words, I told students to ask “why” three times for each reason:

“We didn’t say ‘Do you wanna … ‘ very much.
Why? → Because we didn’t remember it.
Why? → Because we didn’t study before class.
Why? → Because we aren’t interested in English.

Yes, this reveals some hard truths..! Be careful.


??‍♀️ Spyfall

Much like the 2R1B post-play character research above, when students were asked to look for interesting questions online, they did the activity (searching on Google for “spyfall questions”) but they just wrote down the questions verbatim without considering the meaning. Maybe… One question, in particular, tripped students up:

“What brings you here today?” which they thought meant “what did you bring here?” That was a teachable moment!

Students are not engaging in “reproductive literacy” or #TransformedPractice (a term coined by Freebody and Luke, 1990). They are not remixing the questions or thinking about the underlying structures of the questions that they find in order to generate similar questions or even how to use those questions during gameplay….!

Shadowing (or lack thereof)

I noticed that the vaaaaast majority of groups do no shadowing during gameplay at all.
It is definitely something that needs to be brought up. When a particular student in a particular group was shadowing, it was so very very natural for me to hear, and it really helped show his comprehension of questions. I think I need to push this technique more.

Analysis: YouTube video watching

We took a full hour to do this activity. In groups.
By taking each question individually and really drilling down, they produced some surprisingly good results.
Of course, I went around the room and helped them out, as they are not really proficient enough to pull out example sentences from the YouTube videos on their own (in most cases).

Some things we picked up on:

  • “I guess” as an ambiguous answer
    • One student in the group that found this phrase used it in subsequent sessions also which was amazing for me. Even into the “Spyfall Tournament” phase of the semester.
  • “I don’t buy it”
  • “Would you…” as an introduction to conditionals
  • “Where we are” as a way to talk about the location (instead of just “here”)

An interesting ‘difference’ between students and natives was that the natives spoke very fast and had no empty time. Students in my class didn’t think about the fact that the video might be edited…

The learner is not building towards an ideal version of the language which exists in abstract. Rather, the learner is building on and out of his perception of the usage of the language heard in the mouths of other language users and this construction process in the life of every language user is the only meaningful definition of what the language is. (Sockett, 2014, p. 29).

This line made me think of the work I’m doing in the post-play analysis lessons. We are looking at authentic resources to try and supplement their interlanguage.

Grammar instruction

I’ve been experimenting with how I give the grammar instruction between play sessions. Last year, I gave them the worksheet and told them to work through it. Before that, I have asked students to look up particular grammar points on the internet. This time I stood at the front of the room and got them to walk me through the process of making questions in English (in some classes) or got groups to work together on generating a pattern of English sentence formation (in other, more advanced classes). We looked at two particular types of question formation:

  • yes/no questions
  • QUASM questions
    • question word
    • auxiliary verb
    • subject
    • main verb
    • predicate

As an example, writing the first line on the board with separators between the words and getting them to think about the underlying structure of yes/no questions.

Do You Like natto?
Be (are, is) he
she
it
they
your mother

So, instead of me just giving them a “grammar guide” we talk through it or they talk through it in an expert-novice setting, within the ZPD, and from a socially-informed approach to SLA (am I missing any keywords?)

Report phase

I’ve introduced a more formal presentation activity for the final report class. It still needs a rubric created for it though. Some examples:

18RT responses


17RD responses

What is the goal of the class?

This is something that will (probably) need to be fleshed out in a future post. It’s one of the major things on my mind right now and something that I’ve been avoiding. The elephant in the room ? for sure. However, I’ll sketch out the current, basic idea here.

For the most part, the students in my classes have never used English for authentic communicative purposes. By this, I mean that the only oral (and probably written) output they have ever produced has been scripted ? . E.g. “Chapter 2: Planning a party” where students see a model conversation, change a few words and practice using that form as part of a “task.” Such language use is incredibly useless. Useless in that it has no effect on developing their interlanguage. Maybe they get some knowledge of English, briefly, but it has no impact on them as a student, or young Japanese person. What are they to do with a canned conversation about ordering a pizza? No, the tasks given to students in textbooks are not helping to develop students’ interlanguage, or in other words: the L2 version of those students.

From my own experiences as an L2 learner, I used to practice developing my L2 and L2 identity before sleeping. I would lie in bed and try to carry out a conversation in Japanese in my head… Now, not all of the students in my class, in fact, probably none of the students in my classes are prepared to go to that level in terms of their language development, but whilst they are taking my class and participating in oral activities in my class, I’d like them to exercise their communicative competence and practice speaking “without training wheels.

So that’s the main part: Learning about themselves as an L2 speaker, and fostering an improvement in their ability based on structured noticing activities, repeated gameplay sessions, and exposure to native speakers.

But is this “authentic” enough? I’ve been in talks with my friend and colleague Jonathan deHaan over the last few weeks and as always he is keeping me on my toes and pointing at the elephant ??‍? ?? ?. In Jonathan’s context, he has students work towards contributing something to or participating in wider society. As an explicit example, he is currently teaching a class that is specifically about connected learning. I.e., each student will choose something they are interested in, research it, and participate or engage in conversation with people in the wider community around that topic.

The participatory nature of KR is (mostly) missing. It’s a bubble of English practice that exists within the classroom for 100mins a week. This has somecomplexcomplications in terms of the methodological underpinnings of the class. For example, is it really TBLT if it has no “real world” component?

Anyway, focusing on how I can extend their language use outside of the classroom, I am currently trying to do this with the final project, where learners have to produce something of value to be used by the next generation of students. However, I give them an “out” allowing them to decline from making their materials available (via a consent form in the KR textbook). Things students could do as final projects and to increase participation (after seeking ethical approval from TDU of course):

  • Post groups’ videos as public on YouTube
  • BGG review posts or Amazon reviews instead of word documents handed into me
  • Teach and play games with other TDU students outside of the classroom (in an all-encompassing York Game Lab “end of term” gaming event

Playing devil’s advocate

Let’s put it out there:

The activities that students do as part of my course are equally as unimportant as those that they would do with a random textbook, with a random, uninterested, uninformed language teacher.

I don’t want to agree to this, as I have explored how KR can help motivate, re-engage and create a student-centred, productive class (see York & deHaan, 2018). However, I have not measured any learning goals for the course yet. I have nothing to prove the “effectiveness” of the course in helping learners to develop their interlanguage and proficiency in English. So I can’t cling doggedly to the idea that KR is better in some way to the textbook-based class that I taught in the past.

Quite a depressing thought.

How to evaluate students with the KR model…

There have been some problems this semester, and I’m glad they came up because it has given me a lot to think about regarding the next iteration of the model.

The major problem was that I found a group of students during the Spyfall replay class quite unashamedly playing the game in Japanese. All Japanese. They were quick to tell me that they had finished playing in English and were now “killing time” by playing in Japanese. During class time. When the (my) aim of the class was to get the students to a level of proficiency so that they could play in English. My initial reaction was “OK, whatever. Please do what you want,” but as I was walking to the next group which happened to be in the same room, I heard some of the questions that they were asking each other in Japanese. These questions were very basic. They were almost exactly the same as the ones that we had been studying in the first Spyfall class… It was this point that made me react:

  • Why not just play in English?
  • What a waste of a good opportunity!
  • Why would they throw away all of the hard work that they had been doing?
  • What do they not understand about the aim of the class?

A multitude of thoughts flew through my brain which ended with me in a rage.

The biggest shock of the class was discovering how little my goals are being adhered to by students. Like, completely ignored. Or thought of as frivolous? My aim for the class is to gradually build up their speaking skills so that they can use English authentically, without the training wheels (as mentioned above). Now, in this class, the domain prescribed for “authentic” usage just so happens to be gameplay. Is that the problem? By situating the domain for language use inside and around gaming–something that is often thought of as frivolous–maybe that causes students to not take the class seriously. In sum:

It’s all fun and games when using games.

Ba dum tiss!

I provide tools for learners to do serious analysis of their language ability, learn new grammar, teach others, reflect on their learning, improve as an L2 speaker… but they do the bare minimum. What is the problem? By not putting any whips or hurdles (tests/formative assessments) in front of them, do they have no fear of failure and thus no willingness to even try? This is another serious concern.

I threw the whole group out of the class that day and I ended up in the dumps for the whole week. But, with the help of my Japan Game Lab colleagues and a long chat with a former student at TDU, I feel like I am getting somewhere with the “why” and “how to prevent” such behaviour in the future, and it all leads to assessment. Assessment is not an aspect of SLA or education that has really piqued my interest before, and thus I have very little knowledge of how best to design rubrics or evaluate different skills. This is going to be a new area for me, but something that is important to consider for the future of KR.

The framework has a very rudimentary assessment criterion built in as it stands. I take in all worksheets from students and assess them based on how much work they did. A simple “has done” // “has not done” and then plus or minus a few points based on the level in which I feel the work has been completed. However, this assessment criteria is not told to the students so they have no idea. Additionally, I do not assess students in this way until the end of the semester after collecting their work in so they cannot see a “running total.” I think there are a number of areas where assessment could be built into the course, improving the transparency of grading. For example:

  • Learn
    • Explicitly state how many verbs/nouns/adjectives to find.
    • How many questions did they write?
  • Play
    • Total number of transcribed lines.
  • Analyze
    • How many questions did they get correct on the grammar quiz?
  • Replay
    • Total number of transcribed lines.
  • Reanalyze
  • Report
    • Complete a test which focuses on:
      • Game rules
      • Game vocab
      • Typical phrases used in the game
      • Transferability: such as demonstrating how they can use specific grammar points outside of the game context.
    • Write a written review of the games that they played.

      Going forward into the second semester

In conclusion, then, the biggest thing I would like to change is the final “participatory” goals of KR. It could be a proverbial stick, threatening students into taking responsibility for their learning and progression of the class if they know that they have to produce a public artefact at the end of the course. But we’ll see… I have touched on some ideas for wider participatory projects, and will continue to work at this over the next few months. Whether I will have time to implement them this semester, I’m not sure, but I will try!


If you read this far and have any comments, please let me know. I’m very eager to hear opinions on what I am (trying) to do in my teaching context. Also, if you have tried to implement anything that you have read here on the JGL blog, please let us know. Yes, we are named the Japan Game Lab, but we are interested in English teaching as a global phenomenon and practice.

Thanks as always,

James

References

  • Freebody, P., & Luke, A. (1990). Literacies programs: Debates and demands in cultural context. Prospect: An Australian Journal of TESOL, 5(3), 7-16.
  • Sockett, G. (2014). The online informal learning of English. New York, NY: Palgrave Macmillan.
  • York, J., & DeHaan, J. (2018). A constructivist approach to game-based language learning: Student perceptions in a beginner-level EFL context. International Journal of Game-Based Learning, 8(1), 19–40. http://doi.org/10.4018/IJGBL.2018010102

Reflection on the post-play “Analysis” lesson

First up, this is the order of activities that students complete as part of this lesson:

  1. Check English errors
  2. Translate JP to EN
  3. Presentations
  4. YouTube viewing

Set up

Before this class, students played a game and recorded the audio during the play session. For homework (or for the time remaining in the previous class) they collectively transcribed the audio and wrote down how many times each member spoke English in a tally.

Checking errors

The first activity of the class is for students to collectively look at their transcriptions for any mistakes they made in English. Of course, it is not obvious to them when they are making mistakes and I often hear the lamentation 「間違えているか分からないよー」as I walk around. However, they can easily pick up on the fact that they are mostly interacting by using single words (Open. Who? Werewolf. This. You? Next.).

I walk around the class looking at students transcriptions, marking areas that I think they should explore further.

Today I pointed out the following expressions as areas to look into:

  • I want you to…
  • Because / because of / so
  • Should / shall
  • He said / who said…?
  • There is / there is no…
  • There’s a good chance…
  • “a” and “the”

The students also realised that they used a lot of Japanese unnecessarily. Things like ‘そうだね’ or ‘これとこれ’ really shouldn’t be being said in Japanese. It’s not like they don’t know how to say such simple things in English, they just didn’t realise that they were using Japanese until they listened back to the audio and made the transcriptions.

Taking the game ‘Insider’ in particular, the group noticed that the main part of the game occurs after thekeyword has been guessed correctly. They also realised that this was the stage where they pretty much only used Japanese. However, I told them that it was not a problem. Certainly not a negative. The fact that they transcribed all of their Japanese discussion for use in today’s class is great. Why? Because it gives them the opportunity to think about how they could do the same discussion in English. I also told them that some students don’t transcribe the Japanese parts they spoke because they think it’s ‘against the rules’ to speak Japanese or that it is not important for the post-play analysis. Realising that the L1 can be a launchpad to thinking about the L2 is a great step here. That is the purpose of this lesson: to critically evaluate their previous performance and to collectively try and improve their knowledge for the next play session.

Presentations

Each class does the same third stage, which is to present any interesting things they found during the first and second stages. Using the whiteboards that they have available to them, they write out a few English grammar points and Japanese to English translations.

But why present their findings?

I think there are two positive outcomes of presenting.

  1. They reinforce their knowledge of items,
  2. They provide other groups with useful expressions that may not have come up in their own discussions.

I want to write a few thoughts regarding 2) above by referencing what happened in today’s class.

One group (the Resistance: Avalon group) introduced the expression ‘I want you to ….’ I then asked other groups in the class how they could use the same structure in their own games. The Captain Sonar group realised very quickly that the engineer could use this structure. Another group introduced ‘Shall we….’ and ‘You should’ which was picked up by the Avalon group as something they could use when deciding who should be picked to go on a quest. This continued until all groups had introduced some useful expression, phrase, or word that could be used in their games.

The point here is that although the games are different, there is great learning potential in getting students to share their findings because other groups may benefit from incorporating the expressions into their own context. It’s impossible for me to give precise grammar instruction to all groups (or at least, it would take [number of groups] x 90 minutes), so for them to instruct each other is a great compromise, and the laidback, community-based environment is one that I think the students enjoy. It is a real pleasure to see.

Issues:

A criticism of this activity is that students are generally only looking up one-to-one translations for expressions meaning that instead of introducing structures (as shown above) they will introduce extremely specific sentences translated from Japanese, or just one to one lexical items such as ‘当たった is Hit! in English.’ My next goal is to get them looking more deeply at the grammar behind what they want to say.

YouTube viewing session

This is possibly the most difficult part of the class for the students and as a result, very difficult for me to provide guidance. Here’s why:

Issues

  • YouTube auto-translations are incredibly unreliable.
  • Videos sometimes don’t exist (like for Insider)
  • Videos with subtitles are rare.
  • Native speakers talk too fast and interrupt each other a lot which means that the listening activity is too advanced for the students
  • Students don’t take time to pause and reflect on what they are hearing.

However, it is not all doom and gloom(haven).

Although there are not many games that feature subtitles on YouTube, some generous, kind and wonderful people do include subtitles. Subtitles can be accessed by pressing “filter” on search results:

I also had a really interesting experience last week doing this class. One group played Mafia de Cuba, and so watched a YouTube video of native speakers playing the game.

This video:

What was so amazing was that having played the game themselves the previous week, watching the video was a real pleasure for the students. They got to see other people’s tactics, see how the game unfolded, and were enthralled in watching. We spoke about it afterwards and came to the conclusion that because they were deeply interested in the content, and wanted to know how the game would conclude, it didn’t feel like traditional “English class video-watching.” The group were instructed to rewatch the video more critically after the first viewing to understand what was being said, and look for expressions that they could use in their own gameplay.

Therefore, students do pick up expressions from the videos, write them down, and finally share what they found with their group, further increasing their potential repertoire for the next class. However, this whole section needs more work.

Final ideas

Possible ways to improve the class:

  • Provide common grammar error handouts to complete for homework

Such handouts could be created by myself, or the students could be pointed to specific websites to study grammar. The only issue I see here is that, unless they are really motivated to learn the grammar, I can’t see a lot of students being excited to do this activity.

  • Dedicate time to explain how to look for grammar guides online.

As mentioned, Students just look at Google Translate for one-to-one translations, not websites that explain the details behind the how and why of English grammar. I think it would be beneficial for students to learn how to use digital tools to help them acquire correct grammar rules during class time. I also think they should be expected to explain grammar rules to other groups as part of the their presentations.

Possible extensions to the class:

  • Quiz each other at the start of the following class before replaying the game.
  • I could make a short test based on their findings and make a more structured quiz also.

As always, thanks for reading. Comments welcome.