Lessons in Language Assessment

For the last 3 weeks I have been auditing Prof. Hossein Farhady’s Language Assessment course given as a part of Yeditepe University PhD program in English Language Teaching. Though I have finished taking classes and I am on the verge of writing the proposal of my doctorate thesis, I still enjoy participating in Prof. Farhady’s class for 4 hours on Thursday evenings; asking and answering questions, reading articles and books and reflecting on issues related to fundamental concepts and principles of second language assessment, with a lovely group of classmates.

The professor is also my thesis advisor and I believe that our Thursday classes and discussions will help me to develop a critical view on a variety of existing assessment procedures, establish a better understanding of fair practice, forms, functions, uses, and psychometric characteristics of language assessment procedures, paving the way to my future thesis.

“Surrender is easy but don’t”

Thought provoking questions are flagged, real life scenarios are suggested and Prof. Farhady often plays the devil’s advocate when he corners us with his questions, requiring us to analyse the course content and to screen it against our experience as teachers who give tests to their students. When coming with an intelligent and satisfying answer becomes hard and he sees the question marks in our eyes, he says: “surrender is easy but don’t”. So, we promise him that we will keep our discussions in mind and always have a critical eye to our practices and pursue validation and reliability. After all changing the world starts with changing yourself, isn’t it?

This week on our agenda we had assessing speaking and writing and I would like to reflect on lessons to be taken for me.

Assessing Speaking

Do you have a speaking test in your institution? I think that designing a speaking test, coming up with tasks to be used in the test and devising a scoring scale for the test-takers’ performance requires a lot of hard work. In “Assessing Speaking” Sari Luoma (2004) suggests Hymes’s (1972) SPEAKING Framework to make the initial planning of a speaking test.

• Situation (Consideration of physical setting and nature of the test- Is it an end of term test of speaking?)
• Participants (How many examinees to take the test? Will they work in pair work? Group work? What would be the specifications about interlocutor and assessor?)
• Ends (considerations about the outcomes of the test involving formative or summative use, how to provide feedback, test score and fair assessments)
• Act Sequence (the form and content of speech acts that will be elicited through the test)
• Key (How examiners are supposed to conduct their act and presence in assessment situations: Any scripts that will accompany, assessors guide regarding how supportive or impersonal they need to be?)
• Instrumentalities (Which channels or modes (spoken, written, pre-recorded)  and forms of speech (dialects, accents and varieties) will be used?)
• Norms (Which norms of interaction, such as initiating conversation, asking clarification questions, elaborating, and (dis)agreeing, will be involved in the test?)
• Genre

This framework can help design of a speaking test because it raises questions about linguistic, physical, psychological and social dimensions of the situation in which language is used. Consequently, task designer has to take input, goals, roles and settings into consideration.

Also, Prof. Farhady presented types of assessing speaking below:

• Imitative (focus on repetition and pronunciation. E.G. Phone Pass test 
• Intensive(production of controlled language use and short phrases via minimum interaction)
• Responsive (interacting to short conversations)
• Interactive (transactional and interpersonal)
• Extensive (oral presentations, story telling…)

He stressed that differentiating and understanding these types will help us gear our speaking test to better cater for the needs of our students.

Types of speaking tasks

Luoma (2004) provided a comprehensive summary of what speakers are asked to do in assessment situations. According to Brown and Yule types of informational talk encompass; description, instruction, story-telling and opinion expressing/justification (cited in Luoma, 2004, p.31). Bygate differentiates speaking tasks into factually oriented (description, narration, instruction, comparison) and Evaluative Talk (explanation, justification, prediction and decision). In addition to informational talk, there are also communicative speaking tasks. Common European Framework (2001) divided functional competence into Macrofunctions (description, narration, commentary, explanation, and demonstration) and Microfunctions (giving and asking for factual information, expressing and asking about attitudes, suasion-suggesting, requesting, warning-, socialising, structuring discourse and communication repair). There is a variety of task types that could be used in assessing speaking. Then how can task designers for a speaking exam decide which one(s) to use? Luoma (2004) suggests that task designers should make the organising principle for the assessment and teaching curriculum coherent (p.35).

Other considerations when designing speaking assessment tasks

This week in our testing class we once more saw that task designer’s burden is heavy. In addition to types of talks and communicative functions, they need to plan about how to operationalize these tasks. They need to rationalise whether individual, pair or group tasks will be used. Also assessment developers will choose whether to use real-life or pedagogical tasks, tape-based or live testing and determine between use of construct-based and task-based assessment. They also need to manipulate the difficulty of speaking task with regards to complexity of task materials, task familiarity, cognitive complexity and planning time. (Luoma, 2004, p.46)

Examples of Speaking Scales

One of the highlights of this week’s classes was having the chance to discuss a variety of both analytical and holistic speaking scales examples as well as rating checklists. Luoma outlined;

• The Finnish National Certificate Scale
• The American Council for teaching of foreign languages (ACTFL)
• The Test of Spoken English Scale
• The Common European Framework speaking scales
• The Melbourne medical students’ diagnostic speaking scales (2004, p.60)

Metamorphosis; Lessons to be taken

At the end of each week I reflect on our class discussions and ask myself; “How will your future conduct change?”. Here are points to keep in mind for me to change for the better:

It’s important to prepare various versions of speaking scoring rubrics and scales catering for the needs of raters, teachers and examinees.

Holistic and analytical scales have their pros and cons and therefore, their use should be considered carefully. Holistic ones can be accompanied with rating checklists (detailed lists of features describing successful performances on task) for feedback purposes.

To develop good and clear level descriptors stems from examining performances of test-takers from different levels and describe features that makes them a certain level.

Differences between levels should be clear on the speaking scales and should not be blurred with too much dependence on quantifiers such as: many, few, adequately…etc.

I feel that being able to talk about questions in mind, assessment related issues we encounter in real-life and hearing about different perspectives and settings enrich my personal understanding regarding assessment. I really learn a lot…

Thursday testing classes and reflections will continue. Please stay tuned 🙂

Reference: Luoma, S. (2004). Assessing Speaking.The Cambridge Language Assessment Series, CUP, Edinburgh.