Designing Courses & Assessment: Principles in Second Language Assessment

Designing Courses & Assessment: Principles in Second Language Assessment

Document Author: Sam Barclay

Learning Objectives

• Consider different purposes of assessment.
• Consider validity as a multifaceted construct.
• Consider reliability and factors that can impact it.
• Apply these concepts to your own teaching context.

Activity: Associations

• Brainstorm words associated with the terms "Teaching," "Testing," and "Testing/Assessment."

Speaking Test Comparison

Why Assess?

• List various purposes for assessment in language learning.

Focus on Purpose

• Why is the purpose of a test important?
• How does purpose relate to:
  • Decisions you want to make?
  • Information the test needs to provide?
  • The design of the test itself?
• Key Point: Purpose is connected to the representatives of the assessment task, meaning it should reflect real-world language use in the target domain.

Decision-Making with Assessment

• Differentiate between high-stakes, low-stakes, and no-stakes assessments.
• Provide examples of each.

Connecting Assessment to Learning Objectives

• Effective Assessment: Aligns with learning goals and objectives.
  • Learning Objectives define what students should learn.
  • Assessment measures what students have learned.
  • Instruction guides students' learning activities.

When to Assess?

• Assessment can occur at various points throughout a course, not just at the end.
• Consider alignment with course, unit, and lesson objectives.

Assessment Design

Qualities of a Good Assessment

• Evaluation: Goes beyond simply measuring performance to provide feedback for improvement.
• Fairness: Ensures all students have an equal opportunity to demonstrate their abilities.
• Standardization: Maintains consistent administration and scoring procedures.

Designing an Assessment

• Steps involved in assessment design:
• Specifications
• Item writing & review
• Piloting & review
• Implementation
• Post-implementation review

PURPOSE CONNECTED TO REPRESENTATIVENESS 

Validity: A Multifaceted Construct

• Validity: Refers to the extent a test actually measures what it claims to measure.
• Key Points:
  • Validity is multifaceted, with different types of evidence needed to support a test's validity.
  • Validity is not inherent to a test, but rather depends on how it is used and for what purpose.
  • Validity resides in the scores of a particular test administration, not the test itself.

Types of Validity

• Content/curriculum/context validity: Assesses the relationship between the test content and the target language use domain.
• Predictive validity: Measures how well test scores predict future performance in the target language domain.
• Cognitive validity: Examines if the cognitive processes involved in tackling a task match the intended purpose.
• Face validity: Pertains to the appearance of validity, ensuring tasks seem relevant to stakeholders.
• Consequential validity (washback): Evaluates the social and institutional impact of a test on learners and teachers.

Building a Validity Argument

• To demonstrate a test's validity, construct an argument.
  • Identify claims made by the test.
  • Gather evidence to support or refute those claims.
  • Recognize that changes in population or usage may require a new validity argument.

Reliability/Scoring Validity

• Reliability: Refers to the consistency and dependability of test results.
• Key Points:
  • Every test has a standard measurement error.
  • Reliability is impacted by test factors (sampling, item quality, etc.), situational factors (time of day, etc.), and individual factors (test-taker characteristics, raters' consistency, etc.).

Factors Affecting Reliability

• Consider how the following factors can impact reliability:
  • Test factors (number of items, task transparency, etc.)
  • Situational factors (test environment, time of day, etc.)
  • Individual factors (test-taker anxiety, rater bias, etc.)

Example: Analyzing Speaking Tests

• Analyze three descriptions of speaking tests from a reliability and validity perspective.
• Consider factors like face validity, content validity, task types, number of raters, and rater training.

Further Reading

• Jang (2014) - Accessible text on reliability.
• Weir (2005) - In-depth discussion of validity.
• Green (2014) - Clear introduction to reliability and validity.
• Urquhart & Weir (1998) - Example of construct definition (reading)

Designing Courses and Assessments: Session Two - Needs Analysis

Designing Courses and Assessments: Session Two - Needs Analysis

This document outlines the importance of Needs Analysis (NA) in course design and explores different approaches to conducting one.

Learning Objectives

By the end of this session, you will have considered:

• The definition and purpose of Needs Analysis (NA).
• Why NA is important for course design.
• The different components of a thorough NA.
• Who should be included in the NA process.
• How to identify and address constraints.
• Various tools used to gather information for NA.
• Examples of Needs Analysis in action.
• Your own needs as a learner and how they influence course objectives.

Getting Started

Designing Courses and Assessments: Session Two - Needs Analysis

• What is Needs Analysis (NA)? Consider working in pairs to develop a definition.

Definitions

• Graves (2000): NA is a systematic and ongoing process for gathering information about learner needs and preferences, interpreting the data, and using it to make informed course decisions. It acknowledges that needs are multifaceted and can change over time.
• Brown (2009): NA is the systematic collection and analysis of all information necessary to define a defensible curriculum. This includes both subjective and objective information relevant to the specific context and stakeholders involved.

Importance of Needs Analysis

• Why is NA important?

• • Ensures courses are relevant and engaging for learners, leading to higher motivation.
  • Informs course design to ensure learners achieve their specific goals.
  • Provides the foundation for developing learning objectives, teaching materials, activities, assessments, and evaluation strategies.
  • Helps identify the gap between learners' current skills and their target goals.
  • Informs potential curriculum changes and pinpoints existing problems learners face.

Target Situation Analysis (TSA)

• A key component of NA is the Target Situation Analysis (TSA).
• It focuses on the communicative needs learners will encounter in their target situation (e.g., vocabulary, grammar, discourse skills).
• While a TSA is crucial, it might not be enough for a defensible course. Additional information about learner preferences and backgrounds may also be necessary.

Designing Courses and Assessments: Session Two - Needs Analysis

Target Needs vs. Learning Needs

• Target Needs: What learners need to be able to do in the target situation (e.g., writing business letters, using specific vocabulary).
• Learning Needs: What learners need to do to acquire the target skills (e.g., grammar practice, vocabulary building strategies).

Understanding Needs

• Target Needs:

  • Necessities (essential skills needed, e.g., writing business letters).
  • Lacks (areas where learners need improvement).
  • Wants (learner desires and motivations).
  • Required knowledge.
  • Present knowledge.
  • Subjective needs.

• Learning Needs:

  • Preferences of different stakeholders (learners, teachers, administrators).
  • Reasons for taking/running the course (compulsory vs. optional).
  • Learner goals and expectations.
  • Preferred learning styles (teachers and learners).
  • Learner backgrounds and beliefs about teaching and learning.
  • Preferred methodologies.

Process of Needs Analysis

There are different models for conducting a Needs Analysis. Here are two examples:

Graves (2000):

1. Decide what information to gather and why.
2. Determine the best way to gather it (when, how, and from whom).
3. Collect information.
4. Interpret the information.
5. Act on the information (make course decisions).
6. Evaluate the effectiveness of those decisions (potentially leading back to step 1).

Brown (2009):

1. Define the purpose of the NA.
2. Identify the target student population.
3. Choose approaches and syllabuses.
4. Identify constraints.
5. Select data collection procedures.
6. Analyze the data.
7. Interpret the results.
8. Determine course objectives.
9. Evaluate and report on the NA project.

Data Analysis: Qualitative vs. Quantitative Approaches

Most NA approaches are primarily qualitative (e.g., interviews, journals), but quantitative data (e.g., questionnaires) can also be valuable.

Triangulation

Triangulation is the process of using multiple data sources and methods to increase the credibility of your analysis. This doesn't just mean using more methods; it involves choosing methods that complement each other and reveal potential misalignment in perceived needs.

Sources of Information for Needs Analysis

• Insiders: Students, teachers, course leaders, administrators, sponsors, governments, etc. (including artefacts like student work or policy documents).
• Outsiders: Literature, applied linguists, domain experts

Adapting Coursebooks: Moving Beyond Coursebooks Sarah Khan, MA TESOL DCA Session 6

Adapting Coursebooks: Moving Beyond Coursebooks

Sarah Khan, MA TESOL

DCA Session 6

Aims of this Session

In this session we will:

 Discuss the reasons why coursebook adaptation may sometimes be required

•  Analyze different types of adaptations
•  Consider some examples of adaptation
•  Consider use of authentic materials in English language teaching
•  Explore creative ways of using low cost or no cost resources to teach English
•  Discuss the use of available technology to teach English

 Reasons for Adapting a Coursebook

Why Adapt a Coursebook?

1.  There is too much/little material for time constraints.
2.  Unsuitable for age/proficiency of learners.
3.  Activity may require skills/knowledge that learners do not have.
4.  Activities do not meet learner needs (linguistic, skills, content, strategies).
5.  Does not apply important principles from applied linguistics (e.g. recycling).
6.  You want to involve the learners in the curriculum design process.
7.  There are important aspects of language learning not covered.

 Other Reasons for Adapting a Coursebook

Littlejohn (2010):

 There is too much/little material for time constraints.

 Unsuitable for age/proficiency of learners.

•  Activity may require skills/knowledge that learners do not have.
•  Activities do not meet learner needs (linguistic, skills, content, strategies).
•  Does not apply important principles from applied linguistics (e.g. recycling).
•  You want to involve the learners in the curriculum design process.
•  There are important aspects of language learning not covered.

Basically, we adapt for two broad reasons:

•  To better motivate learning.
•  To improve the effectiveness of the materials/course.

To Fill Gaps in the Content:  

Nation’s Four Strands (2009):

•  Form focused Instruction
•  Meaning focused Input
•  Fluency Development
•  Meaning focused Output

The syllabus The textbook:  

Incomplete coverage of course goals and objectives.

• Take account of how people learn/SLA theory:  
• Evaluate materials in relation to how the field of TESOL and education understands how learners:
•  Become more skillful readers, speakers, listeners, writers.
•  Acquire linguistic competence, grammar, vocabulary, pronunciation, pragmatic competence, discourse competence, strategic competence.

• Partial overlap with an examination that is developed separately from the course and textbook:  
•  Mismatch between what students can do at the start of a lesson and the starting expectation of the texts and tasks they meet in the lesson.  
•  Scaffolding may be needed.

Mismatch between textbook and the following:

 Teachers’ familiarity or expertise with methodology of the textbook.

 Teachers and/or students beliefs about language learning.

 Local educational culture.

Issues relating to the following:

•  Class size
•  Classroom layout
•  Lesson length (longer or shorter)
•  Opportunities for practice
•  Cultural appropriacy of lesson topics

Scale of Adaptation (How much can be adapted?)

Activity level Adaptation:  

Changes are made to an activity. The rest of the class, unit, and syllabus remains intact.

 E.g., modifying the vocabulary and grammar of a text/changing an output activity to a 4/3/2 to promote fluency development.

Lesson level Adaptation:  

Changing content of a whole lesson. This implies changes to activities as well, but units and syllabus remain intact.

 E.g., adapting a reading session to promote better coverage of microskills and MFO (Rosenkjar, 2009).

Unit level Adaptation:  

Changes are made to all the lessons in a unit. The syllabus remains intact.

 E.g., changing the content of a unit because it is not appropriate (culturally, etc.) for a specific learner group.

Syllabus level Adaptation:  

Changes to the syllabus as a whole. This will impact units, lessons, and activities.

 E.g., adding a meaning focused input component (e.g., extensive reading) to a syllabus, to ensure balance and meet all the needs of the learners.

 Four Key Questions

•  What is the objective of the activity?
•  What does the exercise actually get the learner to do/learn?
•  What do I want the learner to do/learn?
•  How can I get the exercise to do what I want it to do for the learner?

Important Questions to Ask:

•  Is there sufficient instructional content to meet the learning outcomes?
•  Is there sufficient input to help learners develop linguistic knowledge?
•  Is there sufficient practice? Controlled/Free?
•  Are there opportunities for students to use language in new contexts?
•  Have the subskills for a larger task been taught?
•  Do the materials offer sufficient challenge?
•  Is there sufficient variety to maintain motivation?
•  Are the activities structured to allow differentiation?

Considerations

• What considerations should you keep in mind when adapting?
•  Are your changes quantitative (more) or qualitative (the nature of the task)?
•  Does your adaptation change the intended learning outcome? Is this okay?
•  How does your change affect the overall coherence of a lesson or the course?
•  Are there any unintended consequences of your change?
•  Will your change negatively affect a future lesson?

Terminology in Materials Adaptation

Localisation:  

Graves (2000) – A textbook is written for everyone and it also written for no one. You must adapt textbooks so they meet the needs of the learners in your specific context.

Prabhu (1989)  Because teaching must be matched to the learners’ current knowledge, course books will not be totally effective as they are not in touch with the state of this knowledge.

Supplementation:  

Find the gap (syllabusdriven materials design)  

I need something to practice X, where X is a point in your syllabus that is either not covered in the course book or is not dealt with adequately…If you know something readymade that will fit the bill, the problem is quickly solved. If not, you will need to design material specially for that purpose.  

McGrath, 2016

Find the Gap:

•  Define the gap carefully.
•  Find language examples to use.
•  Produce activities to exploit language examples.
•  Produce materials to present activities to learners.

Adding or Omitting Content:

 Add content for extra practice for important/difficult items.

 Add content to enhance understanding of other content.

 Add content to match state of the art in applied linguistics.

 Omit confusing/unimportant parts.

 Omit culturally inappropriate parts.

 Extend an activity so that it meets other learning outcomes (e.g., add MFO to MFI).

Change the Sequencing of the Content:

 Change the order of the content to help learner motivation.

 Change the order of the content to help comprehension.

 Change the order of the content to help learning.

Change the Format:

 Change a speaking activity to a writing activity.

 Change a reading activity to a reading while listening activity.

 Change an individual task to a group task.

 Change the nature of the task – a dialogue 🡪 fluency focused event.

Change the Presentation:

 Change a paperbased activity to a computerbased activity.

 Change a listening activity to a viewing activity.

 Change examples from the familiar to the unfamiliar/known to the unknown (and vice versa).

Add or Omit Monitoring:

 Add monitoring opportunities to check student progress.

 Add tests to encourage students to do homework.

 Omit tests to reduce pressure on learners.

 Change a summative assessment to a formative assessment.

 Key Questions about Your Chosen Book/Materials

 Does it suit the environment in which it will be used (localisation)?

 Does it meet the needs of the learners (personalisation)?

 Does it apply sensible principles of teaching/learning?

 Do its goals match the goals of the course?

 Does the content match the proficiency level of the learners? (individualisation/simplification)

 Is it interesting and engaging (personalisation)?

 Does it include mechanisms for monitoring student progress?

 Are the units an appropriate length to match the environmental constraints?

 Does it include any behavior/topics that might offend the learners?

 Are the materials up to date (modernisation)?

Examples of Adaptation

Example 1: Syllabus Level Adaptation

Example 2: Lesson Level Adaptation

Example 3: Lesson Level Adaptation

Using Authentic Materials

Moving Beyond Coursebooks:

Conceptdriven Materials Design  

Discussion 1: Teaching Without Coursebooks  

Using authentic materials  

How can you teach with everyday simple materials that you or your students may bring to class?

Choose an everyday object and discuss your object/idea with the people on your table and make a list of educational and language points that you can teach using that object.

Syllabus Design (Graves, 2014):

 Learners

 Purposes for Learning

 Principles of Language Teaching & Learning

 Resources

 Contextual Constraints

Principles of Language Teaching (Tomlinson, 2010: 9597):

 The content and methodology of the teaching should be consistent with the objectives of the course and should meet the needs and wants of the learners.

 The teaching should be designed to help learners to achieve language development and not just language acquisition.

 The teaching should be designed so as to provide the learners with learning opportunities that will help them to develop educationally in the sense that they become more mature, more critically astute, more creative, more constructive, more collaborative, more capable, and more confident as result of the course.

 The teacher needs to be able to personalize and localize the materials and to relate them in different ways to the needs, wants, and learningstyle preferences of individual learners.

Discussion 2: Working with Limited Resources:  

Imagine

that you have been asked to teach in a context without access to coursebooks, internet, audio/video, and photocopying equipment. How might you approach the following?

Choose one of the following prompts and discuss:

1. Pronunciation practice

2. Group projects (focusing on content and language)

3. Speaking activities

4. Writing activities

You may continue to focus on the materials you considered in Discussion Task 1. Use the Padlet link to share your materials and language focus.

[https://padlet.com/sarah_khan/authenticmaterialsforenglishlanguageteachingandlearni40oe4l85gszz2uuh](https://padlet.com/sarah_khan/authenticmaterialsforenglishlanguageteachingandlearni40oe4l85gszz2uuh)

Moving Beyond Coursebooks

Discussion 3: Exploiting Technology

In your groups, discuss:

 What technological devices and facilities are available in the context you have selected for your course?

 What kinds of online tools can you use for language teaching?

 Would there be any difficulties in accessing online learning?

Please discuss and make a list.

Teachers Using Their Own Materials

"I’m more encouraged to teach when I use my own materials … it carries with it my personality, my beliefs about language learning and language teaching … preparing my own materials is of course more convenient, because then I get to do reviews and changes … if I make it myself, it’s more, it’s easier to navigate around." [T2]

Tarrayo & Anudin (2021)

Next Steps

At this stage, you should have:

•  Your goals and objectives clearly written out.
•  Started evaluating books to select a suitable coursebook for your course (if applicable).

Now you should:

 Create a content map for your course and indicate which units from the book you will use in your course (if using a book). Specify which units will be used in which weeks in the course.

 Analyze the units in the coursebook that you wish to use and identify some opportunities for adaptation.

 Indicate adaptations in your course map.

 Further Reading

1.  Copland, F. and Mann, S. (2012). The Coursebook and Beyond: Choosing, using and teaching outside the text. Tokyo: Abax. (especially chapter 5)
2.  Littlejohn, A. (2010). The analysis of language teaching materials: inside the Trojan Horse. In Tomlinson, B. Materials Development in Language Teaching (2nd edition). Cambridge: Cambridge University Press. pp. 179211
3.  Meddings, L. and Thornbury, S. (2009). Teaching Unplugged: Dogme in English Language Teaching. Peaslake, Surry: Delta Publishing.
4.  Tarrayo, V. N. & Anudin, A. G. (2021). Materials development in flexible learning amid the pandemic: perspectives from English language teachers in a Philippine state university. Innovation in Language Learning and Teaching. DOI: 10.1080/17501229.2021.1939703
5.  Tomlinson, B. (2010). Materials Development in Language Teaching (2nd edition). Cambridge: Cambridge University Press.

This session provides a comprehensive overview of the need for and methods of adapting coursebooks to better suit the diverse needs of learners. By considering the principles of language teaching, exploring the use of authentic materials, and addressing technological and resource constraints, educators can enhance the effectiveness and engagement of their English language teaching practices.

Coursebook Selection and Sequencing in the Curriculum

Coursebook Selection and Sequencing in the Curriculum

Recap

Coursebook Selection and Sequencing in the Curriculum

Take 10 minutes to share your course aims and objectives with your peers. Ensure that:

Your aims are clear and general (not vague or too broad)

Your objectives are SMART

Use this opportunity to clarify any uncertainties about your course.

Coursebook Selection and Sequencing

Aims of the Session

In this session, we aim to:

Consider different approaches to coursebook evaluation

Begin analyzing and discussing the content map in different coursebooks

Consider criteria for conceptualizing content (scope) what are you going to teach?

Consider sequencing of content in a syllabus in what order are you going to teach the content?

Analyze some ELT textbooks to notice content conceptualization and sequencing

Start developing a content map for your course

Process of Course Design

Materials in English Language Teaching

What types of materials can you use to teach English? Make a list with your group members.

Curriculum Planning

Language/linguistic content = input = what are we going to teach?

Process/Means = methodology in language teaching (related to teacher beliefs and the nature of language)

Output = Product = learning outcomes

Approaches to Coursebook Evaluation

1. Impressionistic Overview

 Layout: clear, attractive; quality of visuals; sequencing

 Presents a general introduction to the materials

2. The Checklist Method

 Checks for the presence of certain items.

3. Indepth Evaluation

 Content: how specific items about learners’ needs and syllabus requirements are dealt with, and how different aspects of language are addressed.

Consider combining all approaches to evaluate coursebooks.

Impressionistic Overview: Content Map in Coursebooks

When looking at the content map of a book, consider:

Linguistic level (graded materials)

Progression

Opportunities for practicing language skills

Interesting topics and activities

Motivating and inspiring materials

Recycling

Emphasis on fluency (through skills practice)

Emphasis on accuracy (through understanding of grammar)

Opportunities for individual study (to promote learner autonomy)

Promotes effective learning

The Checklist Method

Checking for the presence of certain aspects is:

Systematic

Cost effective (saves time)

Convenient format

Explicit (all categories are well understood)

Develop your own checklist. 'Borrow and adapt' (McGrath 2002). Cite sources appropriately.

The In depth Method

Detailed analysis of the coursebook and scrutiny of some units. This may involve piloting sample units that seem typical of the book.

Consider online analysis on publishers’ websites, discussions on professional boards, or asking publishers’ representatives.

CATALYST Test (Grant 1987)

Communicative

Aims

Teachability

Available addons

Level

Your impression

Student interest

Tried and tested

Perfect Coursebook?

Grant (1987:118) stated, “The perfect textbook does not exist, but the best book available for you and your students certainly does.” The textbook should:

Suit the needs, interests, and abilities of the learners

Suit the teachers

Meet the needs of the syllabus (including exams)

Coursebook Evaluation and Selection

When evaluating and selecting a coursebook for your course, consider:

What will be the content for your course? Why?

If using a coursebook, select one for your context and explain your choice.

How will you organize and sequence your content? Why?

Make a map of your course and show how you plan to use the coursebook (which units in which week).

Write at least 500 words to describe your scope and sequence plan.

Content: Scope and Sequencing

Making sensible, welljustified decisions about content is crucial in curriculum design. Poor content choices lead to poor learning outcomes, despite excellent teaching efforts. (Nation & Macalister 2009)

Conceptualizing Content

What do the students need?

What is the purpose of the course?

What content do you want your learners to use? (what to include and exclude)

Scope

Concerned with the breadth and depth of coverage of items in the course (Richards 2001)

What range of content will be covered?

To what extent should each topic be studied?

Sequencing within a Course

Common Criteria for Sequencing:

Simple to complex to improve learnability

Chronology (e.g., planning and brainstorming as the first stage of writing)

Prerequisite learning (e.g., simple sentence structure SVO for writing sentences)

Learner needs and interests

Recycling/spiral sequencing (language and skills)

Two Types of Sequence Design:

1. Linear

 Lessons depend on what came before

 Spiral: Provides easily monitored recycling of material, allowing learners who were left behind to catch up in the next cycle.

 Matrix: Elements selected from categories of content, but not necessarily in a predictable order.

 Revision Units: Include revision sessions within a linear syllabus.

2. Modular

 Selfcontained modules with specific objectives and assessment at the end of each module.

 Units: Shorter than a module, longer than a lesson, planned around a focus of instruction (also referred to as Scheme of Work).

Further Reading

Christison, M., & Murray, D. E. (2014). What English Language Teachers Need to Know Volume III: Designing Curriculum. Taylor and Francis.

Graves, K. (2000). Organizing the course. In Designing Language Courses: A Guide for Teachers. Heinle CENGAGE Learning.

Littlejohn, A. (2010). The analysis of language teaching materials: Inside the Trojan Horse. In B. Tomlinson (Ed.), Materials Development in Language Teaching (2nd ed.). Cambridge University Press.

Masuhara, H. (2010). What do teachers really want from coursebooks? In B. Tomlinson (Ed.), Materials Development in Language Teaching (2nd ed.). Cambridge University Press.

Nation, I. S. P., & Macalister, J. (2009). Goals, Content, and Sequencing. In Language Curriculum Design. Routledge.

Skela, J., & Burazer, L. (2021). The Architecture of ELT Coursebooks: The Internal Organization of Coursebook Units. Vestnik za tuje jezike, Vol.13 (1), 383407. DOI: https://doi.org/10.4312/vestnik.13.383407

Thank you!

Nottingham Trent University

Session 4: Incidental and Intentional Approaches to Vocabulary Instruction Vocabulary: Learning, Teaching & Assessing

Vocabulary: Learning, Teaching & Assessing 

Session 4: Incidental and Intentional Approaches to Vocabulary Instruction

By Sam Barclay