Designing Courses & Assessment: Principles in Second Language Assessment

Designing Courses & Assessment: Principles in Second Language Assessment

Document Author: Sam Barclay

Learning Objectives

• Consider different purposes of assessment.
• Consider validity as a multifaceted construct.
• Consider reliability and factors that can impact it.
• Apply these concepts to your own teaching context.

Activity: Associations

• Brainstorm words associated with the terms "Teaching," "Testing," and "Testing/Assessment."

Speaking Test Comparison

Why Assess?

• List various purposes for assessment in language learning.

Focus on Purpose

• Why is the purpose of a test important?
• How does purpose relate to:
  • Decisions you want to make?
  • Information the test needs to provide?
  • The design of the test itself?
• Key Point: Purpose is connected to the representatives of the assessment task, meaning it should reflect real-world language use in the target domain.

Decision-Making with Assessment

• Differentiate between high-stakes, low-stakes, and no-stakes assessments.
• Provide examples of each.

Connecting Assessment to Learning Objectives

• Effective Assessment: Aligns with learning goals and objectives.
  • Learning Objectives define what students should learn.
  • Assessment measures what students have learned.
  • Instruction guides students' learning activities.

When to Assess?

• Assessment can occur at various points throughout a course, not just at the end.
• Consider alignment with course, unit, and lesson objectives.

Assessment Design

Qualities of a Good Assessment

• Evaluation: Goes beyond simply measuring performance to provide feedback for improvement.
• Fairness: Ensures all students have an equal opportunity to demonstrate their abilities.
• Standardization: Maintains consistent administration and scoring procedures.

Designing an Assessment

• Steps involved in assessment design:
• Specifications
• Item writing & review
• Piloting & review
• Implementation
• Post-implementation review

PURPOSE CONNECTED TO REPRESENTATIVENESS 

Validity: A Multifaceted Construct

• Validity: Refers to the extent a test actually measures what it claims to measure.
• Key Points:
  • Validity is multifaceted, with different types of evidence needed to support a test's validity.
  • Validity is not inherent to a test, but rather depends on how it is used and for what purpose.
  • Validity resides in the scores of a particular test administration, not the test itself.

Types of Validity

• Content/curriculum/context validity: Assesses the relationship between the test content and the target language use domain.
• Predictive validity: Measures how well test scores predict future performance in the target language domain.
• Cognitive validity: Examines if the cognitive processes involved in tackling a task match the intended purpose.
• Face validity: Pertains to the appearance of validity, ensuring tasks seem relevant to stakeholders.
• Consequential validity (washback): Evaluates the social and institutional impact of a test on learners and teachers.

Building a Validity Argument

• To demonstrate a test's validity, construct an argument.
  • Identify claims made by the test.
  • Gather evidence to support or refute those claims.
  • Recognize that changes in population or usage may require a new validity argument.

Reliability/Scoring Validity

• Reliability: Refers to the consistency and dependability of test results.
• Key Points:
  • Every test has a standard measurement error.
  • Reliability is impacted by test factors (sampling, item quality, etc.), situational factors (time of day, etc.), and individual factors (test-taker characteristics, raters' consistency, etc.).

Factors Affecting Reliability

• Consider how the following factors can impact reliability:
  • Test factors (number of items, task transparency, etc.)
  • Situational factors (test environment, time of day, etc.)
  • Individual factors (test-taker anxiety, rater bias, etc.)

Example: Analyzing Speaking Tests

• Analyze three descriptions of speaking tests from a reliability and validity perspective.
• Consider factors like face validity, content validity, task types, number of raters, and rater training.

Further Reading

• Jang (2014) - Accessible text on reliability.
• Weir (2005) - In-depth discussion of validity.
• Green (2014) - Clear introduction to reliability and validity.
• Urquhart & Weir (1998) - Example of construct definition (reading)