presumed innocent until proven guilty. The (2) establishment of guilt is often a difficult task. One consideration is whether or not there remains a (3) reasonable ...
Vocabulary Assessment P. David Pearson University of California Berkeley
There is only one book in Books in Print with the title, Vocabulary Assessment • What field do you think it is in? – Reading – Oral Language Development – Intelligence Testing – English as a Foreign/Second Language
Special Characteristics of this presentation • Data free presentation • why? • So little new and/or interesting work in vocabulary assessment
Pretest in the form of a Jeopardy Game
Reading Jeopardy • Scientifically based reading research: • What phrase occurs in the Reading First Act more often than the word, the. • Phonemic Awareness: • What happens when a phoneme experiences self actualization?
Reading Jeopardy • • • •
Math Second: What happens when you put Reading First? Reading First: What should you have done before you walked in the wrong gendered rest room? • Consequential Validity • What happens to you when you don’t do what the teacher says?
Reading Jeopardy • Rate, accuracy, and expression • Name 3 words that would have different social consequences for you at a reading conference versus a church social • Reading Recovery: • What do you call a 12 step program for first graders who have overdosed on Accelerated Reader. • Accelerated Reader: • What would you call a person who can read billboards at 120 miles per hour?
A distinction I find important • The world is filled with lots of good indicators that make lousy instructional goals. It is just fine to get better at them as long as you don’t do so by teaching them directly. • Never send a test out to do a curriculum’s job.
Outline for today • Define domain of interest • A short history of vocabulary assessment • Some important features of the domain of vocabulary assessment • A pot pouri of examples of vocabulary assessments
Domain of interest • For sure: Knowledge of word meanings and the conceptual networks in which they exist. BUT – Which words? • in general • a set of specific words from a story, unit, book, etc.
• Perhaps: The ability to use available cues, both inside words and outside of them, to infer--and maybe to learn--the meanings of words
A very short history of vocabulary assessment • Vocabulary assessment has been around as long as we have had – Assessment of any sort – Reading assessment
• Has always been a part of intelligence testing • Has always been a part of reading assessment • Has always been a major part of second language assessment
Trends over time • Early on: test isolated words and find their synonyms or meanings • Not surprisingly, there has been a movement toward contextualization over time – Psycholinguistic and cognitive revolution – Constructivist pedagogies • Whole language and its kissing cousins • Communicative competence (ESL)
Circa 1920s thru 1950s A _______ is used to eat with 1. 2. 3. 4.
Plow Fork Hammer Needle
Foolish 1. 2. 3. 4.
Clever Mild Silly Frank
1970s •
He discovered a new route through the mountains. 1. 2. 3. 4.
•
Note: Vocabulary subtest of this sort correlates .85 to .95 with RC
Wanted Found Traveled captured
Their success came about as a result of your assistance. 1. 2. 3. 4.
according to before because of during
Note that context does allow us to assess abstract words
mid 1980s 1950s: deliberately 1. 2. 3. 4.
Both Noticeably Intentionally Absolutely
1970s: He was found guilty because he did the act deliberately. deliberately 1. 2. 3. 4.
Both Noticeably Intentionally Absolutely
mid 1980s In a (1) democratic society, individuals are presumed innocent until proven guilty. The (2) establishment of guilt is often a difficult task. One consideration is whether or not there remains a (3) reasonable doubt that the suspected persons committed the act in question. Another consideration is whether the acts were committed (4) deliberately. (4)
1. 2. 3. 4.
Both Noticeably Intentionally absolutely
Compared to other formats, this one showed the highest reliability, predictive validity, discrimination
1995: among comprehension questions, insert vocabulary… …Two reasons are usually advanced to account for this tardy development; namely the mental difficulties… The word tardy in line 2 is closest in meaning to 1. 2. 3. 4.
Historical Basic Unusual Late
Note: Still an open question of whether you report vocabulary separately
Now, in the age of on-line assessment… • The Southwest has always been a dry country, where water is scarce, but the Hopi and Zuni were able to bring water from streams to their fields and gardens through irrigation ditches. Because it is so rare, yet so important, water played a major role in their religion. • Look at the word rare in the passage. Click on the word in the text that has the same meaning.
NAEP’s likely influence • That NAEP is in the game will elevate the role of vocabulary • NAEP’s standards (achievement levels) and format will also influence assessment • Good development to attend to. • Note: whether it is a separate scale depends on – Resources (for item development) – Psychometrics (will it scale separately from comprehension)
What will be tested on NAEP? • Assess words characteristic of written language not oral language • Label generally familiar and broadly understood concepts, even if the words themselves are not familiar (akin to Isabel’s discussion of Tier 2): • Stunning but not pretty • Prosperous but not rich • Demonstrate but not show • Are required to built a sensible rendition of the text (and preferably linked to central ideas in the text). • Are characteristic of grade level material (4, 8, 12)
What won’t be tested on NAEP • Words that are narrowly defined and not widely used (appears to be tier 3, technical vocabulary) or just arcane (hamlet or rivulet) • Words that label the main idea of the text (e.g., emancipation in Emancipation Proclamation) • Words that are part of most students’ speaking vocabulary • Words with meanings that are readily derived from context (appositives, parenthetical definitions)
NAEP Distractor features • Can present a more common meaning for the word, which must be ignored in favor of the meaning in the text. • Can present correct information from the text that is NOT the meaning of the word. • May be an alternative interpretation of the context in which the word occurs • Visually or auditorily similar words. Note: Useful to have a theory of distractor generation because it gives meaning to errors
NAEP Achievement levels for vocabulary
Advanced • Advanced readers will have outstanding vocabularies, – With a sound knowledge of words and terms beyond their grade level.
• In addition, they will have an excellent grasp of the multiple meanings of an extensive set of words and complex networks of associations to the words they know. • They will also have a strong base of words that identify complex and abstract ideas and concepts • Finally their sophistication with words and word meanings will enable them to be highly flexible in extending the senses of words they know – to appropriately fit different contexts.
Proficient • Proficient readers will have sizeable meaning vocabularies, – including knowledge of many word and terms above grade level.
• They will also have greater depth of knowledge of words, beyond the most common meaning. • Proficient readers will be flexible with word meanings and able to extend the senses of words whose meanings they know – in order to appropriately fit different contexts and understand passage meaning.
Basic • Readers at the basic level will generally have limited, concrete vocabularies that consist primarily of words at and below grade level. • Knowledge of these words will be limited to the most familiar definition, – making it difficult to identify the appropriate meaning of a word among the distractors.
Dimensions of Vocabulary Assessment • After John Read • Interesting book, entitled Vocabulary Assessment, Cambridge University Press, 2000/2003 • You can assess vocabulary with an eye toward these distinctions: – Discrete--->Embedded (phenomenon) – Selective-->Comprehensive (corpus) – Context independent-->context dependent (format)
Discrete/Embedded • Discrete: vocabulary as an independent construct – (e.g., lots of standardized tests report a separate comprehension score)
• Embedded: vocabulary is assessed but feeds into a score for a larger construct – (e.g., added to comprehension aggregate score) – (e.g., on a typical test, you get • Comprehension • Vocabulary • Total reading
Selective-->Comprehensive • Selective: a targeted set of vocabulary items – – – –
Those in the selection at hand Those in the unit at hand Those in the course of study Those in a certain band of frequency (e.g., 1000 most frequent
• Comprehensive: all the words in some domain or performance – All the words in an essay (when we rate the sophistication of an essay) – All the words in a speech – All the words in a text – All the words in a corpus, language,
Context dependence • Critical aspect is whether you must use the context • Context independent – Don’t have it OR – Don’t need it when you DO have it
• Context dependent: can’t get the right answer without reference to context – Rare words – Nonsense words – Multiple meanings
Context absent • consumed 1.Ate or drank 2.Prepared 3.bought 4.enjoyed
Context present but not critical •
The people consumed their dinner 1. 2. 3. 4.
Ate or drank Prepared bought enjoyed
Context critical •
The people consumed their dinner. 1. 2. 3. 4.
Ate or drank Used up Spent wastefully Destroyed
•
The air conditioner consumed a lot of energy. 1. 2. 3. 4.
Ate or drank Used up Spent wastefully Destroyed
Note: the distractor set matters!
Questions to ask in building or evaluating vocabulary assessments • • • •
What does it mean to know a word What counts as a word How do we choose words How do we know whether selected words are known?
What does it mean to know a word? • Increasing breadth – Know what semantic space it fits into in our heads. What it goes with? • Paradigmatic relations (dog-cat or dog-canine) • Syntagmatic (dog-bark or dog-chase)
– Know how it is used in discourse (recognition and use)
• Increasing depth: – awareness--> acquaintanceship--> ownership – Denotation--> Connotation • Any two words that mean the same at one level of analysis mean something different at another level
What counts as a word? • Words or word families – Extend? Sure – Affixes: Extension, extendable – Inflections: Extends, extended, extending
• Levels of transfer among words in a family (Nagy and Anderson) – Level 1: transparent (they clearly belong to the same family: various-vary – Level 3: stretch: (some metaphorical help) collarbone-collar; visualize-visual – Level 5: accidental: (not much help to know the family: need lots of conceptual support to get from a to b) prefix-fix; peppermint-pepper
An aside: New labels or new ideas? • The RARE words in literary texts tend to emphasize new, more sophisticated, and more precise labels for partially known ideas (just the right nuance) – Misanthrope, discomfited
• The RARE words in informational texts tend to be conceptually central to the selection – Photosynthesis, chlorophyll
• Some RARE concepts in informational texts tend to be secondary senses of common words – Prime, force (a different problem)
How do we choose words • Depends entirely on purpose • How well did students learn the words in the X, where X is a very particular group of words? (story, chapter, unit, text) • How much general vocabulary growth have students experienced over time? • How able are students to use strategies to infer the meanings of words they don’t already know? (How do you know what words a given child knows or doesn’t know? – Context – Morphology (see Joanne Carlisle’s work)
Learning words in a specific corpus • Define the corpus • Draw a sample • Pick a format
Some format options and issues • Checklist test (Anderson-Freebody) • Read each word and put a check beside the ones you know • Put in distractors to correct for kids’ bias toward saying yes (misspellings [parfome]and synthetic/nonsense words [implaceable] or [brep])
Anderson Freebody example Target words __democratic __presumed __establishment __reasonable __committed __deliberately
Non words __ethocracy __preburbed __retableness __reserb __compt __delb
Misspellings __dimecratik __persumed __istableshment __resonible __comitad __deliberately
Note that Anderson Freebody is not unlike some of our informal devices • • • •
I’ve never seen it I’ve seen it but am not sure what it means I know what it means and can explain it I can use it in a sentence or a story
Vocabulary Knowledge Scale 1. 2. 3. 4. 5.
I don’t remember having seen this word before. I have seen this word but don’t know what it means. I have seen this word and I think it means ____________ I know this word and it means __________ I can use this word in a sentence (write the sentence). Paribakht & Wesche, 1997
Format issues • Pictures make life easier… – Reduce the reading load
• And harder… – Ambiguity
• Distractors matter – They largely determine the cognitive focus of the task – What is varied is what you must pay the most attention to..
What is varied is what is assessed • Look at the following examples to determine the focus
Prosperous a. wealthy b. sad c. tall d. happy Prosperous a. doing well b. in trouble c. not very happy d. very lucky
• A person who is prosperous could be said to be… – – – –
doing well financially in trouble not very happy with life very lucky
• A person who is doing well financially could be said to be – – – –
anxious sick open prosperous
• With time things got better and many settlers became prosperous. – – – –
wealthy sad tall happy
• With time things got better and many settlers became ____________ – – – –
anxious sick open prosperous
• With time things got better and many settlers became _____________. – prospered – prosper – prosperous – prosperable
A diversion in assessing breadth/depth • Getting into networks – Semantic associations • What class does it belong to? • What are some examples of it? • What are some of its key features?
– Episodic associations (syntagmatic) • Which of these words go with dog? • Which of these words would you expect to find in the same sentence as dog? – Bark, bite, meow, sleep, soup,
Assessing general vocabulary growth • Usually we resort to some normative assessment: How much growth did they make compared to other kids in the norming sample? – PPVT, vocabulary subtest of any reading assessment,
• Logically possible, but not very practical, to draw samples from a very large corpus – Words in the band of frequency from 1-1000, 3000-5000, etc. – Words in a large but specifiable domain (a semester long course or in a big textbook)
Assessing strategies for inferring word meanings
Context (already discussed) • Rare words that naturally occur – BUT (almost has to be calibrated for individual kids) – What is rare for you may be common for me
• Nonsense words to substitute for real words – BUT (the meanings may be transparent but the nonce will disturb kids)
• Multiple meanings – BUT (assumes that kids will know the various meanings and can discriminate them from one another)
Morphological inferences • Incidental interactions around words (during instructional episodes) • I have not figured out a good multiple choice approach to assessing this important strategy.
Morphological awareness • John is a __________ worker – Capacity – Capable – Capacious – Capability
Morphological awareness • John is good at what he does. He is a __________ worker – Capacity – Capable – Capacious – Capability
Morphological awareness • John is good at what he does. He is a __________ worker – Capacity – Capable – Capacious – Incapable
Morphological Awareness • If laud means admire • then • laudable means something that could be – – – –
admired astonished admire admires
So what is a body to do in assessing vocabulary.
Vocabulary items (picture) Our Picture Assessment in our Seeds and Roots work
restrictions
D
Vocabulary (semantic associations) Our Semantic Associations Assessment in our Seeds and Roots work Stem is a high frequency association: • Target word • Word from sane unit and form class • Word from a parallel unit • High frequency word from same form class and selectional restrictions D
MAZE
D
Make sure your purpose is clear • What do you want to know? • That will determine the domain and the sample you select.
Select a format that fits your situation. • Balances – what we know about the psychometric properties of different formats and approaches – With the realities of our circumstances
• In an ideal world, we might want a carefully crafted set of multiple choice questions, but the realities of our situation might dictate – Fill in the blank – matching
A tip on matching 1. Rapid 2. Cone 3. Incidental
_Used for ice cream _Unintentional _Ugly _Fast _Peppy
Have more choices than there are items to be matched, but… what happens from one item to the next?
Some useful examples
From “Teaching Vocabulary to Adolescents to Improve Comprehension” by Mary E. Curtis and Ann Marie Longo.Reading Online, www.readingonline.org Posted November 2001
1. If Karla was astounded to learn she was chosen, she: (a) expected it (b) was surprised (c) was scared (d) was thrilled 2. If a hotel can accommodate large groups, it can: (a) hire them (b) handle them (c) counsel them The context in the stem limits the (d) train them application of meaning
1. astounded (a) expected (b) surprised (c) scared (d) thrilled 2. accommodate (a) hire (b) handle (c) counsel (d) train Same items without the contextual frame
A nifty rubric for vocabulary use Unit Grade Report (contributes to provision of ongoing assessment and communication about progress) Recognition of Word Meanings You got XX% correct on your post-test. Analysis of Word Relationships • You always completed the Word Activity Sheets in a thoughtful way. • You usually gave some thought to the completion of the Word Activity Sheets. • Too often you did not complete the Word Activity Sheets. Response to What Has Been Read • You did a great job responding to the questions that followed the readings. • I could tell you did some thinking about the articles we read. • Too often you didn’t respond to what we read. Use of Words in Speaking • You participated fully in our discussions of the words and their meanings. • You usually participated in our discussions of the words. • You need to participate more in class discussions. Use of Words in Writing • You did a great job in using your vocabulary words in your writing. • You used some of your vocabulary words in your writing. • Your writing assignments were incomplete or not handed in at all.
Also from Curtis and Longo
Unit Grade Report (contributes to provision of ongoing assessment and communication about progress) Recognition of Word Meanings • You got XX% correct on your post-test. Analysis of Word Relationships • You always completed the Word Activity Sheets in a thoughtful way. • You usually gave some thought to the completion of the Word Activity Sheets. • Too often you did not complete the Word Activity Sheets.
Unit Grade Report Response to What Has Been Read • You did a great job responding to the questions that followed the readings. • I could tell you did some thinking about the articles we read. • Too often you didn’t respond to what we read. Use of Words in Speaking • You participated fully in our discussions of the words and their meanings. • You usually participated in our discussions of the words. • You need to participate more in class discussions. Use of Words in Writing • You did a great job in using your vocabulary words in your writing. • You used some of your vocabulary words in your writing. • Your writing assignments were incomplete or not handed in at all.
Bottom line • We have a long history of assessing vocabulary, but… • Not much research to guide us in selecting the perfect approach • We can get along with many of the tools we have, but… • We need some significant work on the construct validation of vocabulary assessments • The research community, along with the publishing community, needs to provide teachers with better tools. • Don’t forget technology
