For the Love of Counting: The Response Rate Rat Race

2015-03-14_OhNoLogo22-abby3I’m in the midst of our annual summer experiences survey – my office’s push to understand what do students do over the summer? And is it meaningful? We know that getting ALL students to respond to our 3-12* question survey would be near impossible, but, as the assessment person on the team, it’s my job to always chase that dream (let’s be real, it’s my obsession to chase that dream!). And at a small institution like where I work getting a response rate of 100% (~1500 students) is seemingly an attainable goal. But this raises so many questions for me.

A little bit of context about the survey. Students do many valuable things over the summer that add meaning to their college experience; the particular subsection of this data that chiefly interests me (as I rep the Career Center) is the number of students who intern.

Common statistical wisdom would tell me that if I am indeed going to report on how many students intern over the summer then I need a certain response rate in order to make an accurate, broader statement about what percentage of Carleton students intern. This stats wisdom is based on a few factors: my population size, my sample size, and the margin of error with which I’m comfortable (I know, I know…ugh, statistic terms. Or maybe some of you are saying YAY! Statistic terms! Don’t let me stereotype you):

Population size = 1500 (all the upperclass students)

Sample size = 1275 (well…this is the goal…which is an 85% response rate…but do I need this # to be accurate and broad?? Hmm…better look at what margin of error I’m comfortable with…)

Margin of error = um…no error??? Baaaahhhh statistics! Sorry readers, I’m not a stats maven. But that’s ok, because SurveyMonkey greatly helped me to determine this

Margin of Error

Ok, so if I want to be SUPER confident (99%) then my goal of 1,275 students (or an 85% response rate) will get me a VERY small margin of error (read: this is good). But, turns out if I look at this from the angle of sample size, I could have the same small margin of error if I only had 1,103 students respond (74% response rate).

Sample Size

So, at this point, I could ask: Why the heck am I busting my butt to get those extra 11% of respondents??? YARG! And statistically, that would be a valid question.

But I don’t ask that question. I know I chase the 85% and 100% response rate dream because I aim to serve ALL students. And even if statistically all the students after 1,103 respond consistently, there is likely an outlier…one or a few student stories that tell me something that the first 1,103 couldn’t that shape a better student experience for all.

So to all of you regardless of if you have a relatively small population size (like me) or a much larger one (hint, Mark, Michigan Engineering, hint), I say keep up the good work trying to reach and understand the stories of 100% of your students. It may be an impossible dream but that doesn’t make it any less worthy a pursuit.

*3-12 question survey based on what the student did over the summer - skip logic, woot woot!
Advertisements

Whistling Vivaldi

2015-03-14_OhNoLogo22-mark3I’m in the last few days of my 2-week summer vacation and I thought now would be as good a time as any to put together a post. It seems the closer I am to an institution, the more I get thinking about higher ed. Today, I’m at a Bruegger’s Bagels in Northampton, MA — home (or near-home) to a handful of colleges and universities. I’m also plagued by a very agile fly. He likes to fly around my hands. I can’t seem to get him, and fellow patrons are starting to stare.

This summer, we’re reading a book for professional development: whistling vivaldi by Claude M. Steele.  I won’t summarize the entire book for you — admittedly, I’m only about a third of the way into it. Thus far he’s exploring the impact of stigma on performance. Stereotype threat is the idea that our performance (in anything) is impacted by the stereotypes placed upon our identities. The expectations placed upon us by virtue of those identities affect our performance whether we’d like them to or not. Often times, the fear of confirming a stereotype about one of our identities hinders our performance in that identity, regardless whether that stereotype holds merit. We don’t want to give truth to that stereotype.

Consider this situation: In graduate school, we had many conversations in class about identity. As someone with many majority identities (e.g., white, heterosexual, male, etc.), I constantly second-guessed my contributions to class conversations — afraid that everything I said would be an opportunity for a classmate to think “oh, he just doesn’t get it, he’s [straight, white, male, etc.].” You can bet this fear kept me from fully engaging in the class conversations. I didn’t want to be seen as out of touch — or worse, unable to understand.

Stereotypes blur the way we understand the world. In the book, Steele points out the difference between the “observer’s perspective” and the “actor’s perspective.” As we’re often in the observer’s perspective, we’re only able to focus on what we can see or notice. This perspective tends to be a view from the clouds and causes us to miss context in which the actor (i.e., person studied) is making those decisions.

To illustrate his point, Steele references the 1978 Seattle Supersonics basketball team. The team started out the season losing at an alarming rate. Local sports analysts were able to break down, in detail, all of the reasons the team struggled. Shortly after the beginning of the season, the team hired a new coach. From there, the team started to win — and would later reach the NBA finals — despite having exactly the same players with the same skill sets ridiculed in the first few weeks of the season. When viewed from a different lense, characteristics originally seen as contributing to their struggles were now the reasons for their success.

It’s almost as though our expectations highlight the things we expect to see, and hide those we don’t expect.