The analysis of “other”

The analysis of "other" | ProjectsHalfDone.jpg

Today I want to chat a little bit about survey analysis. Recently I’ve heard a few stories that disturbed me. One in particular resulted in what I would consider falsifying data. A large organization had hired an outside consultant to conduct some survey work. In the report from the consultant, the response category of “other” had disappeared from a question and after a careful look at the raw data, the responses of “other” had been split up and lumped into the other existing categories. While some people that responded “other” might have actually provided a response that fit in one of the original categories….they still selected the response of “other”.

Is this cleaning the data or is this mishandling the data?

It’s one thing to clean the data for incomplete responses or responses that did not record properly but it’s another to move categories around after the survey had closed. To me, this indicates poor survey design. This isn’t always the case, but it certainly happens.

Now, not to say that we shouldn’t use the category of “other” or that well designed surveys don’t have this category. Quite the opposite, actually. There are times when you absolutely should be using “other” to allow participants to describe their particular experience or when a program has several parts that may fluctuate and you won’t be able to adjust the survey for last minute additions or issues. “Other” can be a very important and informative category.

So let’s pretend that we’ve created a survey for a local college’s orientation and we do need to include the category of “other”. (Note: The category “other” should not be used when pertaining to gender identity. People aren’t “other”. Don’t make them feel that they are. For more information click here.) We have distributed the survey and collected responses from about 50 participants. One question that used “other” was our question on which part of college orientation students liked best.

 

Analysis of "other" | ProjectsHalfDone.png

Looking at the data, some students picked “other”. (And a lot of students seemed to enjoy the talk on financial aid which is a little ridiculous but hey, it’s not real data) Our reaction is often to just ignore this category…but let’s stop for a second and look at it. If we look at the data for more than 10 seconds we will actually see that “other” had more responses than one of our actual categories. This…might indicate a flaw in our survey. So let’s look at the actual data. We should have had a space for students to fill in what they meant by “other”. If we didn’t, then we really have failed to represent the people we are surveying.

Cateogry of Other | ProjectsHalfDone.png

From reading the responses we can see that several students enjoyed meeting the faculty members. Maybe we should have had that as a category. We can also see that one student did like the campus tour which was an option but they had other thoughts on it, so they wanted to share those. That information is still important and that response should not be moved to be grouped with “campus tour” in my opinion. The respondent chose “other” and we need to respect their choice. To change the category is going to far with “data cleaning”.

By looking at what falls under the “other” umbrella we can get a better sense of what we need to ask in our future surveys and what a reasonable number of our respondents were saying. This was an oversimplified example, but I think it translates. I’ve seen so many rich and important points come from the category of “other”. This category needs to be examined, reported, and considered in future survey design. Too often “other” is swept under the rug. Don’t let that happen in your survey design or reporting!

 

Do you overlook “other”? Have you seen other surveys poorly handling this category? What’s the best information you’ve found hidden in an “other” response? Let me know in the comments!

 

The Language of Evaluation

Language of Evaluation |projectshalfdone.wordpress.com

At a recent staff meeting, I presented my colleagues with a variety of evaluation terms. I was interested to see their reaction to the terms in order to open up a dialogue about evaluative practices in our office. Evaluation culture is something that fascinates me and I thought that perhaps starting at the most basic level, the words we use when talking about evaluation, could give some insight in to how people view evaluation practices in my particular workplace.

I instructed my non-evaluator work-family to write down whatever came to mind regarding the term on the card I had given them. The evaluation terms I had chosen were:

  • Monitoring
  • Assessment
  • Evaluation
  • Performance
  • Judgement

After a few minutes we went around and discussed their thoughts on the words. Not surprisingly, there were several negative connotations surrounding evaluation terms. But let’s focus on the positive for a minute.

The most “friendly” evaluation term seemed to be “Performance”. Responses to this word included “Evaluation”, “Artistic”, “Scale” and “Skills”. I wouldn’t necessarily classify these terms as positive or negative, however this was one of the only cards without the perception that evaluation was a threat.

Regarding “Monitoring” the words that came up were “Fiscal duties” and “Transactions”. Monitoring appears to bring up more financial and budgetary thoughts within my group.

“Assessment” had a lot of writing and a lot of discussion. A few of the concerns that came up regarding this term were academic freedom and accreditation. Within the realm of higher education, this word tends to have more of a threatening connotation than does “Evaluation”. Other terms associated with this word included “Measurement”, “Process”, “Formative/summative”, and “Accountability”.

Assessment  |projectshalfdone.wordpress.com

The first thing written on the card for “Evaluation” was “Sounds better than assessment” which did make me laugh. Other responses included “Not sure if it is the same as assessment” and “May scare some people”. Although this term was decidedly less threatening within the group discussion, there was still a lot of worry and concern surrounding the word.

Evaluation

Somewhat unsurprisingly the term “Judgement” was almost completely negative in perception. Responses included “Criticism”, “Evaluation”, “Unfair” and “Picking sides”. It is easy to see how this term would be threatening or negative to most people, but seeing the exact parallels drawn is really interesting.

After reading over the responses from my colleagues and listening to their thoughts on their language of evaluation I have begun to rethink how I approach conversations with my group. Understanding what each organizations “evaluation language” is can help an evaluator to gain footing with organization members. Perhaps by slowly introducing evaluation terms in a non-threatening way, we can change the perception of some of these more negative terms.

I may even consider doing this activity with a new group of stakeholders when starting a new project or working with a new organization. I think that it would also provide insight for ECB and evaluative thinking work. If nothing else it may be a good icebreaker for new groups. As a field we have to work to show the positives than can come from evaluation, and in many cases we also must demonstrate how evaluation is not a threat. In order to develop a culture of evaluation and a dialogue around evaluative practices it helps if you start off speaking the same “language”, or at least understanding the meaning behind the words that you share.

Have you discussed appropriate evaluation terms in your workplace or with stakeholders? Do you find “Assessment” or “Evaluation” to be more acceptable in your particular field?

 

The Last 6 Months

Ok Blog’o’sphere. I slacked.

I’m done with that now. So let’s review what has happened since we last spoke. There’s been a lot going on!!

 

May:

  • Finished the semester
  • Got married!!

June:

  • Collaborated on the evaluation piece for a grant that is still under review. CROSSING MY FINGERS!!
  • Co-taught an online class. Fun!

July:

  • Helped out with the Governor’s School for Agriculture on campus! Got to work with some awesome kids.

August:

  • Worked with Dr. Archibald on Evaluation Capacity Building with Cooperative Extension

September:

  • Back to life…back to reality (Which kind of means things slow down…)
  • EXCEPT WAIT!! Prelims!!!!!!
  • AHHHH!!!

October:

  • Go to AEA for my first time!!
  • PASS ORAL EXAMS!!!!
  • Officially a Doctoral Candidate.

November:

  • Check back and find out!!! And hear more about the rest of that stuff!

Open Access in Evaluation

It’s on every graduate student’s mind…publish, publish, publish.

But where?

What happens if you publish in an open access journal? What IS open access?

Well, in the fields of Evaluation and Extension, there isn’t a lot to go on. Other fields have expansive Open Access journals that would add to ones CV tremendously. For Extension the only viable option I have located is the Open Access Journal of Medicinal and Aromatic Plants.  Interesting, but with Agriculture being so broad, I know nothing about this topic…however I am interested in herbal remedy type-things so perhaps I will peruse this in my spare time. But for me, this is not an option for submission. Still fun though!

Then looking at Evaluation there was a bit more to go off of. The one I found most notable was Practical Assessment, Research and Evaluation. Much closer to my interests!! It is peer reviewed and has been around for 15 years. Pretty impressive! The articles are all mainly focused on education and statistical tests. I’m familiar with these facets of assessment but it’s not my main focus, however looking over their policy page I do see that they publish “Issue papers” on controversial topics. With my focus on Evaluation Culture I wouldn’t necessarily call this a controversial topic, but I think I could work something out.

Interesting options for publication.

Do you think it’s easier or harder to get published in an open access journal? Do you have any experiences? Would you try it as a graduate student?

Project Personal Mission Statement

A professor in my department has a personal mission statement that she shares when doing her “elevator speech”. It’s usually what she leads with. Hers is really more of a tag-line almost. It’s “Science literacy for all”. As a recovering high school science teacher, I love it. I am totally on board.

And clearly, this personal mission statement sticks with you when you hear it. It’s how I define this professor in my head when thinking about her research interests and general professional work. I look up to her a lot as we have a lot in common, being the former science teachers in the Agriculture department. (Honestly, I look up to her a lot in all manner of ways. She’s a pretty inspiring person.)

So this has gotten me thinking. Perhaps I should craft a personal mission statement or tag line.

What do you think? Do you have one?? I’d love to hear it!

I’ve also discussed this idea with one of the professors on my committee, Dr. A. He said he had seen similar things in the Evaluation field, and I could use it in my email signature or in the blog, or even on the CV. Hmmm…

Right now, mine is rather lengthy…and mostly in my head. So let’s try to get it written out and then maybe work on shortening it, shall we?

Ok. Here goes.

Evaluation should be used to help you improve. It’s a snapshot, not a snap judgement. Evaluation should not be intimidating or used to threaten. It should be used to focus on how you can grow either as an individual or as an organization. I want to take the negative connotation out of the thought “I’m being evaluated”. It should be helpful, not harmful. Productive, not destructive.

What do we think about that? Ok, enough with the “this…not that”. Too many? Haha.

Ok…let’s try for the short version

 Evaluation is a tool. Not a judgement call.

Yes? Maybe? Does that get my point across or allow questions that I could then elaborate my point?

What parts do you like? Anything you disagree with? I’d love to hear from you!

Houston, We Have A Committee

YES!!!!

I’m super excited to have finally nailed this committee thing down. I feel like I can make real progress now and everyone involved seems excited about my research. I can’t thank each of them enough, and although probably only one of them will read this…I really am so incredibly grateful to have them helping me with this process. It’s a huge time commitment to agree to serve on a student’s committee and I recognize that and really value both their input and their time.

As far as my research goes, I plan to focus on the evaluation competencies of extension agents. Evaluation is something that another professor opened my eyes to. I fell in love with it and am so happy I took her class. She was a huge part in helping decide my dissertation topic. Although she is not on my committee I owe so much to her as well.

There are so many people I would put on my committee just because they have been so influential in my graduate studies, but sadly our research interests don’t align. They can be on my cheering committee, perhaps. At least in my mind they are. I feel so lucky to be at this University with such amazing professors. They all have helped get me to this point.

Well, I’m going to take my super-excited-newly-formed-committee-energy and go read some things and move forward!!

Navigating the Evaluation Factor

Driving to campus a week or so ago I was listening to my trusty NPR and caught the end of this story on Charity Navigator and EVALUATION! I was dorkily excited to hear an evaluation topic being discussed on a national level. So I had to go look the full story up and see what it was all about.

Most of us have at least heard of Charity Navigator. They help you as a donor choose which charity you may want to contribute to, based on the data they have collected. You can always just choose by which charity serves your interests best, but it’s nice to have the data and see how your favorite charity is doing and WHAT they are doing with your money!

Of course, those donors who contribute thousands upon thousands of dollars, or granting organizations REALLY want to know what you are doing with their funds. Thus, Charity Navigator.

According to the story, President and CEO of Charity Navigator Ken Berger is concerned with looking at the measurable outputs of the charities.  The old way of doing this was to compare how much money was spent on the programs to the money spent on overhead (workers, resources, the big boss man’s salary, etc.). Ideally we the donors want to see most of our dollars going to the ACTUAL charity work and only a sliver going to the running of the organization. Ideally. When I fundraised for Team in Training (Leukemia and Lymphoma Society) 75% of all donations went straight to programs, doctors, hospitals and research. Only 25% was routed to running TNT and it’s employees. Not a bad split. You’d be shocked at some of the numbers.

But now a new evaluation has been proposed. According to NPR and Berger this evaluation will consist of looking at the OUPUT. (WHAT!?! Logic models maybe!?!?) They will be looking at the problem and actual MEASURES to SOLVE it.

Yeah, the charities freaked out a little bit. My favorite quote was from Doctors Without Borders who wanted to know if they were supposed to go survey their patients in a war torn area on how they enjoyed their care.

They have a point, but still, the evaluator in me is very excited by this new system.

Yes, it will be harder to measure. Yes, it will be complex. Yes, charities will have to PROVE what they are doing. Yes, they will have to have a plan. But most of all…YES YOU NEED AN EVALUATOR!!!

I also appreciated the concern expressed about HOW exactly these evaluations would be judged. Evaluation can have such a negative connotation. Everyone freaks out that they will be judged harshly, funds will be revoked, etc. This is what drives me NUTS about the field that I so much want to make my career in. We are not the bad guys!! Evaluation should be HELPFUL not harmful. It’s honest. It shows you where you need to improve. Kind of like your yearly check-up when you were a kid. We look through things, and tell you where you are doing well, if you’re on track, and where you could improve. Maybe little Suzie needs to exercise a little more. Maybe the charity needs to cut out some overhead or streamline their deliverables. It’s not an end-all-be-all-you-fail-game-over judgement in most cases. Or it shouldn’t be. It’s a status report. It’s up to the directors, CEO’s, powers that be, to take it and act on the information.

I really loved seeing evaluation get some spotlight time. Now if we can just convince everyone we aren’t big, scary, judgy and mean.