Evaluation Research:
Help or Hindrance?

by

K. Edward Renner, PhD ⁽¹⁾

There is help, and there is the kind of help we can all do with out.

Like many things, evaluation research has two sides. It can be the basis for a dynamic and innovative organization, or it can be dancing to someone else's tune.

But, this is the information age, and dance we all will to standards of accountability. The choice is not whether to evaluate, but whether the music will come from within or from without, and whether the process will be embraced or resisted.

I have worked with community groups and organizations for the past 30 years, including a sexual assault service, a soup kitchen for the homeless men, correctional programs, youth, and a police department among others.

As an example, Father John the director of the soup kitchen was spending all his energy putting 350 meals a day on the table. After consultation on evaluation, Father John had several of the soup-line volunteers come out from behind the serving line each day to sit down and talk with the men, one-by-one. All of the clients wanted to be included! Soon, they all were captured in a concise data file. The men had a story of unmet needs and hopes for the future, and Father John could tell their story. Now the soup cottage had food to feed their bodies and knowledge to help sustain their spirits.

Good evaluation research is descriptive of what is, and prescriptively points toward what might be. With evaluation there is a logical transition from today to tomorrow. With an agenda, comes internal purpose and direction, originating from the client and finding expression through the agency. Transition and evolution are natural consequences.

As a second example, one day a police officer in the police department I worked with saw me on the sidewalk while he was engaged in a heated conversation with a citizen. He hollered at me: "Hey, Renner, I have one for us here." What he meant was he was having a troublesome encounter with a citizen which needed to be recorded in our "critical incident" file as one of the examples for problem-solving analysis and discussion.

Before evaluation research had become part of the police routine, this same encounter with the citizen would have been seen as an interaction with a "turkey" who did not understand how hard it is to be a policeman. The transition in the policeman's mind from "turkey" to a problematic "critical incident" for later analysis and potential change is the power of evaluation research. Evaluation nested in everyday experiences changes problems to opportunities for better solutions.

Fairy tales? No.

These kinds of positive outcomes are no accident. They regularly occur as a result of good evaluation programs which follow well established principles, most of which are social and psychological, not technical.

This should not be surprising. With the information age has come the capacity to know and keep track of anything. Collecting and processing information is not a problem.

Wanting to do so is the issue. Father John wanted to do more than just provide food. He had a passion for his client's spirit; he wanted his people to have both a meal and hope. The policeman wanted to have better police-citizen relations; he wanted to work in the comfort of community consensus.

These human aspirations are the energy for evaluation research, for professionalism, for innovation and for continuous quality improvement. Evaluation is a singular process that must be embraced by client, staff, community and agency to be effective. If it is not, it will fail. Evaluation is fragile, like any other interpersonal and social phenomenon. It is first a human process, and second an engineering task.

Of course, the information that is collected, and how it is collected, must be technically sound. Quantitative skills are necessary, but they alone are not sufficient. The danger in evaluation research is that the technical capacity for recording leads to uncritically trying to know everything, all at once. When this happens, the evaluation procedures can become so intrusive and cumbersome as to destroy the essence of what is being measured. Thus, the challenge is to take the time to determine what needs to be known at this moment. There is no "technical" solution to this step; it requires good judgement based on a sense of purpose and direction. The ends must drive the means.

Seen from this perspective, evaluation research, like all good science, is simply a form of story telling. It is a continuously adjusting process that finds a better solution, which changes the context, thus raising new issues requiring still other solutions. And, the circle starts again.

Evaluation is dynamic, not static. It stimulates innovation. It both causes and corrects mistakes. It is professionalism. It is fun. It produces change. It is enabling. It is like being a character in story that does not yet have an end, but whose end will be determined by the actions of the character, although the consequences of the action are unknown at the time of choice. It is as close as we can come to self-direction, self-determination and self-sufficiency.

1. Edward Renner is a consultant in the area of program evaluation and evaluation research. He may be reached at 14241 110th Terrace North, Largo, FL, 33774, (727) 595-3857, or at: erenner@kerenner.com.