The Political Nature of Program Evaluation

Program evaluation is a political process. An evaluator who ignores, avoids, or mismanages the political realities of evaluation limits the effectiveness and usefulness of the process (Fitzpatrick, Sanders, & Worthen, 2010). Ethical complexities wind in and among the more overt political features of evaluation such as financial support, stakeholder allegiance, and social impact. Morris and Cohn (1993) detail several ways in which stakeholders may seek to influence evaluation outcomes, and Fitzpatrick et al. (2010) caution that evaluators also need to be aware of their own potential to taint the evaluative process.

If we accept that evaluation is political (and, therefore, ripe for ethical complication), then we must ask how best to balance the objectivity required in a program evaluation with the political interests of stakeholders. We must ask, “What ethical standards and values need to be emphasized in program evaluation?”

The American Evaluation Association’s (AEA) Program Evaluation Standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011) and Guiding Principles for Evaluators (American Evaluation Association, 2004) provide a broad, somewhat obvious, framework for ethical conduct.

Fitzpatrick et al. (2010) are more specific, encouraging evaluators to be both self-reflective about their role in the evaluation process and circumspect about client requests, so as to minimize the potential for bias and ethical compromise: “…the client may be asking for what the client perceives as editing changes, but the evaluator sees as watering down the clarity or strength of the judgments made” (p. 81). And Schweigert (2007) roots evaluator responsibility in the notion of justice – public, procedural, and distributive.

From this we can extract answers to the question “What ethical standards and values need to be emphasized in program evaluation?” 

Ethical standards:

• Those detailed in the AEA’s and other professionally recognized codes of conduct.

Values:

• Commitment to truth – what Schweigert (2007) calls the priority of justice
• Cultural sensitivity
• Respect (for stakeholders, ourselves, and the evaluation process)

It seems that no professional code nor personal charter can do the whole job. No matter how pointed the professional standards, situational circumstance requires evaluators to make interpretations and best guesses (Schweigert, 2007), which are subject to bias and ethical compromise, as Weiss (2006) lays bare any illusions we may have that we are above or beyond the snare of bias and ethical confusion: “You never start from scratch. We pick up the ideas that are congenial to our own perspective. Therefore, people pick up this thought or that interpretation of a research report that fits with what they know or what they want to do” (p. 480).

I have thought about this a lot over the past few days, returning again and again to Sieber’s (1980) conclusion that “being ethical in program evaluation is a process of growth in understanding, perception, and creative problem-solving ability that respects the interests of individuals and of society” (p. 53). 

Sally Bacchetta

References

American Evaluation Association, 2004. Guiding principles. Retrieved from www.eval.org/Publications/Guiding Principles.asp.

Fitzpatrick, J., Sanders, J., & Worthen, B. (2010). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Boston, MA: Pearson

Morris, M., & Cohn, R. (1993). Program evaluators and ethical challenges: A national survey. Evaluation Review, 17, 621-642.

Schweigert, F. J. (2007). The priority of justice: A framework approach to ethics in program evaluation. Evaluation and Program Planning, 30(4), 394–399.

Sieber, J. E. (1980). Being ethical: Professional and personal decisions in program evaluation. In R.E. Perloff & E. Perloff (Eds.), Values, ethics, and standards in evaluation. New Directions for Program Evaluation, No. 7, 51-61. San Francisco: Jossey-Bass. 

Weiss, C. H., & Mark, M. M. (2006). The oral history of evaluation Part IV: The professional evolution of Carol Weiss. American Journal of Evaluation, 27(4), p. 474-483.

Fostering Behavior Change

In Fostering Behavior Change (Tulgan, 2013), Bruce Tulgan offers seven best practices for creating training that increases knowledge uptake and meaningful behavior change. Two-thirds of my way through a Master’s degree in Instructional Design and Technology, I first thought Tulgan’s tips were obvious. Simplistic. After thinking about it quite a bit, I’m sure they are. Why would Tulgan, an established training expert, tell us what we already know? Because it’s true. Because he’s right.  

There is no magic to training, and all the cool whiz-bang technology in the world doesn’t change the fact that effective training is a product of sound design and delivery. Tulgan’s tips should seem obvious, because he is reminding instructional designers and trainers of what we already know, yet sometimes fail to execute. We need to leverage needs assessments to align instructional objectives with identifiable skill and knowledge gaps, link instructional content to real-life, and deliver content to multiple memory centers. Sticky training offers actionable solutions and learning extensions. Finally, we need to follow up and cultivate support for ongoing learning.

We know this. We need to do it. Every time.

Make a great day,

Sally Bacchetta

Reference

Tulgan, B. (2013, January/February). Fostering behavior change. Training, 50(1), p. 9.

The Future of Distance Learning

I was recently asked to share my thoughts on the future of distance learning, and I’m struggling to be original about the topic. It seems self-evident that the future of distance learning is expansive and inclusive and ever-so-much-more-so. Distance learning has burgeoned far beyond the realm of training for medical transcription and is ubiquitous in corporate settings, higher education, and K-12 education. But the real driver is not just that more people are learning at a distance; it’s that more people are recognizing distance education as a viable alternative to traditional F2F learning. The quality of distance learning coupled with the broad availability of emerging technologies have transformed distance learning from a fall-back, Plan B position to a deliberate first choice.  

The growing acceptance of distance learning is fueled by a global increase in online communication. As more of us spend more time together online, engaging with more diverse groups than we ever would in person, the complications of distance matter less, and the benefits matter more, to individuals, corporations, and educational institutions (Laureate Education, Inc., 2009).

If there is a popularity ceiling for distance learning, it may be framed and fortified by distance education institutes themselves. A study by Gambescia and Paolucci (2009) found that few institutions effectively leverage their academic integrity in their promotions, relying instead on convenience and flexibility to appeal to potential students. The study didn’t reveal reasons for this, but I’ll speculate on two reasons:

1. Convenience and flexibility are big draws for distance learning. It’s slam dunk marketing.
2. It’s easier to leverage innate characteristics of distance learning than it is to ensure the academic integrity of distance education.

As Gambescia and Paolucci note, “to ensure a high-level of academic fidelity and integrity for online degree programs is not simply a matter of the university transferring current academic assets to the new online degree programs—throwing it over the fence, so to speak. Transferring such academic assets to online degree programs will understandably call for changes, as the inputs and outputs of online degree program offerings by design can be quite different” (Gambescia & Paolucci, 2009).

Assuming the momentum gathering around distance learning is indeed forward momentum, distance learning institutions themselves have work to do. Enrollment – ballooning. Acceptance – growing. Perceived quality – ?

More is expected of those to whom more has been given; so it is in distance learning as in other aspects of life. Those of us who are distance learners or work in distance learning are uniquely positioned to cultivate positive impressions of distance learning. We do this best by doing well in our endeavors and demonstrating the high standards of distance education today.

Sally Bacchetta

References

Gambescia, S., & Paolucci, R. (2009). Academic fidelity and integrity as attributes of university online degree program offerings. Online Journal of Distance Learning Administration, 12(1). Retrieved from http://www.westga.edu/~distance/ojdla/spring121/gambescia121.html

Laureate Education, Inc. (Producer). (2010). The future of distance education [DVD]. Baltimore, MD: Author

Schmidt, E., & Gallegos, A. (2001). Distance learning: Issues and concerns of distance learners. Journal of Industrial Technology, 17(3). Retrieved from http://atmae.org/jit/Articles/schmidt041801.pdf

Scope Creep: A Horror Story

This week brings another interesting assignment for my Project Management course. I am asked to reflect on an experience I had with scope creep and, well... see for yourself. 

Describe a project, either personal or professional, that experienced issues related to scope creep.  In a former career I managed a residential treatment program for adults with mental illnesses. I supervised the staff and residents of a ten-apartment facility, provided counseling, assisted with activities of daily living, and participated in inter-disciplinary planning and treatment for each resident. The need for mental health care far outpaced available resources, and the CEO continually scouted for new properties to acquire and convert to accommodate our long waiting list.

Such a property was found not far from my facility, and I was flattered when the CEO asked for my help in establishing the new facility. He explained that I would maintain my current duties and also be responsible for interviewing, hiring, and training the new staff. I would also supervise them until a manager was hired and trained. I had a good deal of experience and a solid team, and I was confident that I could take over the additional responsibilities without compromising either program.  


What specific scope creep issues occurred?  Almost immediately my role at the new location changed from temporary manager to design consultant-construction site supervisor-accounting rep-professional cleaner-building superintendent. With each passing day I found myself making decisions I had neither the qualifications nor the desire to make.


What interior paint colors do you want? Sally can decide.
We need to order furniture. Let’s ask Sally to do it.
What equipment do we need to set up the new office? The new kitchen? The residents’ bedrooms? Have Sally develop a list. Give Sally the corporate credit card. Let’s have Sally be there to take delivery.
There are bats in the fireplace. My invoice hasn’t been paid. We found mold in the basement. The porch foundation didn’t pass inspection. Call Sally!


How did you or other stakeholders deal with those issues at the time?  My staff stepped up and took on extra responsibilities. The contractors became progressively less motivated, less patient, and less concerned with quality. The CEO went on a 14-day cruise with his wife, and I quit exercising, dusting, and cooking decent meals; I drank too much coffee and slept very little, always with a pager by my side.


Looking back on the experience now, had you been in the position of managing the project, what could you have done to better manage these issues and control the scope of the project? What I could have done better is to actually manage the project instead of scrambling to keep up with the scope creep. I lacked project management experience, and I was so focused on the overwhelming need for more formalized mental health support that it didn’t occur to me to refuse (or question) any of the tasks that were dumped on me.


If I knew then what I know now, I would have drafted some type of work breakdown structure (Portny, Mantel, Meredith, Shafer, Sutton, & Kramer, 2008). I would have listed Level 1, 2, 3, etc., tasks and sub-tasks and identified and allocated resources for each. I would have outlined an appropriate chain-of-command and sought approval for each of these documents (Greer, 2010). And I would have said “no.”


“No” to hauling office furniture up a flight of stairs. “No” to manually seeding the acre lot using an old rusty spreader so we could save a few bucks. And absolutely, positively, “no” to checking to see if there really were bats in the fireplace. Yeah, there were. There sure were.

Sally Bacchetta

Resources

Greer, M. (2010). The project management minimalist: Just enough PM to rock your projects! (Laureate custom ed.). Baltimore: Laureate Education, Inc.

Portny, S. E., Mantel, S. J., Meredith, J. R., Shafer, S. M., Sutton, M. M., & Kramer, B. E. (2008). Project management: Planning, scheduling, and controlling projects. Hoboken, NJ: John Wiley & Sons, Inc.

Managing Your Project Schedule

One of the most challenging aspects of instructional design project management is managing the project schedule. The schedule must be rigid enough to provide structure, yet fluid enough to allow for the inevitable inevitabilities of project work. One way to develop your own scheduling style is to read and learn about what works for other people, so here is my two cents.

As often as possible I work back-to-front to draft my ideal network diagram, beginning with the end date of the final deliverable and working backwards to identify prerequisite (predecessor) activities and deliverables, time requirements, and event deadlines, based on required, procedural, and logical relationships (Portny et al., 2008).

I then compare my ideal network diagram to what I call “probable reality,” which is a consideration of my experience with:

• projects in general
• similar projects
• what I know about this client
• what I know about this project team
• my own schedule and workload

Probable reality reflects my best guess of project limitations and unknowns (Portny et al., 2008) and their anticipated impact on the project. For example, let’s assume that my project team has 21 days to submit a deliverable of a module outline with instructional objectives. I consider my experience with projects in general and similar projects, and determine that I need 14 days (span time) to complete the deliverable.

In my ideal network diagram I allow 3 days for the client to review and approve the outline (17 days). However, I know this client is slow to review and approve materials, so I build an extra 2 days into the activity phase (19 days). That leaves the team only 2 days to review client feedback, make revisions, and re-submit the outline to the client. 

If I know the project team is highly organized and on point, I may take a risk and go with that. But if I don’t know the project team well, or if I know them to be slow or unorganized or very busy, I will decide that it’s not enough time and I need to cut some time somewhere else in the process.

Every activity eats up time (Laureate Education, Inc., 2012), and when time is tight, my first choice is to cut time on my end where I have the most control. I then consider my schedule and workload to find a way to complete the deliverable in less than 14 days.

Freelance work can be very patchwork, and I sometimes function as ID, writer, and PM all in one. In those situations I can track everything with a simple Word document and an Excel spreadsheet. When working with a team, however, project management software is almost essential (Fabac, 2006). It facilitates consistent communication and sharing of timelines, milestones, progress, and changes between team members and clients.

Project work is a wild ride every time. It can be stressful, but the feeling of success is a rush!

 

Sally Bacchetta

References

Fabac, J. N. (2006). Project management for systematic training. Advances in Developing

Human Resources, 8(4), 540–547.

Laureate Education, Inc., (Producer). (2012). Creating a project schedule. [Multimedia Program]. Baltimore, MD: Author.

Portny, S. E., Mantel, S. J., Meredith, J. R., Shafer, S. M., Sutton, M. M., & Kramer, B. E.

(2008). Project management: Planning, scheduling, and controlling projects. Hoboken, NJ: John Wiley & Sons, Inc.