Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice - Softcover

Schwandt, Thomas

Softcover

ISBN 10: 0804786550 ISBN 13: 9780804786553

Verlag: Stanford Business Books,US, 2015

Inhaltsangabe

Evaluation examines policies and programs across every arena of human endeavor, from efforts to stop the spread of HIV/AIDS to programs that drive national science policy. Relying on a vast array of methods, from qualitative interviewing to econometrics, it is a "transdiscipline," as opposed to a formal area of academic study. Accounting for these challenges, Evaluation Foundations Revisited offers an introduction for those seeking to better understand evaluation as a professional field. While the acquisition of methods and methodologies to meet the needs of certain projects is important, the foundation of evaluative practice rests on understanding complex issues to balance. Evaluation Foundations Revisited is an invitation to examine the intellectual, practical, and philosophical nexus that lies at the heart of evaluation.

Thomas A. Schwandt shows how to critically engage with the assumptions that underlie how evaluators define and position their work, as well as how they argue for the usefulness of evaluation in society. He looks at issues such as the role of theory, how notions of value and valuing are understood, how evidence is used, how evaluation is related to politics, and what comprises scientific integrity. By coming to better understand the foundations of evaluation, readers will develop what Schwandt terms "a life of the mind of practice," which enables evaluators to draw on a more holistic view to develop reasoned arguments and well fitted techniques.

Über diesen Titel

Über die Autorin bzw. den Autor

Thomas A. Schwandt is Professor of Education at the University of Illinois at Urbana-Champaign. A Fellow of the American Educational Research Association, in 2002, he received the Paul F. Lazarsfeld Award from the American Evaluation Association for his contributions to evaluation theory. He is the author of Evaluation Practice Reconsidered and the Dictionary of Qualitative Inquiry. Schwandt is Editor Emeritus of the American Journal of Evaluation.

Auszug. © Genehmigter Nachdruck. Alle Rechte vorbehalten.

Evaluation Foundations Revisited

Cultivating a Life of the Mind for Practice

By Thomas A. Schwandt

STANFORD UNIVERSITY PRESS

Copyright © 2015 Board of Trustees of the Leland Stanford Junior University
All rights reserved.

ISBN: 978-0-8047-8655-3

Contents

Acknowledgments, 
Prologue, 
1. Variability in Evaluation Practice, 
2. Evaluation Theory and Practice, 
3. Values and Valuing, 
4. Reasoning, Evidence, and Argument, 
5. Politics and Policymaking, 
6. Use, 
7. Professionalism and Professionalization, 
Epilogue, 
Notes, 
Bibliography, 
Index, 

CHAPTER 1

Variability in Evaluation Practice

Best practices have become the most sought after form of knowledge. Not just effective practices, or decent practices, or better practices—but best.

A focus on best practices, toolkits, practice guidelines, and the like arises fairly naturally in the context of concerns about variation in professional practice. The terms variation and variability, as commonly used, signify unevenness, deviation, or divergence from norms or standards, discrepancy, or inconsistency.

Every profession is concerned about variability both for reasons of quality assurance and from the standpoint of ensuring that practitioners consistently deliver on the fundamental purpose of the practice regardless of the particular circumstances in which they work.

There are two general views on the nature of variability in professional practice: One holds that variation is a challenge to the rational basis of practice and could be eliminated if practitioners had clear guidelines, including protocols and rules for decision making. Researchers as well as some practitioners who argue that decisions made in practice are too often based on habit and intuition and lack a firm grounding in empirical evidence endorse this view. (One contemporary example of this idea is evident in the book Moneyball, the story of Billy Beane, who pioneered the use of sabermetrics—objective knowledge about baseball players' performance based on players' statistics—versus the tradition of relying on the intuitions of a team's scouts to evaluate players.) Concerns about restoring the rational basis of practice are also informed, in part, by nearly fifty years of empirical studies examining whether predictions made by expert clinicians are superior to those made by simple statistical rules or algorithms—about 60% of the studies have shown significantly better accuracy for the statistical rules. Finally, support for this way of thinking also comes from those who argue that practice ought to be primarily technically based; that is, it should consist of the application of scientifically validated knowledge.

A second view holds that variability is inherent in multiple dimensions of the environment where a practice is performed (think of variation in both resources and the composition of patient populations in hospitals in rural versus metropolitan areas, for example) and thus it is always an aspect of normal practice. In this view, by definition, practice involves flexibility and constant adjustments and modifications. Generally, those who hold this view subscribe to a judgment-based view of practice as comprised of actions informed by situated judgments of practitioners. Rather than encouraging the development of practices that are protocol-driven and rule following, advocates of this view of variability support the idea of developing practical wisdom. They also challenge the idea that intuition often employed by practitioners is an irrational, unscientific process and cannot be improved or sharpened, so to speak. However, this perspective on variability as a normal dimension of practice does not necessarily mean that addressing and managing variation in practice is not a problem.

Responses to practice variation encompass a range of different actions. Evidence-based approaches to practice promote the use of protocols, practice guidelines, and in some cases rules (consider rules regarding nursing care for patients with dementia, for example) for how practitioners should provide services. Another response involves developing performance measures for practitioners based on protocols. In order to identify behaviors or actions considered outside practice norms, a practitioner's performance is measured and compared to standards or targets; in other words, professional practice is audited and the results of the audit fed back to the practitioner to change behavior. Still other responses involve developing lists of best practices and toolkits that, while perhaps not intended to achieve complete standardization of practice, aim to help practitioners operate on some common ground with shared understandings of concepts, methods, ethical guidelines, and so on.

As suggested by the broad description of the field in the Prologue, variation in evaluation practice is common. No doubt, heightened awareness of this state of affairs of the practice and the motivation to address it have been fueled by the evidence-based movement that has developed across the professions of nursing, social work, teaching, counseling, and clinical medicine. At the heart of this movement is the idea that practitioners ought to use models and techniques that have been shown to be effective based on scientific research. (However, whether what are often touted as best practices are actually backed by scientific evidence is another matter.) Exactly where and when the idea of best practices that originated in the business world migrated to the field of evaluation is not clear, yet the term is no longer a buzzword confined to business enterprises. The literature is full of best practice approaches for evaluating just about everything, including leadership development programs, faculty performance, think tanks, public health interventions, and teacher education programs, to name but a few targets. Moreover, there is a growing sense in the field that although evaluators operate in a world marked by complex contextual conditions, that world "is not so fluid that meaningful patterns cannot be appreciated and used as a basis for action." Hence, in recent years we have witnessed efforts to develop practice guidelines for matching methods to specific evaluation circumstances, as well as guidelines for choosing appropriate means for determining program value in different contexts.

Multiple sources of variability in evaluation practice will be discussed throughout this book. Here, I focus on four primary sources: how evaluation is defined, what methods an evaluator ought to employ, how the professional evaluator relates to and interacts with parties to an evaluation, and how the purpose of the practice is understood.

Defining "Evaluation"

There is no universally agreed upon definition of evaluation, although there are two primary points of view. The first emphasizes that evaluation is an activity concerned with judging value; the second views evaluation as a form of applied research.

In a precise sense—what one would find in dictionary definitions—evaluation refers to the cognitive activity of determining and judging the value of some object, which could be an activity, event, performance, process, product, policy, practice, program, or person. Evaluation is a matter of asking and answering questions about the value of that object (its quality, merit, worth, or significance). The four-step logic involved in doing an evaluation defined in this way is as follows:

1. Select criteria of merit (i.e., those aspects on which the thing being evaluated must do well on to be judged good).

2. Set standards of performance on those criteria (i.e., comparative or absolute levels that must be exceeded to warrant the application of the word "good").

3. Gather data pertaining to the performance of the thing being evaluated on the criteria relative to the standards.

4. Integrate the results into a final value judgment.

Defenders of this understanding of evaluation argue that unless one is using this logic and employing evaluation-specific methodologies directly concerned with means of determining value—for example, needs and values assessment and evaluative synthesis methodologies (combining evaluative ratings on multiple dimensions or components to come to overall conclusions)—one is literally not doing evaluation. In this view, evaluation is a particular kind of critical thinking that follows a specific logic of analysis necessary for appraising value. That logic can be applied to the evaluation of literally anything. A strong statement of this view appears in the Encyclopedia of Evaluation, where we find that it is the "judgment about the value of something ... that distinguishes evaluation from other types of inquiry, such as basic science research, clinical epidemiology, investigative journalism, or public polling."

However, many practitioners of evaluation define it differently as a specific type of applied social science research (i.e., evaluation research) concerned with the processes of collecting, analyzing, interpreting, and communicating information about how a program or policy is working and whether or not it is effective. These practitioners employ the standard tools of the social scientist (experiments, surveys, interviews, field observations, econometric methods) to monitor program processes and to answer questions of whether a policy or program works and why. A prominent concern in evaluation research is establishing the causal link between a program or policy and intended outcomes. A strong advocate of this way of understanding evaluation summarized it as follows: "Evaluation is social research applied to answering policy-oriented questions. As such, an important criterion for judging evaluations is the extent to which they successfully apply the canons of social science."

The central issue here is whether this disagreement in definition is a difference that makes a difference, and for whom. From the perspective of many agencies both domestically and internationally that promote and conduct evaluation, it appears that the two perspectives are sometimes combined and broadly interpreted in yet even different ways. For example, the W.K. Kellogg Foundation defines evaluation not as an event occurring at the completion of program but as a process "providing ongoing, systematic information that strengthens projects during their life cycle, and, whenever possible, outcome data to assess the extent of change" and "helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors." The United Nations Development Programme (UNDP) Independent Evaluation Office defines evaluation as "judgment made of the relevance, appropriateness, effectiveness, efficiency, impact and sustainability of development efforts, based on agreed criteria and benchmarks among key partners and stakeholders." For the U.S. Government Accountability Office, "a program evaluation is a systematic study using research methods to collect and analyze data to assess how well a program is working and why." Finally, the Department of International Development (DFID) of the U.K. government argues that evaluation is a collection of approaches "focuse[d], in particular, on whether planned changes have taken place, how changes have impacted, or not, on different groups of people and investigates the theory behind the change...."

From another perspective, held by at least some evaluators, it is quite important to take an analytical philosophical approach to answering the question, "What can and should legitimately be called the activity of 'evaluating' irrespective of the circumstances in which it is conducted, the object being evaluated, and expectations for its use?" It is important because defining what is actually "evaluation" is intimately related to establishing one's professional identity as an evaluator as distinct from the identity of others engaged in related pursuits like applied social research, program auditing, organization development, and management consulting. After all, claims to expertise are built around distinctive knowledge, theories, and methods. If evaluators cannot lay claim to and agree on the definition and central purpose of their practice (and their own unique body of knowledge) then their identity as a specific kind of professional practice is at risk. Other evaluation practitioners appear to be less concerned with this matter of professional identity, or at least do not see that the issue depends on resolving this definitional dispute. Many who regard evaluation as the application of social science methods are content to view evaluation as a compendium of approaches, concepts, and methods serving multiple purposes and taught as a specialization within social science fields such as economics, psychology, sociology, and political science.

The definitional problem surfaces in an additional way in the international development evaluation community, where it is not uncommon to find the expression monitoring and evaluation (M&E). Many in the evaluation community aim to clearly distinguish monitoring from evaluation. The former is, more or less, the ongoing, routine tracking of a program to diagnose problems in implementation—are we taking the actions we said we would take, delivering the services we said we would deliver, reaching the clients we said we would reach?—and to make adjustments accordingly. It is also a process for tracking performance against stated goals. In this case it is often referred to as performance measurement or performance monitoring. Monitoring is a management tool that assists in timely decision making, ensures accountability, and provides a basis for learning and for evaluation. However, evaluation focuses on objective, independent judgments of value around outputs, processes, and outcomes. When compared to monitoring, the focus of evaluation is less on the determination of what is being accomplished and more on the value of those accomplishments. The difference is summarized in Table 2.

In this book evaluation is regarded primarily as a practice concerned with judging the value of programs and policies. Those judgments can take several generic forms including those listed below:

• X is good as determined by its performance against a set of criteria.

• X is better (more valuable) than Y as determined by comparing X and Y on a set of criteria.

• X is better (more valuable) than Y because X produces the intended outcome and Y does not.

• X is better (more valuable) than Y because, although they both produce the intended outcome, X is more effective or efficient than Y in producing that outcome.

• X is better (more valuable) than Y because, although they both produce the intended outcome, X is more relevant and sustainable.

Choosing Methods

Almost from its beginnings, the field of evaluation has exhibited considerable variation in (and conflicts over) the choice of evaluation methods. Debates, often quite heated, raged in the 1970s and 1980s over the relative merits and appropriate uses of quantitative data and statistical methods associated with psychological and achievement measurement and experimental designs versus qualitative data as generated in case studies and via the use of ethnographic methods. Method debates are generally proxies for deeper differences surrounding what evaluation should be examining. On the one hand, there are evaluators who view policies and programs as "thing-like" objects with identifiable properties (e.g., inputs, processes, outcomes, impacts) that can be readily defined and measured using methods for collecting and analyzing quantitative data. On the other hand, there are evaluators who view programs as multifaceted compositions of human experiences permeated with multiple meanings. Thus, rather than employing methods suited to a variable- and measurement-oriented approach to evaluation, these evaluators use methods associated with ethnographic work including open-ended or unstructured interviews, field observations, and so forth. These methods are used to generate data and evaluation findings that build upon "experiential, personal knowing in real space and time and with real people" and are "steadfastly responsive to the chronological activity, the perceptions, and the voices of people" associated with what is being evaluated.

(Continues...)

Excerpted from Evaluation Foundations Revisited by Thomas A. Schwandt. Copyright © 2015 Board of Trustees of the Leland Stanford Junior University. Excerpted by permission of STANFORD UNIVERSITY PRESS.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Details zur Ausgabe

• Verlag: Stanford Business Books,US
• Erscheinungsdatum: 2015
• Sprache: Englisch
• ISBN 10: 0804786550
• ISBN 13: 9780804786553
• Einband: Tapa blanda
• Anzahl der Seiten: 215
• Kontakt zum Hersteller: Nicht verfügbar
• Verantwortliche Person: Nicht verfügbar