What are the potential benefits of integrating evaluation training into sociology programs? Should sociology departments collaborate with evaluation practitioners or orga

Should Sociology Departments Train Students for Careers in Evaluation? An Analysis of Evaluation Methodology Courses

Allison Nichols1 & Deborah A. Eckberg2 &

Barbara Thomas3

Published online: 12 September 2015 # Springer Science+Business Media New York 2015

Abstract This paper seeks to establish whether sociology departments should provide training in evaluation methodologies. First, the authors look at the current status of evaluation methodology courses within the offerings of sociology departments by examining course offerings from a sampling of sociology departments in doctoral and master’s level programs across the United States. The authors then discuss whether evaluation courses should be offered in sociology departments at the undergraduate, masters, or doctoral levels and whether there are career opportunities for sociologists trained in evaluation research. The authors conclude with reasons why evaluation courses should be offered within sociology departments.

Keywords Evaluation courses . Evaluators . Applied sociology. Sociology departments .

Sociology careers . Sociology course offerings . Jobs for evaluators . Evaluation methodologies

Introduction

Given the high cost of education, parents of college students are concerned about how their children will be able to pay back expensive higher education loans after college, and are interested in how likely it is that their college degrees will lead to good-paying jobs. Sociology programs, which are competing with more vocationally-focused

Am Soc (2015) 46:486–499 DOI 10.1007/s12108-015-9290-9

* Allison Nichols [email protected]

1 West Virginia University Extension, Morgantown, WV, USA 2 School of Law Enforcement and Criminal Justice, Metropolitan State University, St Paul,

MN, USA 3 College of Literature, Science and the Arts, University of Michigan, Ann Arbor, MI, USA

majors, are under pressure to find ways to place their students in discipline-related jobs after graduation. Some college administrators want sociology to become one of what Brint (2002) calls the Bpractical arts^ which lead directly to career opportunities. The areas of sociology in which there have been an increase in course offerings are applied areas such as: Crime/Law (21.4 %), Culture (18.6 %), Gender/Sexuality (16.8 %), Race and Ethnicity (15.3 %), and Health (14.2 %) (Spalter-Roth et al. 2013). The authors of this article propose that, in addition to careers represented by the applied sociology offerings listed above, sociology departments should also consider preparing students for careers in evaluation since many of the courses used to train evaluators are already in place within sociology curricula.

What is the Scope of Evaluation Related Instruction in Sociology Departments Today?

Method and Sample

Using a 2009 list of sociology departments in the United States that participated in a survey administered by the American Sociological Association (ASA) as the sampling frame, all sociology departments at institutions identified as Research I, Doctoral I and II universities and every tenth department at institutions categorized as Masters I colleges or universities were included in the sample (Spalter-Roth and Scelza 2009). This strategy resulted in a sample totaling 133 sociology departments across the United States.

To support the goals for the study, which were to analyze the content of sociology department websites regarding program evaluation course and degree offerings and to understand how and whether evaluation content fits within sociology programs and courses, one of the authors developed a protocol to systematically collect these data. The content analysis protocol included fields for evaluation course content, including general evaluation theory, design, implementation; evaluation theory; evaluation de- sign; evaluation methodology; evaluation reporting; and a category to capture evalua- tion courses that did not fit the aforementioned topical areas. Data regarding the program degree level at which program evaluation course content were offered, whether coursework in research methodologies used in program evaluation were offered and, if so, at what program degree level such coursework was offered, whether program evaluation is mentioned as a possible career choice and whether faculty program evaluation expertise is noted. Content analysis data were collected in July and August 2014 using Qualtrics.

Findings from the Content Analysis

The content analysis is based on sociology department website data from 64 Research I, 29 Doctoral I, 21 Doctoral II and 19 Master’s I institutions; the vast majority of the departments were sociology-only departments, while fewer were sociology and anthropology, sociology and social work or sociology and another discipline departmental units.

Am Soc (2015) 46:486–499 487

Of the 64 sociology departments at Research I institutions, 58, or nearly 91 %, did not offer classes specifically aimed at evaluation designs, methodologies, or techniques. Among the sociology departments at Doctoral I institutions, almost 59 % did not offer coursework specifically focused on evaluation designs, methods or techniques; 17 of the 21 sociology departments at Doctoral II institutions did not offer this coursework and 8 of 19 of the Master’s I institutions in the sample did not offer classes specifically aimed at evaluation designs, methodologies, or techniques.

In sum, of the 133 sociology department websites sampled, only 19 offered classes focused on evaluation designs, methodologies, or techniques. When this type of coursework was noted on a sociology department website, typically just one or two courses were described. It should be noted that 11 sociology department websites did not contain sufficient information to definitively ascertain whether coursework focused on evaluation designs, methodologies, or techniques was available. Table 1 lists the names and course numbers of evaluation courses in sociology departments of 17 universities/colleges.

Table 1 Sociology courses in evaluation

Institution Major/department Course number

Name of course

Boston College Sociology 5554 Qualitative methods for applied settings; seminar

Boston College Sociology 2254 Community service research

Boston College Sociology 2255 Advanced community service research seminar II

Catholic University of America Sociology 350 Computer applications to policy research

Catholic University of America Sociology 631 Social policy analysis and evaluation

University of Alabama, Birmingham

Sociology 715 Program evaluation

University of Arizona Sociology 596 Theory and research in the non-profit sector

University of Hawaii Sociology 701 Seminar in evaluation research

Loyola University, New Orleans Criminal Justice 805 Program planning, implementation, and evaluation

St. Peter’s College Criminal Justice 805 Criminal justice policy formation and analysis

Loyola University of Chicago Sociology 711 Human services administration – includes evaluation design, program monitoring, and social impact measurement

Marquette University Sociology 476 Research special areas-engaged methodologies

St. John’s University Sociology 301 Evaluation research

South Dakota State Sociology/ Comm. Dev.

631 Evaluation of organizations and programs

University of Louisville Sociology 617 Program evaluation & impact analysis

Western Michigan Sociology 6870 Evaluation research

University of Central Florida Sociology 6657 Program design and evaluation course

University of Nevada, Reno Sociology 738 Methods and innovations in assessment

University of New Hampshire Sociology 894 Evaluation of social programs

University of New Hampshire Sociology 794 Evaluation of social programs

West Virginia University Sociology 693 Evaluation research methods

488 Am Soc (2015) 46:486–499

The types of content in the evaluation courses offered within sociology departments included evaluation theory and techniques and analysis of models to study organiza- tions holistically as well as specific programs within organizations (http://www.sdstate. edu, http://www.wmich.edu). Some courses focus specifically on case studies toward an understanding of evaluation utilization, and require students to design a program evaluation in their substantive field of interest (http://www.unh.edu). One course offered as a seminar in an applied sociology master’s level program, prepares students for careers in evaluation by requiring that each student prepare a comprehensive program evaluation proposal that he or she can show to potential employers. The professor assumes that students have learned basic research skills in their other courses and focuses on change theory, logic modelling, planning, implementation, and reporting (http://soca.wvu.edu/graduate). See Appendix for a sample syllabus.

Perhaps not surprisingly, the vast majority of sociology department websites sam- pled indicated course offerings in research methodologies used in program evaluation.Sociology department websites at Doctoral I institutions were more likely to indicate evaluation expertise among faculty members then the other two categories of institutions. Nearly 38 % of Doctoral I institutions mentioned evaluation expertize of faculty members, while only 20 % of sociology department websites at institutions in the remaining sample indicate faculty expertise in program evaluation.

The analysis also showed that sociology faculty members at a number of different colleges and universities are engaged in conducting and publishing evaluation research. The projects in which they have recently been involved range from evaluations of criminal justice initiatives, to drug and alcohol abuse treatment programs, to interven- tions in education, health care, family issues, and a number of other areas. For example, Virginia Aldige Hiday of North Carolina State University recently published an evaluation of a criminal mental health court (Hiday et al. 2013), and Paul Roman from the University of Georgia conducted an evaluation of a substance abuse treatment program (Abraham et al. 2014). Rachel Gordon of the University of Illinois at Chicago evaluated home and center based day care for preschoolers (Gordon et al. 2013) while Derek Griffith from Vanderbilt University evaluated dietary interventions and physical activity among African-American men (Newton et al. 2014). These are just a few examples of the types of evaluations in which sociology faculty are engaged.

Discussion

How Might Evaluation Courses Fit Within the Spectrum of Sociology Programs and Courses?

Bachelor’s Level

Although evaluation jobs generally require an advanced degree, there are opportunities for individuals with bachelor degrees in sociology and other social sciences to assist with evaluation protocols. They might perform tasks such as data entry, archival research, and data collection. A presentation by Roberta Spalter-Roth, Director of Research on the Discipline and Profession at the ASA, includes a graph that shows a

Am Soc (2015) 46:486–499 489

high percentage of responding senior sociology majors reporting that they highlight their ability to use statistical packages in social science on their resumes, yet this is the same skill that these majors were least likely to strongly agree that they had learned (Spalter-Roth 2011). These results show a mismatch between vocational skills used in job searches and the conceptual and methodological skills learned as part of the sociology curriculum. Other skills that sociology majors said that they put on their resumes included writing a report understandable by non-sociologists, interpreting the results of data gathering, developing evidence-based arguments, using computer re- sources to develop reference lists, evaluating different research methods, and identify- ing ethical issues in research (Spalter-Roth 2011). All of these are all skills used in evaluation, yet there is little evidence that they represent a significant proportion of the sociology curricula in most undergraduate departments.

Master’s Level

Master’s degree programs in sociology may be the most logical spot to position evaluation courses because many sociology departments currently offer terminal mas- ter’s degrees that are focused on equipping students to find jobs in applied areas. A 2009 study showed that 64 % of sociology departments offered a free-standing master’s degree, and of those, over half (53 %) offered an applied, a professional, or a clinical track (Spalter-Roth and Scelza 2009). A 2011 survey of directors of graduate studies in sociology showed that almost 50 % of master’s programs offer terminal master’s degrees that are labeled as applied, professional, or clinical degrees (Van Vooren and Spalter-Roth 2011).

In 2009, the ASA Department of Research on the Discipline and Profession conducted a longitudinal study of 224 sociology master’s programs and their outcomes. It focused on students who completed terminal master’s degrees with regard to job satisfaction factors, the adequacy of what respondents indicated that they learned in their master’s programs and what they wished they had learned, and the added value of the master’s degree in sociology. The study found only two factors to be statistically significant in explaining job satisfaction: adequacy of research training and whether or not the job graduates held was closely related to sociology (Van Vooren and Spalter- Roth 2011). A third variable, working in research positions, was positively associated with job satisfaction and had a high correlation with closeness to sociology. According to the authors of the study, this finding suggests that respondents working in research positions consider these jobs to be closely related to their sociological training, and these are also the jobs associated with greater job satisfaction (Van Vooren and Spalter- Roth 2011).

Graduates with master’s degrees in sociology say that they rely heavily on the skills they learn in their research courses and wish they had more specific training. About one-third of graduates of terminal master’s programs wish that they had a better grasp of statistical packages compared to only 17 % of PhD candidates (Van Vooren and Spalter-Roth 2011). According to Van Vooren and Spalter-Roth (2011), an explanation for the difference may be attributed to either master’s programs not offering a enough research methods courses or students not taking these courses because they do not fit their vision of what sociology is or what sociologists do. The latter explanation may be true, since in the 2009 study, students did not say that they went into sociology to learn

490 Am Soc (2015) 46:486–499

research methodology or statistics. In addition, a higher percentage of sociology students said that they sought professional service and management jobs after gradu- ation than said that they sought research or teaching jobs. However, once on the job, they acknowledged that they used research skills more than they originally anticipated. Almost one-third (30 %) of those with terminal master’s degrees spent most of their working hours doing applied or basic research activities (Van Vooren and Spalter-Roth 2011).

A sample syllabus for an evaluation research methods course for master’s level students within a sociology department at a Research I institution is outlined in the Appendix.

Doctoral Level

Doctoral programs in sociology, on the other hand, tend to prepare their students for academic positions rather than applied positions, and may be less receptive than master’s degree programs, to the inclusion of evaluation courses. Data from an ASA survey of PhDs who received their degrees between July 1996 and August 1997 showed a strong preference for employment in tenure-track academic jobs at that time. In fact, the vast majority of respondents reported having received little or no encour- agement to pursue nonacademic careers while in graduate school (American Sociological and Research Program on the Discipline and Profession. Research Brief 2000). Approximately 85 % of the PhD graduates indicated that faculty had not encouraged them to pursue nonacademic jobs, and 63 % reported that they had not had opportunities to interact with nonacademic professionals. Only 18 % of those in nonacademic careers and 13 % of those employed inside the academy were encouraged to pursue such jobs.

While the most common source for academic jobs was the American Sociological A Employment Bulletin, the most common job search method for non-academics was informal networks (e.g., colleagues and friends). Those who found nonacademic employment were more likely to use newspapers (42 versus 27 %) and electronic resources (33 versus 20 %). Faculty advisors were used by half of those who obtained jobs as professors and by only one-third of those in nonacademic jobs. According to a 2000 report entitled New Doctorates in Sociology: Professions Inside and Outside the Academy, those employed in the nonacademic workplace after graduate school at the time were more likely to have worked outside of the academy while in graduate school. Almost half of non-academics in the study reported this to be the case, compared to less than one-third of the academics. As of the late 1990’s, those who began in academia but decide later to pursue a non-academic job found that the cost of switching to jobs in the private sector depends on whether or not they worked at those types of jobs while in graduate school (Mangematin 2000).

Faculty in doctoral programs may not be able to avoid encouraging PhD graduates to pursue nonacademic jobs for a number of reasons. First and foremost, the job market for academic positions continued to be extremely competitive in the early 2000’s, as not as many positions were as available as some had predicted in the early 1990’s (Jones 2002). While a study by Jacobs and Spalter-Roth (2008) suggests that this problem has corrected itself and demand may now outweigh the supply of sociology PhDs, the authors acknowledge that some of these positions are being filled by individuals with

Am Soc (2015) 46:486–499 491

doctorates from other disciplines, perhaps due to the more interdisciplinary nature of modern day sociology. Second, even if recent doctorate holders can work in academia, they can also earn additional income consulting, which might include working as a program evaluator (Jones 2002). In recent years the field has seen more and more PhDs working in both the private sector (Garcia-Quevedo et al. 2012) as well as government jobs (Kim et al. 1998). Some programs are beginning to consider providing doctoral students with more transferable skills (Patel 2014), as skills taught in graduate school often have to extend beyond the classroom (Kim et al. 1998). Furthermore, on income- based indicators, non-academic careers win out. For example, non-academic careers typically offer higher salaries than academic careers. Based on the 1998 data in the research brief, 28.1 of academics vs 19.7 % of non-academics had a salary of less than $35,000 and 19.5 % of academics vs. 47.5 % non-academics expected a salary of over $50,000. Non-academics were more likely to work a 40-h a week job (54 vs. 31 %) and less likely to work over 50 h a week (39 vs. 26 %).

However, culturally speaking, incorporating evaluation courses into sociology doc- toral departments may be a hard sell because it goes against the image of a Ph.D. level sociologist. According to the 2000 Research Brief of the ASA, Ph.D’s need to be in academic positions in order for them to feel that their jobs are related to their studies. Only 13 % of those in non-academic jobs reported that their position was what thy expected to be doing at the start of their Ph.D. training vs 43 % of academics at the time. Academics were almost three times more likely than non-academics to Bstrongly agree^ that their positions were related to professional sociology, and almost twice as likely to strongly agree that their positions were compatible with their education and training.

According to the 2000 Research Brief of the ASA, non-academics also had lower networking scores (31 vs 18 %) than academics. They were significantly less likely than academics to be members of the ASA (60 vs 76 %). In the early career stages of those with doctorates in sociology, the publication rate for those with academic and non-academic jobs was about the same, however the difference was that non-academics published technical or scholarly reports, while academics published articles in peer- reviewed journals. Those in academic settings were significantly more likely to present papers at national or regional meetings, and non-academics who did present papers primarily presented them at professional society meetings other than the ASA or regional sociology meetings.

Given all the barriers to training in evaluation for Ph.D. students in sociology, it is surprising to learn that some see the value of learning evaluation skills. In a presenta- tion by Roberta Spalter-Roth (2011), entitled BFindings from ASA Surveys of Bachelor’s Master’s and PhD recipients: Implications for Departments in a Jobless Recovery,^ Dr. Spalter-Roth includes a chart entitled BSkills Match between Graduate Training and Current Job for PhD Sociologists Working in Applied and Research Settings.^ The chart shows that 51.9 % of the respondents thought that they were undertrained in program evaluation (22 % indicated that their program provided well- matched job skills and training in evaluation and 26 % indicated that they were over- trained.) Additionally, in a list of recommendations entitled, BPhD Respondent Recommendations for Improving Graduate School Curriculum,^ 18.3 % of respon- dents want more information about non-academic careers. Similarly 16.4 % of respon- dents wanted to learn more about various methods of applied or evaluation research,

492 Am Soc (2015) 46:486–499

and 15.1 % want mentoring and networking (outside of academy) including internships and working with clients.

Will There be Jobs Available for Sociologists who Train as Evaluators?

Private evaluation firms want to hire individuals with the skill sets that many sociology students acquire in graduate school. The following are examples of evaluator job announcements from the American Evaluation Association website www.eval.org in December 2014. Note the emphasis on degrees in social science and research methodology skills taught in most sociology departments.

Program Evaluator

BMaster’s degree in social work, public health, or related field. Successful experience in engaging the community in program evaluation, conducting multi method program evaluations, project coordination of complex projects involving multiple participants, and/or on -profit service. Experience with training to develop program evaluation skills. Proficient in Excel and SPSS or other statistical application Experience with ArcGIS preferred. Able to work profes- sionally, independently, and collaboratively with faculty, community clients, students, and staff.^

Senior Evaluator

BMinimum of a Master’s degree (with PhD preferred) in a behavioral or social science and 7+ years of experience in program evaluation. Must have knowledge and experience with mixed methods evaluations, including design- ing and implementing cross-site evaluations; experience with process, out- come, and cost studies; longitudinal data collection; collection and analysis of case-level and administrative level data; use of a variety of evaluation design methodologies and techniques including random assignment and propensity score matching; experience evaluating systems change efforts and conducting evaluations in complex political environments with multiple stakeholders. Must be able to critically review evaluation plans and provide technical assistance to States in strengthening plans. Strong organizational and analyt- ical skills; ability to work effectively as part of a team and independently; and excellent written and oral communication skills are also required. Famil- iarity with State and Federal child welfare legislation, programs, and policy issues is helpful.^

Senior Evaluator

BEducation: Doctoral or master’s degree in a field in which evaluation related skills were learned and practiced. Preference given to candidates who possess or are working towards a doctoral degree in an evaluation related field.

Am Soc (2015) 46:486–499 493

Experience: At least 2 years of data preparation, analysis and reporting experience. Additional Qualifications:

& Excellent proficiency in SPSS and statistical analysis methods is required & Able to flex work schedule based on travel and online conferencing responsibilities & Strong Word and Excel skills & Excellent writing, editing, and proofreading skills & Outstanding organizational and time management skills detailed orientated & Ability to manage multiple large-scale, multi-faceted projects and meet deadlines & Ability to seek out resources to locate and analyze data from secondary data sources & Provide leadership, direction, supervision, and training to evaluation associates and

assistants & Ability to lead a team, delegate responsibilities, and hold others accountable & Ability to work independently to plan project tasks in advance and identify steps

that lead to project completion^

Conclusions

Should sociology departments train students for careers in evaluation? The authors conclude that they should. First, they should because evaluation courses are a natural extension of existing sociological content. Sociology master’s and doctoral programs already train students in systematic inquiry, quantitative and qualitative methods, research design, data analysis and interpretation (Stevahn et al. 2005), all of which are highly desirable to those seeking to hire program evaluators. Sociology has the opportunity to train students, particularly those at the masters and doctoral levels, for careers other than college teaching by offering additional courses in evaluation (Kim et al. 1998). These courses would include skill building in conducting multi-method program evaluations, implementing cross-site evaluations, evaluation systems, evaluation political environments, and engaging the community in evaluations. All of these skills could easily be included in one or two courses on evaluation methodologies.

Secondly, sociology departments should offer evaluation courses because the skills that are taught lead to career opportunities directly related to sociology curricula. Early in this paper, it was mentioned that sociology graduates are more satisfied with their jobs if they are able to use content and skills that they learned in their courses (Van Vooren and Spalter-Roth 2011). Rather than steering sociology graduates to occupa- tions that are only loosely related to sociology, or not at all, why not prepare them for careers that are directly related to traditional sociological content. To illustrate, here is a quote from a letter one of the authors recently received from a former master’s student in her evaluation class.

I’m a graduate research assistant for the evaluation coordinator of the statewide GEAR UP grant. I’m also on the evaluation team for the state evaluation association and will be attending the conference next week. I really appreciate everything I learned in your class and it was probably one of the most useful/ practical SOCA classes I took (even if I didn’t know it at the time). It’s likely that

494 Am Soc (2015) 46:486–499

I will continue doing evaluation work the entire time I’m working on my PhD and beyond.

Thirdly, sociology departments should offer evaluation courses because sociology would benefit from increased representation in the professional associations associated with evaluation. Today the American Evaluation Association has a growing member- ship of 7,700 from multiple disciplines and 52 topic interest groups (http://www.eval. org). National evaluation associations are increasing annually around the world including societies in Africa, Australia, Canada, Europe, the United Kingdom, Malaysia, Central American, and Japan (Preskill and Russ-Eft 2005). Membership in these associations would expand networks for sociologists and would enhance creden- tials for Federal grants and other sources of funding for social re