Background and Objectives
An enduring image of academia is that of an ivory tower, disconnected from the messy problems of the world. It is a recurring complaint that the primacy of basic research and the academy’s emphasis on abstract theory has eroded higher education’s connection to the world, isolating scholars from society and making their work obscure or irrelevant to the general public (Barker & Brown, 2009; Gibson, 2006). Decreasing appropriations for higher education heighten the need to convince the public about the value of university research. In response to prodding from Congress, federal funding agencies are increasingly requiring academic research grant proposals to include indicators of public impact that would result in social good.
In 1997, the National Science Foundation (NSF) established a policy that all funding proposals submitted to the agency would be evaluated on two criteria: intellectual merit and broader impacts. Broader impacts refer to “specific, desired societal outcomes” (NSF, 2012, p. III-2) such as the participation of underrepresented groups in science, technology, engineering, and mathematics (STEM); enhancing STEM education; public scientific literacy and engagement; and partnerships between academia, industry, and others. According to Arden Bement, former director of the NSF, “The [broader impacts] criterion was established to get scientists out of their ivory towers and connect them to society” (quoted in Lok, 2010, p. 416).
A national review of the effectiveness of these criteria conducted in 2000 revealed that, while most researchers had little difficulty identifying the intellectual merit of their plans, many struggled to adequately articulate the broader impacts of their proposed investigations (NSF, 2005). The study also found that the broader impacts criterion was consistently weighted less than intellectual merit in the proposal review process and that many in the scientific community were resistant to or dismissive of the requirement (NSF, 2005). The NSF responded with efforts to educate the scientific community about the agency’s rationale and expectations for broader impacts. A subsequent national evaluation of the NSF’s review criteria conducted in 2010 found that problems with the execution, understanding, and acceptance of the broader impacts criterion persisted, that assessment was unclear and inconsistent, that there was little variety in the type of activities performed to address the broader impacts criterion, and that principal investigators (PIs) needed greater institutional support to respond effectively to this requirement (NSB, 2011).
The purpose of this study was to investigate the factors that shape PI response to the NSF broader impacts criterion. The following research questions guided the study:
What are the different types of activity that PIs engage in to meet the broader impacts criterion of NSF-sponsored research?
What is the quality of the broader impacts activities, as assessed by PIs using evaluation criteria for the scholarship of engagement?
To what extent do PIs’ individual characteristics (expertise, epistemology, academic discipline, and rank) and institutional support (climate and resources) for community engagement predict the type of their broader impacts activities?
To what extent do PIs’ individual characteristics and institutional support predict the quality of their broader impacts activities?
The author gathered survey data to identify the types of broader impacts activities that PIs conduct, and their perceptions of the quality of these activities. The author then determined how much variance in type and quality of activity could be explained by the respondents’ personal characteristics and perceptions of institutional support for community engagement. In the survey, community engagement was defined as “The collaboration between institutions of higher education and their larger communities (local, regional/state, national, global) for the mutually beneficial exchange of knowledge and resources in a context of partnership and reciprocity” (Carnegie Foundation for the Advancement of Teaching, n.d., n.p.).
This study is grounded in the theoretical framework of the scholarship of engagement. Such scholarship is academically relevant faculty work that simultaneously meets campus goals and community needs, incorporates community issues, and is integrative across teaching, research, and service (Clearinghouse and National Review Board for the Scholarship of Engagement, 2002). Although there are subtle distinctions between the terms “scholarship of engagement,” “engaged scholarship,” and “community-engaged research,” they are used interchangeably throughout this study.
The NSF’s broader impacts merit review criterion has been an ongoing challenge within the scientific community, particularly in terms of conceptual clarity, assessment, and philosophical resistance. The persistence of the problem may stem, in part, from the paucity of literature on the broader impacts of academic research (Buxton, 2011). While scant research has been conducted on such impacts, volumes have been written on the scholarship of engagement. The engagement movement has spawned multiple journals, institutes, conferences, and consortia dedicated to studying engaged scholarship in its many forms. This study drew from this extensive literature base to better understand the issues that surround the broader impacts of academic research and connections between the academy and society. Framing the study within the context of the scholarship of engagement is appropriate not only because broader impacts activities may be viewed as an expression of engaged scholarship, but also because both struggle to achieve legitimacy and support within the academy.
The Scholarship of Engagement
The concept of engaged scholarship is often credited to Ernest Boyer, a renowned leader of educational reform. Boyer (1990) argued that the traditional definition of scholarship as research that advances disciplinary frontiers of knowledge was too limited. He countered the prevailing hierarchical view of scholarship with a more inclusive vision that added to the ranks of conducting original research such activities as identifying connections between concepts, bridging theory and practice, and effectively communicating knowledge. Boyer posited that faculty scholarly work included four “separate, yet overlapping, functions” (1990, p. 16) which he identified as the scholarships of discovery, integration, application, and teaching. Boyer (1996) proposed the term scholarship of engagement as interaction across these four realms to address community needs. He described the scholarship of engagement as “connecting the rich resources of the university to the most pressing social, civic, and ethical problems…creating a special climate in which the academic and civic cultures communicate more continuously and more creatively with each other” (pp. 19- 20). Boyer asserted that it was time for higher education to renew its covenant with society and to partner with communities in directly addressing problems, with a reciprocal flow of knowledge.
Models of Knowledge Production
Epistemologies about knowledge production and distribution shape perceptions about the role of higher education in society, define relationships between campus and community, and govern the values, norms, and practices of scholars. Boyer raised critical questions of how knowledge is constructed and what is accepted as legitimate knowledge in the academy. Deeply embedded in academia is the traditional linear model of knowledge production and distribution, which prizes basic research that is disciplinary, removed from influence by outside interests, and conducted without consideration of use (Bush, 1945). Boyer countered the prevailing belief that basic research was the most essential form of scholarly activity, with publications and teaching flowing from it. He argued that “knowledge is not necessarily developed in such a linear manner. The arrow of causality can, and frequently does, point in both directions. Theory surely leads to practice. But practice also leads to theory” (1990, p. 16). The national dialogue that ensued prompted academics to identify new models of knowledge creation, including that of use-inspired research (Stokes, 1997) and Mode 2 science (Gibbons et al., 1994).
The dominant, linear model of knowledge production begins with basic research, which may lead to applied research and technological development, and on to production or operations. This model has shaped U.S. science policy since it was first proposed by Vannevar Bush in his 1945 report to President Roosevelt (Mazuzan, 1994). Bush saw an inherent tension between the goals of understanding and use, and warned that the creativity of basic research would be stifled by premature thoughts of practical use.
One year after Boyer’s call for engaged scholarship, Stokes challenged the traditional dichotomy between basic and applied research. Stokes (1997) argued that Bush’s premise that flows between science and technology are uniformly one way, from scientific discovery to technological innovation, was flawed and offered a limited understanding of how knowledge is generated and put to use. Stokes contested Bush’s cannon that the goals of understanding and use are dichotomous, arguing that research is often influenced by both goals. In his model, theoretical and practical research and application come together to create a dynamic cycle of innovation driven by changing conditions and the competitive landscape.
Gibbons and his co-authors (1994) offered another alternative to the static linear model of knowledge production. They determined that Bush’s old paradigm of discipline-specific, autonomous, university-grounded scientific discovery (“Mode 1”) was being supplanted by a new approach to knowledge production (“Mode 2”). Gibbons et al. posited that the new mode emerged from a parallel expansion in the demand for specialist knowledge and the number of potential knowledge producers, adding government laboratories, industry, consultancies, and others to the domain formerly ruled by universities. They described this new mode as socially distributed, application-oriented, trans-disciplinary, and subject to multiple accountabilities. This evolution, they argued, occurred in response to significant trends changing the research environment: the steering of research priorities, the commercialization of research, and the accountability of science. As a result, the research process was no longer an objective investigation, but rather an intense dialogue between research actors and subjects.
In the dominant, technocratic culture of higher education, knowledge flows in one direction: from credentialed, detached experts ensconced in the university to its place of need and application in the community (Hartley, Saltmarsh, & Clayton, 2010). The scholarship of engagement challenges the belief that knowledge is produced by academics and then transferred to the community. Instead, engaged scholarship builds reciprocal relationships between scholars and community partners, dynamic synergies between theory and practice, and research that benefits both the community and the academy (Nicoreta, Cutforth, Fretz, & Thompson, 2011). Such reciprocity has been defined as “democratic engagement” and described as “an epistemological shift…that favors mutual deference between lay persons and academics” (Hartley, Saltmarsh, & Clayton, 2010, p. 401).
The study’s conceptual model, depicted in Figure 1, illustrates the congruence of NSF goals and broader impacts activities (bulleted lists) with Boyer’s domains of scholarship and the models of knowledge production described above. The model aligns these elements along the oppositional axes of democratic versus technocratic orientations and the goals of improving understanding versus improving technology.
Within the context of this conceptual framework, the author explored the relationships between the individual characteristics of faculty members, support at their home institutions, and the type and quality of their broader impacts activities. The scholarship of engagement literature suggests possible associations between engagement in public scholarship and individual characteristics such as epistemology (Austin & Beck, 2010) and expertise (Baskurt, 2011), as well as demographic factors including discipline (Lunsford & Omae, 2011) and rank (Glass, Doberneck, & Schweitzer, 2011). The literature also indicates that institutions influence individual behavior by means of overall climate (Vogelgesang, Denson, & Jayakumar, 2010) and the allocation of resources for administrative support, faculty development, and incentives (Doberneck, Brown, & Allen, 2010). Figure 2 outlines the variables investigated in this study.
The population for this study was the 3,635 faculty members at institutions of higher education across the United States who received NSF research grants during Fiscal Year 2009 (October 1, 2008 – September 30, 2009). This period was selected because it not only follows on the heels of the NSF’s 2007 release of broader impacts criterion guidelines, but it also allows sufficient time for the completion of most projects. Research grant is a term used by the NSF to represent what may be considered a typical research award, particularly with respect to the award size (NSF, 2009). Education research grants are included in this category. Excluded are large awards such as centers and facilities, as are equipment and instrumentation grants. Also excluded are grants for conferences and symposia, grants in the Small Business Innovation Research program, Small Grants for Exploratory Research, and education and training grants. EAGER and RAPID grants were also excluded as they are exempt from external review.
A stratified random sample of 700 grant recipients was drawn from the population. This sample size was chosen to provide a pool of respondents large enough to restrict error to 5%, assuming a response rate of approximately 50% (Patten, 2002). A stratified random sample design was selected to reduce normal sample variation and to produce a sample more likely to resemble the total population than a simple random sample. Given the importance attributed to academic discipline in the literature, the sample was stratified along NSF Directorates to provide representation across a broad spectrum of disciplines. One hundred grant recipients were selected from each of the following directorates: Biological Sciences; Computer and Information Science and Engineering; Education and Human Resources; Engineering; Geosciences; Mathematics and Physical Sciences; and Social, Behavioral, and Economic Sciences. The population of grant recipients in each directorate was assigned consecutive numbers and the desired sample was drawn using a table of random numbers.
As no instrument existed to collect the data necessary for this study, the author developed a special purpose survey derived from an extensive review of the literature. The survey items generated fell into five subsections: broader impacts activities, epistemology, expertise, institutional support, and demographics. The first section asked participants about the type and quality of their broader impacts activities and community partnerships. The second section attempted to capture participants’ epistemologies by inquiring about their views on knowledge production and the democratization of science. The section on expertise asked participants about their prior experience, training, and sense of ability with activities similar to those conducted to address the broader impacts criterion, both as graduate students and as faculty members. The next section asked participants about the existence of a supportive climate and resources for community engagement at their home institutions. The final section of the survey gathered demographic information on academic discipline and rank. The questions on discipline, rank, and activity type were multiple choice; the remaining items used a five-point scale, ranging from 1 (not at all) to 5 (fully).
The next step in survey development was a pre-survey evaluation conducted by a panel of field experts in order to identify and correct any foreseeable problems prior to executing the survey. The panel members were selected based on their expertise in engaged scholarship, survey design, and sponsored research. The panel provided input on the conceptual accuracy of the items, the length and legibility of the scale, and any missing or misrepresented dimensions of the constructs to be studied. After integrating feedback from the expert panel into the instrument, the survey received an exempt review and approval from the Institutional Review Board of the University of South Dakota.
This cross-sectional study used an online survey to collect data from faculty at academic institutions across the United States. The primary concern about web-based surveys is that they may introduce error if online use is not universal. This concern does not apply to the study at hand because the sample population, university faculty, typically demonstrate competency in the use of email and other forms of online communication (Shannon, Johnson, Searcy, & Lott, 2002). Consequently, given considerations of convenience and economy, a web-based survey was deemed the most expedient approach to data collection.
The survey was formatted online using the survey software QuestionPro. The researcher collected email addresses for members of the sample from the NSF website and sent each member an email message explaining the project, inviting their participation, and directing them to the survey web link. The survey was distributed on February 4, 2013, and remained open through March 1, 2013. Follow-up with non-respondents consisted of reminder emails. All sample non-respondents were sent a first, second, and final email reminder after weeks one, two, and three, respectively. QuestionPro’s Respondent Anonymity Assurance was enabled on the survey to ensure that, although computer generated identification numbers for individuals were created, the researcher did not have simultaneous access to both a respondent’s email address and response data.
The Statistical Package for the Social Sciences was used for all data analyses and the .05 level of significance was used for all inferential statistics.
Research question 1. What are the different types of activity that PIs engage in to meet the broader impacts criterion of NSF-sponsored research? Type of broader impacts activity was a categorical variable. The categories, identified by the NSF, were education, broadening participation, infrastructure, dissemination, and other. Frequencies of responses to the question about type of activity were tabulated to derive percentages of participation in each activity.
Research question 2. What is the quality of the broader impacts activities, as assessed by PIs using evaluation criteria for the scholarship of engagement? Quality of broader impacts activity was a summed score of responses to the questions about each area of assessment (goals, preparation, methods, results, presentation, critique, and ethics of the activity, as well as the reciprocity, communication, structures, and duration of community partnerships). Means and standard deviations generated from these items, and composite means for each element, provided information related to the areas of relative strength and weakness of broader impacts activities.
Research question 3. To what extent do PIs’ individual characteristics (expertise, epistemology, academic discipline, and rank) and institutional support(climate and resources) for community engagement predict the type of their broader impacts activities? Expertise was a summed score of questions about the respondent’s prior experience, training, and sense of ability related to community-engaged activities. Epistemology was a summed score of questions about the respondent’s views on knowledge production and the democratization of science. Responses were grouped into two categories: democratic and technocratic orientations. Climate was a summed score of questions about the respondent’s perceptions of institutional climate as expressed through rhetoric, recognition, and rewards for community engagement. Resources was a summed score of questions about the respondent’s perceptions of the administrative support, faculty development, and incentives available for community engagement. Academic discipline was operationalized as the NSF directorate to which the respondent applied. Academic rank was captured with dummy variables for assistant professor, associate professor, full professor, research scientist, and other.
After reverse coding negatively worded items and removing cases with missing observations on multiple indicators, a multiple logistic regression was conducted to determine how the variables ranked in terms of highest to lowest when it came to predicting the type of broader impacts activity.
Research question 4. To what extent do PIs’ individual characteristics and institutional support predict the quality of their broader impacts activities? Quality of activity was a summed score of assessment questions based on the scholarship of engagement criteria. The independent variables were the constructs identified in the analysis of research question 3 (individual characteristics and institutional support). Because the dependent variable was interval-scaled, a multiple regression analysis was employed. A stepwise multiple regression determined how much of the variance in quality of broader impacts activity scores could be explained by the respondents’ expertise, epistemology, rank, discipline, institutional climate, and resources.
The rate of response to the study’s survey was low. Of the 700 surveys distributed, 15 were mail returns, 246 were viewed, 209 were started, and 124 were completed. The completed response rate was 18.1%. Although the limited number of cases restricted the options for statistical analysis, the low response rate did not necessarily invalidate the survey. An examination of the results of 81 national surveys with response rates ranging from 5% to 54% revealed that surveys with much lower response rates were only minimally less accurate (Holbrook, Krosnick, & Pfent, 2007).
Of the 124 respondents who completed the survey, half held the academic rank of full professor (n = 62, 50.0%), while the bulk of the rest were associate professors (n = 40, 32.3%), with some assistant professors (n = 14, 11.3%). Table 1 reports the frequencies and percentages of the academic rank categories.
The degree to which the sample is representative of the population of NSF grantees is unknown because the NSF does not track the academic rank of their grantees. The NSF does, however, categorize their grantees as either “Early Career PIs” (investigators within seven years of receiving their last degree) or “Later Career PIs” (those who received their last degree more than seven years before the award date). In FY 2009, 78% of NSF grantees were classified as Later Career PIs; 22% were Early Career PIs (NSB, 2010). The fact that the promotion from Assistant to Associate Professor generally occurs within six years on the tenure track suggests that the distribution of the sample is reasonably representative of the population.
The respondents were fairly evenly distributed across the seven NSF directorates, as Table 2 depicts. The Directorate of Biological Sciences had the largest representation (n = 27, 21.8%); the next five directorates were closely clustered, with the number of respondents ranging from 19 (15.3%) to 15 (12.1%). The Directorate of Geosciences had the fewest respondents (n = 11, 8.9%).
Type of Broader Impacts Activity
Research question 1 asked about the different types of activity that PIs engage in. The categories of activity, identified by the NSF, were education, broadening participation, infrastructure, dissemination, and other (NSF, 2008a). Frequencies of responses to the question about type of activity were tabulated to derive percentages of participation in each activity. Education was by far the most common activity (n = 71, 57.3%), while dissemination was the second most reported activity (n = 29, 23.4%). The remaining categories were substantially lower. Descriptive statistics generated from this item are presented in Table 3.
Quality of Broader Impacts Activities
Respondents assessed the quality of their broader impacts activities in terms of goals, preparation, methods, results, presentation, critique, and ethics of the activity. The response scale was 1 (not at all) to 5 (fully). Respondents expressed the greatest confidence in their selection and use of methods (composite M = 4.18, SD = 0.64). High scoring items included “I used methods appropriate to my goals” (M = 4.36, SD = 0.65) and “I effectively applied the methods selected” (M = 4.11, SD = 0.79). Scores were also relatively high for preparation (composite M = 4.16, SD = 0.82), with the leading item being “I brought together the skills and resources necessary to move the project forward” (M = 4.30, SD = 0.74). Interestingly, the area of ethics included both the highest and lowest scoring items: “I followed all policies concerning the responsible conduct of research” (M = 4.61, SD = 0.74) and “I subjected my activity to university IRB review” (M = 3.10, SD = 1.82).
The area that showed the greatest weakness was critique (composite M = 3.49, SD = 0.98). The lowest scoring items in this category were “I used an appropriate breadth of evidence for the critique” (M = 3.39, SD = 1.11) and “I used this evaluation to improve the quality of future work” (M = 3.51, SD = 1.01). Respondents also rated themselves lower in using the results (composite M = 3.87, SD = 0.80). The items “I presented the results to both academic and non-academic audiences” and “The results of the activity added consequently to the field” received the relatively low mean scores of 3.73 (SD = 1.11) and 3.76 (SD = 0.97), respectively.
Thirty-eight respondents (30.6%) reported that their broader impacts activities involved collaboration with a specific community agency (such as a school, museum, organization, or other group). These respondents assessed their community partnerships in terms of reciprocity, communication, structures, and duration of the collaboration. The scoring for this area of assessment was noticeably lower than the others (composite M = 3.07, SD = 1.07). The highest scoring items were “Communications between the community partner and I were ongoing and bidirectional” (M = 3.53, SD = 1.13) and “The relationship continued after the project ended” (M = 3.47, SD = 1.43). The items receiving the lowest ratings were “A formal agreement outlined roles and responsibilities” (M = 2.55, SD = 1.33) and “The community partner and I shared power and responsibility equitably” (M = 2.74, SD = 1.25).
The literature suggests possible associations between engagement in public scholarship and personal and institutional characteristics. This study investigated these variables as predictors of type and quality of broader impacts activities.
Expertise. This variable was a summed score of questions about the respondents’ prior experience, training, and sense of ability related tocommunity-engaged activities. Respondents indicated that they had a good deal of prior experience with activities similar to their broader impacts work as faculty members (M = 4.01, SD = 1.01), but they had received little faculty training for such work (M = 2.72, SD = 1.22). As graduate students, they received even less training (M = 2.39, SD = 1.35) and had less experience (M = 2.78, SD = 1.39). Respondents reported that they had a relatively strong sense of ability to implement broader impacts activities (M = 3.75, SD = 0.90), but they were less confident in their ability to evaluate these activities (M = 3.27, SD = 1.09).
Epistemology. This variable was a summed score of questions about the respondents’ views on knowledge production and the democratization of science. Responses were grouped into two categories: democratic and technocratic orientations. A technocratic orientation is characterized by the valuing of basic, discipline-specific, curiosity-driven research conducted exclusively by highly specialized academic experts. A democratic orientation values applied, trans-disciplinary, problem-centered, use-inspired research that welcomes collaboration with partners outside the academy. Survey responses indicated orientations that were more democratic (M = 3.22, SD = 0.68) than technocratic (M = 2.85, SD = 0.68). Respondents tended to view public engagement in science as a way to increase acceptance of the results and generate information that meets user needs (M = 3.85, SD = 0.93), and to take into account public values and contexts as well as local expertise (M = 3.09, SD = 1.13). Respondents did not agree that public ignorance should be corrected through a one-way flow of information from the academy to the community (M = 2.17, SD = 1.01), or that scientific research would be most productive if left to scientists alone (M = 2.49, SD = 1.21).
Climate. This variable was a summed score of questions about the respondents’ perceptions of institutional climate as expressed through rhetoric, recognition, and rewards for community engagement. The overall mean for this variable was moderate, at 3.02 (SD = 0.79). Respondents indicated that the executive leadership at their institutions explicitly promoted community engagement as a priority (M = 3.42, SD = 1.11), but their promotion and tenure policies did not reward such research (M = 2.15, SD = 0.93).
Resources. This variable was a summed score of questions about the respondents’ perceptions of the administrative support, faculty development, and incentives available for community engagement. The overall mean for this item was low, at 2.31 (SD = 0.82). Respondents reported that faculty training was not available in engaged scholarship (M = 2.19, SD = 1.04) and that there was little financial support (M = 2.20, SD = 1.10) or infrastructure to facilitate collaboration (M = 2.38, SD = 1.07) and assessment (M = 2.40, SD = 1.06) of community engaged activities.
Influence on Type of Activity
A multinomial logistic regression analysis was conducted to predict the type of broader impacts activity the respondents engaged in, using expertise, epistemology, institutional support, discipline, and rank as predictors. A test of the full model against a constant only model was significant, indicating that the predictors, as a set, reliably distinguished between types of broader impacts activity, χ²(52, n = 102) = 87.61, p = .001. A Nagelkerke’s R² of .644 indicated a moderately strong relationship between prediction and grouping. The model’s overall prediction success was 68.3%. The likelihood ratio tests demonstrated that expertise, resources, and discipline contributed significantly to the model; expertise emerged as the strongest predictor (based on p value). Table 4 presents the contribution of each variable to the model.
Influence on Quality of Activity
A multiple regression analysis revealed that the linear combination of individual characteristics and institutional support was significantly related to quality, F(15,84) = 5.16, p = .000. The sample multiple correlation coefficient (R²) was .70, indicating that approximately 49% of the variance in quality could be accounted for by the linear combination of individual characteristics and institutional support. A stepwise multiple regression presented a more parsimonious model with an equally significant regression equation, F(5,90) = 14.89, p = .000, and an adjusted R² of .42.
The stepwise model revealed that expertise accounted for 20% of the variance, and that the inclusion of climate into the model resulted in an additional 8% of the variance being explained. Table 5 presents indices of the relative strength of each predictor. These results suggest that faculty expertise is the greatest predictor of the quality of broader impacts activities. Aninstitutional climate supportive of community engagement and a democratic attitude toward knowledge production are also predictors of high quality broader impacts activities. The results also indicate that full professors are more likely to report high quality broader impacts activities, while researchers who submit to the NSF Directorate of Biological Sciences are less likely to do so.
The survey concluded with an invitation for participants who wished to comment on the survey to do so in an open text box. Feedback from participants covered the following topics: appreciation for the study (4); suggestions to improve the study by clarifying the language (5) and allowing respondents to select multiple options (4); and complaints about the broader impacts criterion in terms of how the NSF manages it (2), how researchers respond to it (2), and the need for support (5). The strongest feedback focused on the need for support with broader impacts activities. This concern was expressed both in the comment box as well as through email responses to the author from sample members who chose not to complete the survey.
All of the individual traits postulated in the literature to influence faculty community engagement examined in this study emerged, to varying degrees, as determinants of the broader impacts activities of academic faculty.
Expertise. Broader impacts activities generally push investigators beyond their particular areas of specialization and require additional skill sets. Many scientists believe they do not have the skills necessary to develop appropriate outreach activities (Lok, 2010; Mathieu, Pfund, & Gillian-Daniel, 2009; Tretkoff, 2007). This theme pervading the literature was supported by the study’s finding that respondents had received little training to develop and implement work similar to their broader impacts activities. Sample members in the survey and email responses to the researcher also reiterated this belief.
The data analysis revealed that expertise was the strongest predictor of the type and quality of broader impacts activities. Because this variable emerged as such a significant predictor, the author repeated the analysis with a more conservative computation of faculty expertise by removing the items referring to prior experience and training as a graduate student. This narrower perspective on expertise did not diminish the significance of this variable in any of the regression analyses. The survey findings and supplemental responses underscore the need for faculty development and training in broader impacts, as well as institutional support to assist with implementation and evaluation.
Epistemology. The influence of epistemology was manifested in this study in more than one way. Negative bias in the scientific community against the broader impacts criterion appears to have limited the survey response rate. Several members of the sample opted to reply to the author with negative comments rather than complete the survey. The following example of a response to the author’s email invitation to complete the survey provides anecdotal evidence of persistent resistance to the broader impacts criterion:
“Most everyone I know that submits to NSF *HATES* to come up with arguments about broader impacts for NSF proposals. They hate doing it, they hate thinking they had to do it, etc. So, when a request like yours comes in, most folks can’t delete your unwelcome spam-like message fast enough.”
The literature reviewed documents with similar negative responses from researchers, describing the broader impacts criterion as burdensome, counterproductive, and punitive (Tretkoff, 2007); confusing and frustrating (Sarewitz, 2011); irrelevant, ambiguous, and poorly worded (Holbrook & Frodeman, 2005); and undermining the autonomy of the scientific enterprise (Holbrook & Frodeman, 2011).
Some of this resistance stems from the fact that the revisions to the merit criteria had little impetus from the scientific community, but were instituted in response to Congressional prodding (Bozeman & Boardman, 2009). The broader impacts criterion is interpreted by some as an introduction of extraneous political, cultural, or economic concerns into basic research (Holbrook & Frodeman, 2007); bringing considerations external to science into the peer review process (Holbrook, 2005), and politicizing the value-neutral pursuit of science (Holbrook & Frodeman, 2007). Holbrook and Frodeman (2007) suggested that agitation over the broader impacts criterion is to be expected from anyone who believes that beneficial societal impacts will eventually follow from a laissez faire approach to basic research. Thus, reaction to the broader impacts criterion may depend on one’s conception of knowledge production, one’s views on the democratization of science, and the degree to which one supports the Bush model of opposition between basic and applied research.
Survey responses revealed that epistemology shaped participants’ response to the broader impacts criterion, with a democratic orientation towards knowledge production emerging as a predictor of the quality of broader impacts activities. A democratic orientation recognizes that public engagement increases the acceptance of research results, generates information that meets user needs, accounts for public values and knowledge, and improves the quality of science policy decisions. Participants shared an orientation towards knowledge production that was more democratic than technocratic. Yet, despite this appreciation for the socially embedded context of application, respondents favored research that was curiosity-driven rather than use-inspired. The apparent incongruity between respondents’ valuing of public engagement to increase the acceptance of results and meet user needs, while favoring research that is curiosity-driven rather than use-inspired, may indicate an emerging culture shift. Researchers appear to be embracing an alternative value system that acknowledges the need for public engagement while clinging to traditional norms regarding what constitutes proper academic science.
A similar dynamic tension was evident in questions about how knowledge is generated. Survey respondents indicated a strong belief that knowledge is generated through trans-disciplinary collaboration that brings together multiple sources of distributed knowledge, a view characteristic of a democratic orientation. Interestingly, they also expressed the concurrent belief that knowledge is generated through objective disciplinary expert-led investigation, suggesting they did not perceive these opposing approaches to knowledge production as dichotomous.
Rank. While prior research suggested a significant relationship between an investigator’s professional career status and engagement in community-based research, the direction of the relationship was inconclusive. It has been reported that the types of publicly engaged scholarship faculty members pursued varied by rank in statistically significant ways, with full professors more likely than associate professors to report such work (Glass, Doberneck, & Schweitzer, 2011); that lower ranked faculty were more frequently involved in public service (Antonio, Astin, & Cress, 2000); that faculty members of various ranks reported using scholarship to address community needs at similar levels (Vogelgesang, Denson, & Jayakumar, 2010); and that tenured and untenured faculty showed similar levels of frequency in conducting the scholarship of engagement (Braxton & Luckey, 2010). Some argue that junior faculty trained in an environment where engaged scholarship is increasingly viewed as an important part of the mission of modern universities are more likely to engage than senior faculty who were not trained in such an environment ( Jaeger & Thornton, 2006). Others contend that successful senior scholars with tenure have the flexibility and freedom to pursue non-traditional research (Scott, 2007). One finding that is consistently reported is that junior faculty members are routinely counseled to avoid service activity that may interfere with their research productivity (Stanton & Wagner, 2010).
This study found that academic rank was a predictor of the quality of broader impacts activities. The finding that being a full professor was a significant predictor of quality aligns with reports in the literature that full professors are more likely than associate professors to conduct community engaged work (Glass, Doberneck, & Schweitzer, 2011) and that successful senior scholars with tenure have the flexibility and freedom to pursue non-traditional research (Scott, 2007).
Discipline. The literature suggests that the ways in which faculty members define, design, and enact their community-engaged scholarship are largely driven by discipline (Doberneck, Glass, & Schweitzer, 2010). This aligns well with the study’s finding that discipline contributed significantly to the predictive model for type of broader impacts activity. In this study, discipline was operationalized as the NSF directorate to which the researcher applied. The results revealed that researchers who submitted to the NSF Directorate of Biological Sciences were less likely to report high quality broader impacts activities. This supports various reports in the literature that community engagement was less prevalent in the physical sciences (Lunsford & Omae, 2011; Vogelgesang, Denson, & Jayakumar, 2010).
One of the key predictors of engaged scholarship is institutional practices and policies that support engagement (Lunsford, Bargerstock, & Greasley, 2010). Faculty motivation to engage in public scholarship is influenced by institutional characteristics including mission, resources, norms, and evaluation policies (Colbeck & Wharton-Michael, 2006). Some theorists have noted that institutional characteristics have a greater impact than personal motivation and commitment when it comes to scholarly productivity, and that faculty productivity may be stimulated or stifled by organizational culture and administrative structure (Freedenthal, Potter, & Grinstein-Weiss, 2008). A study based on the Higher Education Research Institute’s 2004-2005 national survey of college and university faculty revealed that the perception of institutional support matters, even above and beyond the individual dispositions of faculty members, and even when disciplinary culture is accounted for. Every one-step increase in perceived institutional support was associated with a 10% increase in the likelihood of a faculty member collaborating with the local community in research or teaching (Vogelgesang, Denson, & Jayakumar 2010). Institutional support can be conveyed through both climate and resource allocation.
Climate. Campus climate is a function of interpersonal interactions and includes dimensions such as “perceptions, attitudes, and expectations” (Cress, 2002, p. 390). A multiple regression analysis revealed that an institutional climate supportive of community engagement is a strong predictor of high quality broader impacts activities, accounting for 8% of the variance explained by the model. The literature identified the university reward system as the greatest obstacle to community engaged research (Pfirman, Collins, Lowes, & Michaels, 2005). Responses to the author’s survey concurred that promotion and tenure policies were the feature of institutional climate least supportive of community-based research.
Resources. While faculty recognition and rewards are important motivators to initiate change, organizational structures must be in place to sustain the work (Brukardt, Holland, Percy, & Zimpher, 2004; Stanton, 2007). When the NSF asked the scientific community what role their institutions should play to ensure that broader impacts are realized, the majority response was that home institutions should increase support for broader impacts by establishing mechanisms for those activities, increasing financial or in-kind support, or encouraging involvement through changes in value systems and incentives (NSB, 2011). The allocation of resources for administrative support, faculty development, and incentives is the second largest barrier to engaged scholarship (Pfirman et al., 2005). Although resources emerged as a strong predictor of type of broader impacts activity, participants reported limited access to training or infrastructure to facilitate collaboration and assessment of community- engaged activities.
Type of Broader Impacts Activity
NSF guidelines to proposers and reviewers identified general areas of broader impact: education, broadening participation, enhancement of infrastructure, broad dissemination, and other social benefit (NSF, 2008a). Despite admonitions from the NSF that PIs “are expected to go beyond their normal teaching duties and faculty commitments” (NSF, 2008b, p. 1), the category of education is by far the most common broader impacts activity, often with researchers simply repackaging what they are already doing (Lok, 2010). The chemistry division director at the NSF reported that “Overwhelmingly, the number one broader impacts that most people in the chemistry division are using is ‘training graduate students and postdocs’” (Lok, 2010, p. 417). In 2010, the National Science Board (NSB), the agency charged with NSF oversight, commissioned a topic modeling of the project summaries of approximately 150,000 NSF proposals. The analysis revealed that the education category occurred in more than 60% of the proposals (NSB, 2011). This was roughly three times the size of the next largest category. The results of the present study were congruent with the topic modeling; education was by far the most commonly reported activity at 57.3%.
The America COMPETES Reauthorization Act of 2010 (P.L. 111-358), which provided reauthorization for the NSF, established new goals and policies for the broader impacts criterion. The Act stipulated that the NSF must apply this criterion to achieve an array of societal goals, including increasing economic competitiveness; developing a globally competitive workforce in science, technology, engineering, and math (STEM); expanding the participation of women and underrepresented minorities in science and engineering; increasing partnerships between academia, industry, and others; improving STEM education at PK-12 and undergraduate levels, including teacher development; increasing public scientific literacy; and expanding national security (Sec. 526[a]-). The NSF integrated this expanded vision of broader impacts into a new Proposal and Award Policies and Procedures Guide (NSF, 2012). Congress’ identification of eight national goals to which broader impacts activities should be directed underscores the need for greater variation in the types of broader impacts activities PIs pursue. The results of this study suggest that limited resources and training constrain researchers to engaging in broader impacts activities that are the most familiar and least demanding (i.e. education).
Quality of Broader Impacts Activity
Hampering effective peer review of broader impacts is the fact that no standards have been set and no evaluation target provided (Bozeman & Boardman, 2009). The NSF does not systematically track how its broader impacts requirements are being met, nor does it have a system in place to evaluate project effectiveness (Lok, 2010). The first comprehensive analysis of the NSF’s review criteria, conducted in 2000, recommended that the NSF develop an evaluation strategy based on measures and performance indicators to track the objectives and implementation of the new criteria supported by both qualitative and statistical data collection methods capable of measuring incremental movement towards achieving the NSF’s strategic goals (National Academy of Public Administration, 2001). One of the six major themes of the second national review of the NSF criteria, conducted 10 years later, was that post-award assessment of broader impacts activities was weak and should be improved (NSB, 2011). Given the variation in size and scope of projects, the report indicated that the effectiveness of broader impacts would be more meaningful if they were aggregated at a higher level than the individual project.
Without established metrics and data collection, it is impossible to objectively assess the broader impacts of sponsored programs. Instead, this study asked PIs to evaluate the processes with which they devised and implemented their broader impacts activities using the standards of scholarship (clear goals, adequate preparation, appropriate methods, significant results, effective presentation, reflective critique, and ethical conduct). Given the subjectivity of self-assessment, better objective metrics of the quality of broader impacts activities are clearly needed. The criteria used in this study appeared to engender a social desirability bias. Respondents showed little variation in their high ratings of the quality of their broader impact activities, presumably because they wanted to believe that they were practicing good scholarship. It is likely that those who reported working with a specific community agency were more critical of these partnerships because such activity is traditionally less highly regarded in the academy and thus subject to a more frank assessment. Identifying valid and feasible ways to assess research impact more objectively should be a priority of future studies. Self-reported information needs to be balanced with multiple kinds and sources of data. An important contribution to the field would be the development of multiple well-defined measures, consisting of both hard and soft data, which can be systematically analyzed.
This study makes several contributions to the field of research administration. By providing a snapshot of the type and quality of broader impacts activities conducted by PIs, it enhances understanding of the training and support faculty members need to successfully meet the broader impacts requirements of federally sponsored research. In addition to data collection, the study examined the associations between relevant personal, professional, and institutional factors, and broader impacts activities, identifying patterns and highlighting barriers and facilitators to guide the decision making of higher education administration as well as federal funding agencies seeking to expand or enhance the broader impacts of university research. This study was designed to stimulate discussion among higher education administrators about community engagement, inform initiatives to expand interactions between campuses and external entities, generate strategies to improve the quality of broader impacts activities, and identify faculty development objectives that are scholarship-based. Linking broader impacts activities with forms of engaged scholarship reinforces the value of such work and establishes new indicators for the institutionalization of engagement at research universities.
The results of the study also hold practical implications. Expertise emerged as the strongest predictor of broader impacts activity type and quality, yet respondents reported receiving little training in such work. The study identified the need for faculty professional development in effective evaluation, putting the results to use, establishing clearly defined partnerships with community agencies, and equitably distributing power and responsibility with community partners. Such training need not be limited to faculty. Respondents reported that they had received little to no training and experience in community engagement as graduate students. Because graduate school is where researchers are socialized as scholars and where they develop their professional identities, extending training to graduate students and postdoctoral researchers will help create a cadre of professionals who conduct high quality broader impacts activities as a natural part of their research programs.
While there is clearly a need for training, faculty may be reluctant to pursue it. This disinclination was expressed both via email responses and through the survey’s open comment box. Participants asserted that they lack both the time and desire to learn how to implement broader impacts activities; that it did not make sense to require scientists to do such work; that trained experts are better suited to public outreach than research scientists; and that it would be preferable to partner with experts in broader impacts than to learn how to conduct such activities. These comments point to the need for campuses to provide professional staff trained in implementing broader impacts activities to collaborate with and support researchers.
Survey respondents reported limited infrastructure available to assist with their broader impacts work. Existing service-learning and civic engagement centers, or other campus-wide coordinating units, can fill that gap by providing researchers with practical assistance; links to community partners; information about established, well-aligned projects with which to connect; and a venue for interdisciplinary partnerships. Technical assistance would be especially important in assessment and evaluation of the outcomes of broader impacts activities.
Beyond investing in training and infrastructure, institutions can also enhance and expand the quality and type of their broader impacts activities by providing incentives (such as stipends, release time, and seed grants) for faculty to build engagement into their work. An institutional climate supportive of community engagement was a strong predictor of high quality broader impacts activities. An institution’s climate is shaped by values expressed through rhetoric, public recognition, and reward systems. Respondents indicated that the executive leadership at their institutions explicitly promoted community engagement as a priority, but their promotion and tenure policies did not recognize anything other than peer-reviewed publications and grants as measures of productivity. If researchers are to be motivated to conduct high quality work in the community, tenure and promotion guidelines must be revised to ensure that they receive professional recognition and advancement for engaged scholarship.
Overhauling the faculty-reward system will not work without a broad shift in the attitudes of professors, especially those on promotion and tenure committees. Simply adding new metrics or categories of work to evaluate as part of the faculty reward systems is not enough. Academics must be encouraged to think differently. Executive leaders can inculcate a supportive climate by publicly promoting community engagement as a priority and ensuring that engagement activity is publicly recognized through award programs, university publications, and events that showcase the work of engaged scholars. By creating new traditions, rituals, and symbols that reinforce the value of community-based work, institutions can foster an epistemological shift toward democratic engagement and ease the dynamic tension accompanying the transition from traditional to alternative value systems, creating enduring and pervasive change.
With decreasing appropriations for public higher education and shrinking budgets, academics are under growing pressure to secure external funding for their research. In response to prodding from Congress, the NSF and other federal funding agencies are placing more, not less, emphasis on public accountability for and engagement in federally sponsored research. As prestigious federal grants increasingly require an outward-looking orientation, more and more scientists are finding themselves compelled to practice engaged scholarship. Research administrators must continue to explore how investigators can be both socialized and supported so that they embrace community engagement as an important and necessary element of federally funded research.
This paper is extracted from the author’s dissertation entitled “Evaluating the Broader Impacts of Sponsored Research through the Lens of Engaged Scholarship.” This paper won the Best Paper Award for the 2014 SRA Symposium.
Dr. Dianne Nagy
Grant Proposal Specialist South Dakota State University 1011 Rotunda Lane
Brookings, SD 57007
America Competes Reauthorization Act of 2010, Pub. L. No. 111-358 (2011).
Antonio, A. L., Astin, A. S., & Cress, C. M. (2000). Community service in higher education: A look at the nation’s faculty. Review of Higher Education, 23(4) 373-398. doi:10.1353/ rhe.2000.0015
Austin, A. E., & Beck, J. P. (2010). Integrating outreach and engagement in faculty work. In H.E. Fitsgerald, C. Burack, & S. Seifer (Eds.), Handbook of engaged scholarship, volume 1 (pp. 235-250). East Lansing, MI: Michigan State University Press.
Barker, D. W. M., & Brown, D. W. (2009). A different kind of politics: Readings on the role of higher education in democracy. Dayton, OH: Kettering Foundation Press.
Baskurt, O. K. (2011). Time series analysis of publication counts of a university: What are the implications? Scientometrics, 86, 645-656. doi:10.1007/s11192-010-0298-1
Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. Princeton, NJ: Carnegie Foundation for the Advancement of Teaching.
Boyer, E. L. (1996). The scholarship of engagement. Journal of Public Service and Outreach, 1(1), 11-20.
Bozeman, B., & Boardman, C. (2009). Broad impacts and narrow perspectives: Passing the buck on science and social impacts. Social Epistemology, 23(3-4), 183-198. doi:10.1080/02691720903364019
Braxton, J. M., & Luckey, W. (2010). Ernest Boyer and the scholarship of engagement. In H. E. Fitsgerald, C. Burack, & S. Seifer (Eds.), Handbook of engaged scholarship, volume 1 (pp. 71-93). East Lansing, MI: Michigan State University Press.
Brukardt, M. J., Holland, B., Percy, S. L., & Zimpher, N. (2004). Calling the question: Is higher education ready to commit to community engagement? Retrieved from http://www.servicelearning.org/filemanager/download/215/calling_the_question.pdf
Bush, V. (1945). Science - The endless frontier. Retrieved from http://www.nsf.gov/about/history/nsf50/vbush1945.jsp
Buxton, M. (2011). The payback of ‘Payback’: Challenges in assessing research impact. Research Evaluation, 20(3), 259-260. doi:10.3152/095820211X131185836358
Carnegie Foundation for the Advancement of Teaching. (n.d.). Classification description. Retrieved from http://classifications.carnegiefoundation.org/descriptions/community_engagement.php
Clearinghouse and National Review Board for the Scholarship of Engagement. (2002). Evaluation criteria for the scholarship of engagement. Retrieved from http://schoe.coe.uga.edu/evaluation/evaluation_criteria.html
Colbeck, C. L., & Wharton-Michael, P. (2006). Individual and organizational influences on faculty members’ engagement in public scholarship. In R. A. Eberly & J. Cohen (Eds.), New directions for teaching and learning, No. 105 (pp. 17–26). San Francisco: Jossey-Bass.
Cress, C. M. (2002). Campus climate. In A. M. Martinez, & K. A. Renn (Eds.), Women in higher education: An encyclopedia (pp. 390–397). Santa Barbara, CA: ABC-CLIO, Inc.
Doberneck D. M., Brown, R. E., & Allen, A. D. (2010). Professional development for emerging scholars. In H. E. Fitsgerald, C. Burack, & S. Seifer (Eds.), Handbook of engaged scholarship, volume 1 (pp. 391-409). East Lansing, MI: Michigan State University Press.
Doberneck, D. M., Glass, C. R., & Schweitzer, J. (2010). From rhetoric to reality: A typology of publically engaged scholarship. Journal of Higher Education Outreach and Engagement, 14(4), 5-35. Retrieved from ERIC.
Freedenthal, S., Potter, C., & Grinstein-Weiss, M. (2008). Institutional supports for faculty scholarship: A national survey of social work programs. Social Work Research, 32(4), 220-230. doi:10.1093/swr/32.4.220
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London: Sage.
Gibson, C. M. (2006). New times demand new scholarship: Research universities and civic engagement; a leadership agenda. Retrieved from http://www.compact.org/resources/research_universities/
Glass, C. R., Doberneck, D. M., & Schweitzer, J. H. (2011). Unpacking faculty engagement: The types of activities faculty members report as publicly engaged scholarship during promotion and tenure. Journal of Higher Education Outreach and Engagement, 15(1), 7-30. Retrieved from ERIC.
Hartley, M., Saltmarsh, J., & Clayton, P. (2010). Is the civic engagement movement changing higher education? British Journal of Educational Studies, 58(4), 391-406. doi:10.1080/0007 1005.2010.527660
Holbrook, A., Krosnick, J., & Pfent, A. (2007). The causes and consequences of response rates in surveys by the news media and government contractor survey research firms. In J. M. Lepkowski, N. C. Tucker, J. M. Brick, E. D. De Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, & R. L. Sangster (Eds.), Advances in telephone survey methodology (pp. 499-528). Hoboken, New Jersey: John Wiley & Sons.
Holbrook, J. B. (2005). Assessing the science-society relation: The case of the US National Science Foundation’s second merit review criterion. Technology in Society, 27(4), 437-451. doi:10.1016/j.techsoc.2005.08.001
Holbrook, J. B., & Frodeman, R. (2005). Policy dimensions of NSF’s criterion 2. Ogmius, 13, 4-5. Retrieved from http://sciencepolicy.colorado.edu/ogmius/archives/issue_13/
Holbrook, J. B., & Frodeman, R. (2007). Answering NSF’s question: What are the “broader impacts” of the proposed activity? Professional Ethics Report, XX(3), 1-3. Retrieved from http://www.aaas.org/
Holbrook, J. B., & Frodeman, R. (2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20(3), 239–246. doi:10.3152/095820211X129413718767 88
Jaeger, A. J., & Thornton, C. H. (2006). Neither honor nor compensation: Faculty and public service. Educational Policy, 20(2), 345-366. doi:10.1177/0895904805284050
Lok, C. (2010). Science funding: Science for the masses. Nature, 465, 416-418. doi:10.1038/465416a
Lunsford, C. G., Bargerstock, B. A., & Greasley, P. A. (2010). Measuring institutional engagement and faculty-based engaged scholarship. In H. E. Fitsgerald, C. Burack, & S. Seifer (Eds.), Handbook of engaged scholarship, volume 2 (pp. 105-130). East Lansing, MI: Michigan State University Press.
Lunsford, C. G., & Omae, H. N. (2011). An examination of factors that shape the engagement of faculty members and academic staff. Innovations in Higher Education, 36, 343-358. doi:10.1007/s10755-011-9179-7
Mathieu, R. D., Pfund, C., & Gillian-Daniel, D. (2009). Leveraging the NSF broader- impacts criterion for change in STEM education. Change, 41(3), 50-55. doi:10.3200/ CHNG.41.3.50-55
Mazuzan, G. T. (1994). The National Science Foundation: A brief history. Washington, D.C.: National Science Foundation.
National Academy of Public Administration. (2001). A study of the National Science Foundation’s criteria for project selection. Retrieved from http://www.csid.unt.edu/files/2001NAPAreport.pdf
National Science Board. (2010). Report to the National Science Board on the National Science Foundation’s merit review process fiscal year 2009. Retrieved from http://www.nsf.gov/ nsb/publications/2010//nsb1027.pdf
National Science Board. (2011). National Science Foundation’s merit review criteria: Review and revisions. Retrieved from http://www.nsf.gov/nsb/publications/2011/meritreviewcriteria.pdf
National Science Foundation. (2005). Report of the National Science Board on the National Science Foundation’s merit review system. Retrieved from http://www.nsf.gov/nsb/ documents/2005/0930/merit_review.pdf
National Science Foundation. (2008a). Broadening participation at the National Science Foundation: A framework for action. Retrieved from http://www.nsf.gov/od/broadeningparticipation/nsf_frameworkforaction_execsummary_0808.pdf
National Science Foundation. (2008b). Dear colleague letter – Broader impacts. Retrieved from http://www.nsf.gov/pubs/2008/nsf08062/nsf08062.jsp
National Science Foundation. (2009). FY 2008 report on the NSF’s merit review process. Retrieved from http://www.nsf.gov/nsb/publications/2009/nsb0943_merit_review_2008.pdf
National Science Foundation. (2012). Proposal and award policies and procedures guide, January 2013. Retrieved from http://www.nsf.gov/publications/pub_summ.jsp?ods_ key=nsf13001
Nicotera, N., Cutforth, N., Fretz, E., & Thompson, S. S. (2011). Dedication to community engagement: A higher education conundrum? Journal of Community Engagement and Scholarship, 4(1), 37-49.
Patten, M. L. (2002). Understanding research methods: An overview of the essentials (3rd ed.). Los Angeles: Pyrczak Publishing.
Pfirman, S. L., Collins, J. P., Lowes, S., & Michaels, A. F. (2005). To thrive and prosper: Hiring, supporting, and tenuring interdisciplinary scholars. A Project Kaleidoscope Resource. Retrieved from http://www.pkal.org/documents/Pfirman_et-al_To-thrive-and-prosper.pdf
Sarewitz, D. (2011). The dubious benefits of broader impacts. Nature, 475, 141. doi:10.1038/ 475141a
Scott, J. (2007). Engaging academia in community research: Overcoming obstacles and providing incentives. Center for Urban and Environmental Solutions, Florida Atlantic University. Retrieved from http://consensus.fsu.edu/bogfcrc/pdfs2/Engaging_Academic_ in_Community_Research_FAU.doc
Shannon, D. M., Johnson, T. E., Searcy, S., & Lott, A. (2002). Using electronic surveys: Advice from survey professionals. Practical Assessment, Research & Evaluation, 8(1). Retrieved from http://PAREonline.net/getvn.asp?v=8&n=1
Stanton, T. K. (2007). New times demand new scholarship II: Research universities and civic engagement; opportunities and challenges. Retrieved from http://www.compact.org/resources/research_universities/
Stanton, T. K., & Wagner, J. (2010). Educating for democratic citizenship: Antecedents, prospects, and models for renewing the civic mission of graduate education at research universities. In H. E. Fitsgerald, C. Burack, & S. Seifer (Eds.), Handbook of engaged scholarship, volume 1 (pp. 411-436). East Lansing, MI: Michigan State University Press.
Stokes, D. E. (1997). Pasteur’s quadrant: Basic science and technological research. Washington, D.C.: Brookings Institution Press.
Tretkoff, E. (2007). NSF’s “broader impacts” criterion gets mixed reviews. American Physical Society News, 16(6). Retrieved from http://www.aps.org/
Vogelgesang, L. J., Denson, N., & Jayakumar, U. M. (2010). What determines faculty-engaged scholarship? The Review of Higher Education, 33(4), 437-472. doi:10.1353/rhe.0.0175
Broader Impacts, Sponsored Research, Institutional Support, Community Engagement