Leadership development program evaluation: A social network analysis approach

 Kevan W. Lamm, University of Georgia, kl@uga.edu 

Hannah S. Carter, University of Florida, hcarter@maine.edu

Abstract

When asked about the benefits of participating in agriculture and natural resource (ANR) leadership development programs, one of the most frequent responses is the network one can develop. However, despite the ubiquity of the perceived benefit there have been few empirical studies conducted to examine network development within ANR leadership development programs. With improved social network data capture and analysis techniques, contemporary ANR leadership development programs, and leadership educators more generally, are well-positioned to take advantage of these developments. The results of the current study indicate social network analysis is an appropriate tool for establishing evaluative measures of network emergence and development within ANR leadership development programs. 

Introduction 

When asked about the benefits of participating in agriculture and natural resource (ANR) leadership development programs, one of the most frequent responses is the network one can develop (e.g. Kelsey & Wall, 2003). However, despite the ubiquity of the perceived benefit there have been few empirical studies conducted to examine network development within ANR leadership development programs. From an educational perspective, ANR leadership development programs also represent non-traditional composite learning environments and typically include both formal and non-formal educational components (Kaufman, Rateau, Carter, & Strickland, 2012). 

At the most fundamental level, leadership is about interactions between people. Northouse (2013) has defined leadership as, “a process whereby an individual influences a group of individuals to achieve a common goal” (p. 5). Leadership development programs therefore should naturally encourage and enable the development of these connections between participant learners. However, one of the critiques of leadership development programs more generally is a lack of rigor and accountability related to outcomes and impacts (Kellerman, 2012). Consequently, there seems to be a persistent challenge to quantify and empirically report what is more tacitly experienced by participant learners. 

As social creatures (e.g. Ryan & Deci, 2000), humans and the interactions between oneself and others is a fundamental aspect of humanity (Bass, 2008). Social networks are a natural extension of this shared experience. ANR leadership development programs are generally composed of cohorts of individuals sharing common experiences and interacting throughout the program (Kaufman et al., 2012). Based on Bandura’s Social Learning Theory (1977) it has been established part of the learning process is conditional on the shared learning experience amongst participants. From a social interaction perspective, the connections between individuals is paramount, “what happens to a group of actors is in part a function of the structure of connections among them” (Borgatti, Everett, & Johnson, 2018, p. 1). 

With improved social network data capture and analysis techniques, contemporary ANR leadership development programs, and leadership educators more generally, are well-positioned to take advantage of these developments. With more sophisticated techniques available to measure, monitor, and evaluate social network, educators can be more responsive to the needs of learners (McKeachie & Svinicki, 2013). Furthermore, social network data can provide additional evaluation data educators can use to quantify the ANR leadership development program experience and outcomes 

Priority area five of the National Research Agenda: American Association for Agricultural Education 2016 – 2020 (Roberts, Harder, & Brashears, 2016) addresses efficient and effective agricultural education programs with a particular focus on, “What evaluation methods, models, and practices are effective in determining the impacts of educational programs in agriculture and natural resources?” (p. 43). The purpose of this research is to present an evaluation approach and methodology for capturing and reporting leadership development programming impacts and outcomes, specifically the emergence and composition of networks among program participants. 

Conceptual Framework 

The conceptual framework for this study was based on social capital (Coleman, 1988) and social networks (Borgatti, Mehra, Brass, & Labianca, 2009). The integration of the two theory bases is intended to provide both a theoretical basis for phenomenon to occur, in this case social capital preceding network emergence, and a theoretical framework in which to quantify the outcome, social networks. 

Social Capital 

As Coleman (1988) established, social capital plays a significant role in human capital development. An individual amasses network connections and assets, these assets are then available to employ when appropriate (Burt, 2009). Network assets, or social capital is thus comprised of both strong and weak connections throughout a network (Lin, 2008). Additionally, social capital is composed of norms within a network to facilitate mutual understanding and expectations (Woolcock & Narayana, 2000). 

Fundamentally, social capital may be considered to be a measure of informal power among a heterogenous group, or network (Bass, 2008). Although an individual with a higher level of social capital may not hold a formal position of authority within a network, such an individual is generally viewed as holding a degree of influence and access to resources beyond those of their peers (Rogers, 2003). More specifically, “the concept of social capital refers to the ways in which people make use of their social networks in getting ahead.” (Hsung, Lin, & Breiger, 2010, Location No. 319). 

Within the literature, social capital has been examined extensively. Stemming from the seminal works of Coleman (1988) and Lin, Fu, and Hsung (2001), social capital remains a relevant theory base for inquiry and analysis. For example, Mollenhorst, Völker, and Flap (2008), examined the relationship between social contexts and building personal networks. According to the researchers the place where people meet their network members is important to the resulting relationship. Additionally, Erickson (2004) found within the context of a local community organization engaged in the sale of goods and services amongst the group that social capital accrued at the local level had a relationship with social capital at a higher order level. Consequently, the existence of social capital locally within the organization was related to more social capital outside of the organization, in the community at large. 

Social Networks 

Although contemporary social network analysis is done within the context of methodological rigor, social networks should not be confused with a methodology. Instead social networks are a representation of social phenomenon grounded in theoretical concepts intended to explain the social world (Borgatti & Halgin, 2011). Humans, and social interactions amongst humans, are complex based on the multitude of variables that may influence such interactions. However, despite the acknowledged challenges associated with observing and quantifying interactions, social network analysis has been employed widely to capture and analyze the phenomenon (Borgatti et al., 2009). 

For example, Johnson, Boster, and Palinkas (2003), analyzed small group development among individuals. From an organizational interaction perspective, Lamm and Lamm (2017) examined the nature of relationships between funding agencies as reported by Biological Science educators. Additionally, Scott, Jiang, Wildman, and Griffith (2017), analyzed the emergence and of leadership networks in teams as well as the effectiveness of such networks. As it relates to leadership and social networks, Chrobot-Mason, Gerbasi, and Cullen-Lester (2016) analyzed the relationship between organizational identity and leadership identification, finding “individuals who identify strongly with their organization and team are more likely to see others as sources of direction, alignment, and commitment” (p. 307). 

Purpose & Research Objectives

The purpose of this study was to analyze social network characteristics of an ANR leadership development program. The study was driven by the following research objectives: 

  1. Describe the nature of existing relationships amongst class members. 
  2. Describe the nature of advice seeking within network. 
  3. Describe the nature of support seeking within network. 
  4. Describe the nature of industry decision influence within network. 
  5. Describe the nature of industry influence within network. 

Methods 

A social network research design was employed for this study, specifically a whole-network design. An online questionnaire was developed based on recommendations within the literature (Borgatti et al., 2018). The questionnaire was developed for the purposes of the research and reviewed by a panel of experts to ensure content and face validity. 

The questionnaire was sent to all 30 of the leadership development program participants in September 2016, prior the first session of the program. The timing was intentional to establish a robust baseline network measure and to minimize the effects of in-person interactions which occur after the program began (Borgatti et al., 2018). There were 29 responses for a 97% response rate. Consistent with the recommendations within the literature (Bono & Anderson, 2005), respondents were first asked to indicate whether they knew each of the 29 other class participants. All results from the analysis replaced respondent names with an ID number placeholder to preserve anonymity (Borgatti et al., 2018). 

Next, respondents were asked how likely they were to seek advice from each of their classmates. There were two items used to assess advice. First, “If you needed help, you would seek advice from this person.” Second, “You would seek support from this person if you wanted to implement a new idea.” The items were adapted from sources previously established within the literature (e.g. Bono & Anderson, 2005; Ibarra, 1993; Salk & Brannen, 2000). Individuals indicated their response on a five-point, Likert-type scale. Possible responses to each item included: 1 – Strongly Disagree, 2 – Disagree, 3 – Neutral, 4 – Agree, 5 – Strongly Agree. Consistent with recommendations within the literature (Borgatti et al., 2018) scores were then converted to a dichotomous scale to facilitate analysis. Scores of 4 or 5 were coded as 1 and all other scores were coded as 0. 

To examine influence within the network two questions were asked. First, “This person has a great deal of influence on the decisions that get made in your industry.” Second, “This person has a great deal of influence on what happens in your industry.” The items were adapted from sources previously established within the literature (e.g. Bono & Anderson, 2005; Brass & Burkhardt, 1993; Salk & Brannen, 2000). Participants responded to these questions for each classmate, using the same 5-point scale used for advice–likelihood. Consistent with recommendations within the literature (Borgatti et al., 2018) scores were then converted to a dichotomous scale. Scores of 4 or 5 were coded as 1 and all other scores were coded as 0. 

The Ucinet 6 software package was used to visualize the network. Nodes represent participants and the lines connecting them indicate an individual (or multiple individuals) have a relationship. Node color is based on participant sex as reported by the participant. Blue indicates a male and pink indicates a female. Node size is determined by centrality within the network. Larger nodes indicate a more central location within the network. Line color is an indicator of whether the connection is reciprocal or one-way. Reciprocal relationships are displayed in red whereas one-way relationships are displayed in grey. 

Results 

Visualization of Existing Relationships amongst Class Members 

Using the Ucinet 6 software package, the network of pre-existing known relationships amongst class members was visualized. A complete network map is provided in Figure 1. Within the group there were three individuals who did not have any pre-existing connections to other class members. Additionally, there was one pair of isolates only connected to each other, as well as four pendants, or individuals only connected to only one other person. Nevertheless, there were actors, ID2, ID1, ID15, ID10, ID19, ID7, and ID13 who had a high number of both reciprocal and unidimensional ties within the network. 

Leave a Reply