An Assessment of the Visions in Methodology Initiative: Directions for Increasing Women’s Participation

By Tiffany Barnes, Emily Beaulieu, and Yanna Krupnikov

The following post is written by Tiffany Barnes, Assistant Professor of Political Science at the University of Kentucky, Emily Beaulieu, Assistant Professor of Political Science at the University of Kentucky, and Yanna Krupnikov, Assistant Professor of Political Science at Northwestern University.

The Visions in Methodology (VIM) conferences developed through the work of the Society for Political Methodology’s Diversity Committee, founded in 2005.  With the support of an NSF grant, five VIM conferences have been held and a sixth is scheduled to take place in Spring 2014.  The broad goals of VIM are to facilitate networking and mentorship, and to support women in political methodology.  This article provides a preliminary assessment of the impact of the VIM program.  Based on an internet survey, fielded in January 2014, we are able to report on some general characteristics of VIM participants, as well as provide comparison to individuals who have not attended VIM.

In a survey of political science professors in the U.S., Sedowski and Britnall (2007) found that 26% of political science faculty were women. And while women comprise a minority within the discipline as a whole, particularly at more advanced ranks, Methodology stands out as the sub-field specialization with largest gap between men and women who identify it as one of their fields.[1] Sedowski and Britnall report a gap of 4% between the proportion of men and women identifying as methodologists—which does not seem terribly large until one considers that by the authors’ indicators only about 10% of men identify methodology as an area of specialization, which means that nearly twice as many men identify as methodologists, compared to women.

VIM was created to provide “opportunities for scholarly progress, networking, and professional mentoring in research and teaching in order to support women in the political methodology community.”[2] Networking, mentorship, and career productivity and advancement stand out among the goals for the VIM conferences.  A typical VIM conference includes research presentations by selected participants (typically 45 minutes long), followed by discussant and audience feedback. Interspersed throughout these research presentations are professional development sessions that focus (broadly) on succeeding in academia.  Finally, a critical component of VIM is its opportunities for networking and interaction with senior and junior scholar attendees.

Research in economics has hypothesized that its gender gap is likely due to 1) a lack of access to “research networks” that facilitate co-authorship and publication (McDowell et al. 2006) and 2) a lack of role models and relationships between junior and senior faculty (Blau, Ferber and Winkler 2010). Blau et al. (2010) found that participants randomly assigned to participate in a small conference that matched junior female economists with senior mentors – such as VIM does for political scientists – were significantly more productive in terms of grants and publications when evaluated five years after their participation, compared to women not selected to participate in the program. In sum, this research suggests that these types of programs have succeeded in enhancing the productivity of female scholars in economics.

To investigate the extent to which VIM has had similar success, we fielded an online survey that went to all previous VIM participants and to political scientists (male and female) who have not attended VIM, but were affiliated with the same institutions as VIM participants. This survey will not allow us to draw causal conclusions about the impact of VIM. Participation in VIM is not a random process – participants self-selected into the applicant pool, and subsequently, were selected to participate based on the quality of their proposal.  As a result, it is difficult to determine whether post-VIM career outcomes are due to the VIM program, or due to VIM participants having particular characteristics which made them more likely to apply and be accepted to VIM in the first place. Thus, without making causal conclusions, we highlight and discuss some interesting differences between those individuals who have participated in VIM and those who have not.

The Survey and Characteristics of the Sample

The survey for this project was designed to collect information from individuals who had participated in VIM. The focus was on both their experiences with VIM and various aspects of their professional development. We also collected comparable professional development information from a group of individuals who had not attended VIM.  The sample for the survey was constructed by first identifying everyone who had presented a research paper at any of the VIM conferences (66 in all), and their institutional affiliation at the time of attendance.[3] Given that most (though not all) VIM participants are in early stages of their career (or at least were at these early stages during the time they participated in VIM), we then formed a control group by identifying every assistant professor, associate professor, and Ph.D. student currently on the job market from every institution that VIM participants were affiliated with at the time of their attendance.  Excluding VIM participants, the survey was successfully sent to 874 individuals[4].  We received responses from 45 individuals who participated in VIM, and from 288 individuals in the non-VIM group (35.3% of those successfully contacted).

Although our sample undoubtedly over-represents political scientists at PhD-granting institutions, the gender distribution of our non-VIM participants from our sample compares favorably with other surveys of the discipline.  Table 1 compares the proportion of female political scientists at different ranks from a 2007 APSA survey of gender in the profession and a 2012 NSF survey of PhDs conferred.  Our survey has a slightly higher proportion of women at the associate rank, and a slightly lower proportion of female PhD students, but still suggests largely similar rates.

Table 1: Comparison of Our Sample to APSA, NSF Studies

Even more importantly, we were also able collect publicly available information about those individuals who were selected to be part of our sample but did not complete the study.[5]. Specifically, we gathered the following information using available vitas for everyone in our sample:  individual’s current rank, rank of current institution, rank of Ph.D. granting institution, and number of years to obtain Ph.D.[6] Using these data, we evaluate whether there were any systematic patterns to which of the contacted non-VIM participants actually took the survey. In a model predicting survey participation that includes all of these factors and gender, none of these characteristics play a significant predictive role in determining participation in the survey (Table 2).[7]

Table 2: Factors Predicting Survey Participation (non-VIM participants)

Who attends VIM and What do they Experience?

Over eighty percent of women who have attended VIM are currently employed in tenure-track jobs.  Most completed their PhDs within the past five years, with many having completed within the past two years.  The modal VIM respondent reports receiving her PhD from a “Top 25” institution.  Currently, 47.2% of VIM participants who hold tenure-track jobs report that they are employed at “top-ranked research institutions,” while 52.8% report being employed at “lower ranked research institutions.”

When asked how they decided to apply to VIM, 76% report being encouraged to apply by another person.  Fewer than 5 percent of those encouraged indicated that such encouragement came from their advisor; most were encouraged by someone they describe as an “other mentor” (58%) or a peer (38%).

Those who attend VIM largely report positive experiences.  Table 3 shows participants’ perceptions of the utility of VIM. Sixty-seven percent of VIM participants describe the experience as “useful” or “very useful” (on a six-point scale from very useless to very useful).  Among those who found the conference to be useful, over eighty percent identified networking opportunities as the aspect of the conference they found to be particularly useful and 64% identified opportunities to network with other junior scholars as “extremely useful.” VIM participants found networking with junior scholars somewhat more useful than networking with senior scholars (a 13-percentage point difference). Indeed, 22.2% of those who found the conference useful also reported that, if they could change something about the conference, they would add more opportunities to network with peers.

Turning to professional development sessions, 74% of those who found the conference useful mentioned finding professional development sessions particularly useful. Specifically, participants found sessions on career development and career road -blocks to be particularly useful (44.7% report that these sessions were “extremely useful”).  Fewer participants (21.05%) reported that sessions on work life balance were useful were “extremely useful.”

With respect to research presentations, 48.7% found the discussant comments they received to be “extremely useful” and 43.2% found comments from other VIM participants to also be “extremely useful.” Additionally, 33.3% report that if they could change something about VIM, they would add more research feedback.

While we do not wish to speculate about why individuals were somewhat less enthusiastic about the professional development sessions, there are several clues to the reason behind this result in our survey. Specifically, in our study we gave individuals the chance to answer an open-ended question that simply invited them to share any information they wish about their VIM experience. For reasons of anonymity we do not share these open-ended responses verbatim; instead we discuss them broadly.

The open-ended responses suggest that some participants believed that professional development sessions tended by be dominated by just a few VIM attendees, who did not give others a chance to speak and used the sessions to vent frustrations with their own departments. Others reported these sessions were sometimes needlessly discouraging.  Open-ended responses also indicated that these sessions were too unstructured and required more moderation.  Additionally, participants had hoped for more concrete professional advice (e.g., grant writing, teaching, etc).

It is important to note, however, that the open-ended responses further reinforce the importance of networking to this program. A number of participants wrote that the chance to network with other junior scholars at VIM was extraordinarily positive and important to their careers.

Table 3: Usefulness of VIM: Participant Perceptions

Table 3b: Perceptions of VIM: Networking

Table 3c: Perceptions of VIM: Quality of Research Presentations and Feedback

On balance, then, most VIM participants report a positive experience with the conference and highlight elements that are consistent with the goals of VIM, but how do they compare to other individuals who participated in our survey?  We will highlight several differences between VIM participants and our non-VIM participants: perceptions of linked fate, experiences with networking and mentorship, and article submission patterns. These differences are notable, and in some cases obtain conventional levels of statistical significance, but we caution readers that it is not our intent to argue these differences are necessarily caused by VIM participation.

Demographic Comparisons

We begin by comparing the basic “demographic” career characteristics of our VIM and non-VIM participants. For ease of discussion, we refer to those individuals who did not participate in VIM as the “control group,” though again it is not our intent to suggest that this is a causal or controlled comparison.

The modal male participant in our survey completed his Ph.D. in the past two years, the modal female participant (who did not attend VIM) completed her Ph.D. in the past 5-8 years.  Thus, the men in our control group are at the same point in career as VIM attendees, or earlier, while the women in our control group are somewhat further along in their career.  The modal participant from the control group reports earning his or her Ph.D. from a “ Top 10” institution. In the control group, women report current employment at a “top ranked” research institution, while men report employment mostly at a mix of “top ranked” and “lower ranked” research institutions. Individuals in our control group tend to have received Ph.D.s from somewhat better-ranked institutions, and are currently somewhat more likely to work at better-ranked institutions than our VIM participants – though none of these differences are substantively large.

Fittingly, our VIM participants were somewhat more likely than control group women to report that the term “methodologist” describes them very well. Among VIM participants, 27.9% report that the term methodologists describes them “very well,” while only 12.4% of our control group women report that this term describes them “very well”. In contrast, 18.4% of control group men report that “methodologist” is a term that describes them “very well” – perhaps reflective of overall discipline differences.

VIM participants also had slightly broader definitions of what constitutes a methodologist than either control group men or women. We presented our participants with a list of statements that could describe someone who is a methodologist. This statements included “ develops new methodological techniques,” “applies advanced quantitative methods to substantive questions,” “focuses mostly on quantitative methodology,” “publishes work specifically focused on methodological issues,” and “studies questions of measurement and operationalization.”  A participant could select anywhere from none of these statements to all five.  On average, VIM participants selected significantly more statements than control group women, though behaved similarly to control group men.

Perceptions of Linked Fate

A key component of VIM is its emphasis on gender, and so we compared perceptions of “linked fate” among both our VIM and control group women. Linked fate considers the extent to which individual members of a group believe that their future is tied to the future of other group members (Dawson 1994). Here we adapted a linked fate measure often used in research on race and politics and asked all women who took the survey how much they believed their own success was tied to the success of other women in the profession.

Thirty-eight percent of VIM attendees reported that the success of other women had “a lot” to do with their own success, compared to 22% of women who had not attended VIM.  This 16-percentage point gap in perceptions of linked fate does not reach statistical significance, which is likely the function of the low number of VIM participants. On another gender measure, however, both VIM participants and control group women behaved in a similar manner.  In both groups, 43% of participants reported that their gender identity was at least “somewhat important” to them.

Mentorship and Networking

We next consider the way our participants describe their mentorship network (Table 4). Although statistical significance may be an imperfect baseline in this case, we do present this information. Bold values indicate those differences that reach statistical significance.  Reported p-values provide an additional letter marker to indicate if they are reflecting comparisons to control group women or men (represented with w or m).

Table 4: Networking and Mentorship

Overall, Table 4 suggests that VIM participants generally perceive a good deal of support from peers and have relationships with mentors at higher (though not consistently significant) rates than control group women and men. VIM participants are significantly more likely to have a mentor outside their own department than either control group men or women.

Another indication of networking is how often individuals are invited to give research talks at other institutions.  We present the average number of invited talks (excluding job talks) individuals gave over the last year and over the entire course of their careers. The results show that VIM participants give invited talks at a higher rate (both annually, and over the course of their career) compared to both control group men and women, though the difference is only statistically significant compared to men. In general, women give more invited talks than men in our survey, but the difference between control group women and men is not statistically significant. These differences are not a function of differences in career stages and hold when we compare only faculty at the same ranks.

Table 5: Invited Talks

Article Submissions

One way to gauge productivity is article submission   Table 6 presents average number of article submissions per year for our survey participants.[8] Here we see that men submit more articles on average than women—though the difference is only statistically significant when graduate students are included in the analysis; the difference does not reach statistical significance we limit comparisons to only faculty. As one may expect, faculty have higher article submission rates.  Among faculty we also see difference in rates of submission between VIM participants and the control group – with VIM participants having slightly higher rates of submission. While the difference between submission rates for VIM women and control group men is not significant, the difference between VIM women and control group women is significant, regardless whether we consider all participants or only faculty.

Table 6: Average Article Submission By Year

The variance around these means is reasonable and we have no reason to suspect that any of these averages are being driven by a small number of outliers. Furthermore, these patterns remain once we adjust for individuals who spent the bulk of their time working on a book.

Another consideration for productivity and career advancement is the journal outlets individuals target when they submit article manuscripts. Table 7 shows a breakdown of proportions of individuals who report sending to various types of journals, based on categories that were presented in the survey. Because individuals could select more than one category of journal, percentages will not sum to 100. We see that there is a slightly higher tendency among VIM participants to submit to higher-level outlets, than among control group women in particular.  Indeed, across all of our groups, control group women are least likely to submit to one of the top three journals.

Table 7: Targeted Journals

As a final step, we considered the steps individuals take prior to submitting a journal article.  We asked participants whether they circulate manuscripts either within their departments or to readers outside the department (excluding conference panelists) prior to submitting the manuscript to a journal. We find that our control group men and women are somewhat more likely than VIM participants to circulate manuscripts within their own departments prior to journal submission.  In contrast, VIM participants were somewhat more likely to circulate manuscripts to people outside their own departments than both control group men and women. The modal reason for not circulating manuscripts (across all groups) is the concern that circulation burdens colleagues with extra work.

Discussion and Limitations

Our results point to some differences between VIM participants and peers who did not participate in VIM. There are essentially three ways we might interpret these differences.  One, VIM makes a difference: women who go to VIM gain opportunities to expand their networks and subsequently increase the quantity and quality of their productivity.  A second possibility, however, is VIM participation is simply a proxy for another set of characteristics that differentiates VIM participants from the control group. Our survey results show, for example, that the majority of the women who come to VIM were typically encouraged by another person to apply. This may hint that the women who attend already have some type of a support network.  As a result, the fact that they apply to VIM might just be an indication of a general level of ambition that explains their differential productivity.  The third possibility is that VIM has some effect, but that effect is difficult to quantify given the selection issues identified in the second possibility.

Conclusions

The goal of VIM has been to create a network for female methodologists (often interpreted broadly).  Our results suggest that it is indeed the network component that VIM participants have found most valuable. In particular, VIM participants are most enthusiastic about having a chance to interact with their peers.

Although participants are generally positive about their VIM experiences, they are least enthusiastic about the professional development sessions – something which is also an important part of VIM. While our participants felt career discussions were useful, relatively few viewed sessions on work-life balance to be as useful as other parts of VIM.

In sum, then, while the results of our survey point to the overall positive impact of VIM, they also point to several ways in which the VIM program may be improved. First, our survey suggests that VIM attendees may have different needs and interests. As a result, it may be helpful to implement a pre-VIM survey to help guide discussion in professional development sessions. Such a survey may have the added benefit of serving as an initial measure of participant characteristics, which may help untangle the causal effect of VIM.

Second, based on responses to this survey, VIM may want to implement changes to the professional development sessions. Specifically, the open-ended responses indicate a need for more structured sessions on career development, where all participants feel equally welcome to speak.

Third, our results show that most of those who participated in VIM did so after a mentor or colleague encouraged them to do so.  This suggests that these are individuals who already have at least some network connections.  In the future, VIM may try to broaden recruitment to reach individuals who may not necessarily have a mentor or colleague to inform them of opportunities – these types of people may be particularly likely to benefit from VIM.  One possibility may be to randomly select VIM participants. This may again have the added benefit of determining the causal impact of the program.

In this study we have been able to identify differences between women who have attended VIM and peers, and have offered several suggestions for ways the program might be improved, which might allow for clearer causal inference of the impact of VIM in the future.  Even if the causal effect of VIM remains unclear, or even if it becomes apparent that VIM is a forum for already ambitious and productive female methodologists; in a discipline and subfield where women are under-represented, we submit that such a modest function deserves continued support.

Notes

[1] The eight sub-field specializations included by the authors are: American, Comparative, IR, Methods, Public Administration, Public Law, Public Policy, Theory.

[2] http://visionsinmethodology.org/

[3] In this process two individuals who served as discussants were included in place of two individuals who presented research—the inclusion of the two discussants did not change our sample of institutions in any way, and the institution for one of missed presenters was included in the sample, but the other was not.  These errors were completely random and we have no reason to think that they affected the results of our survey.

[4] An additional 58 were included in the sample, but could not be reached by email.

[5] The survey was fielded with a mailer feature in Qualtrics, which allows us to identify those individuals who took the survey and those who did not. Results were anonymized so that we cannot attach any particular results to a given participant.

[6] Rank information based on the most recent rankings from U.S. News and World Report, which covers the top 86 political science doctoral programs in the country.

[7]  This model excludes VIM participants, as they were targeted separately, with an additional email. When VIM participants are included, gender is a significant predictive factor in participation. This is to be expected, as all VIM participants are women. These results are also robust to a model that just uses information available on departmental websites, even for individuals who do not have available vitas on the web.

[8] For VIM participants the average is calculated by asking how many submissions since VIM, and then dividing by years since VIM; for control group women and men the average is calculated based on asking how many submissions in past 3 years, and then dividing by 3.