Letting the right ones in: obstacles in graduate admissions  

Our previous article discussed admission metrics such as the GRE (Graduate Record Examination), undergraduate GPA (grade point average), previous research experience, recommendation letters, personal statements and in-person interviews, and their relation to graduate student performance. Several studies (Joshua Hall et al., Liane Moneta-Koehler et al., and Orion D. Weiner) indicate that certain quantitative measures used in graduate admissions are not accurate predictors of success. Below we discuss the obstacles that exist in graduate admissions.

Defining graduate student success is one of the issues confounding the admissions process. Following the Orion D. Weiner study, alumni of the Tetrad program (Sean M. Bell et al.) pointed out the difficulty of using success as an outcome measure. “Successful” graduate students may be those who published more, worked longer hours, had more amiable personalities, and eventually obtained an academic position at a high-profile institution. Or, “successful” students may simply be those who met the required cultural expectations of graduate school. Ultimately, they argue that grouping students into only two categories (“successful or underperforming”) is a simplification of this complex argument.

When should we evaluate graduate student success? Beyond graduate school, the definition of success extends into creating leaders who can contribute valuable expertise to their chosen fields (Sean M. Bell et al.). Therefore, the focus of graduate programs tends to rely heavily on postgraduate outcome. Orion Weiner responded in turn by discussing the need to focus on training students during graduate school. Evaluating graduate students at various levels may ultimately be valuable in improving both graduate admissions and graduate training programs.

Who should evaluate graduate student success? Faculty evaluations appeared to be very meaningful for evaluating both current and past graduate students in the University of California, San Francisco Tetrad program, given the ample time during which they observed students at the bench (O. Weiner). Although subjective, this is an interesting alternative to ranking students based on traditional criteria. In contrast, other studies (Joshua Hall et al. and Liane Moneta-Koehler et al.) aimed to use more objective measures by examining graduate student productivity. Directly examining graduate students or relying on their advisors can both yield important information for shaping graduate training.

The graduate admissions committee can greatly affect admissions decisions.  Julie Posselt, Assistant Professor of higher education at the University of Michigan, highlighted the role of admissions committees in her recently published book on graduate admissions. Posselt obtained permission from six highly ranked departments at three research universities to watch their reviews of candidates, and interviewed faculty members at four other universities. The ten departments in the study, anonymous in her book, were from the humanities, social sciences, and natural sciences. The first stage of the review process was focused on academics (GRE scores, undergraduate GPA), including a consideration for the prestige of undergraduate programs from which the applicants came. The second stage focused on evaluating the candidate’s personality and character by letters of recommendation, personal statements, writing samples, and interviews. The admissions process beyond the first stage was “very idiosyncratic” and included other factors beyond the candidate’s control, according to Posselt. These include admissions committee composition, the qualifications of other applicants, and committee expectations for that year, as she explains in this podcast.

GRE scores are still a major metric used by admissions committees to initially evaluate applicants. Julie Posselt discusses in her study that admissions committees prefer the GRE as a “simple measure with which to compare applicants and to exclude some” during these early application stages. According to this interview, some faculty don’t consider applicants with GRE scores of less than 700, and “that cuts usually two-thirds of applicants.” Demanding high GRE scores during the admission process may put certain groups at a disadvantage, limiting the ability of underrepresented groups (Valantine and Collins) to enter graduate school. Conversely, low GRE scores can put students at a disadvantage financially. According to Joshua Hall, for a student who has to take the test multiple times, the cost “skyrockets, not to mention the time they are sinking into studying!” Hall continued by saying that “this becomes a moral issue if we in continue requiring our applicants to shoulder significant financial burdens to take a test that is in no way valuable or predictive for success in our [biomedical training] programs.” Even for students accepted into graduate school, low GRE scores can be detrimental long-term, when used as criteria for funding decisions.

Diversity, equity, access, and inclusion in graduate admissions are affected by an over-reliance on metrics. Research potential is another criterion that admissions committees looked for in Posselt’s study, and thus were “unabashed elitists.” Quoting from one of the professors, “[admissions committees] make a lot of inferences about the quality of someone’s work and their ability based on where they come from.” Admissions committees were mostly male and mostly white, only 3% were U.S.-born professors of color, and 30% were female according to Posselt’s study discussed in this podcast. In addition, race and ethnicity of graduate school candidates were considered as a factor, but only for the short list of candidates who were otherwise equal by other criteria. This practice is still keeping “certain forms of inequality alive,” according to Posselt.

The GRE itself is a racially and socioeconomically biased test (Miller and Stassun) for several reasons. The GRE can be a heavy financial burden, and students with a low socioeconomic status (SES) perform worse on standardized tests (P. Sachs). “If you’re a student who takes the GRE and applies to eight schools, you will pay $313, not including any test prep. You can certainly improve your score with test prep, and students from higher SES groups will have easier access to that,” says Joshua Hall. For international students, another issue is that they may have high GRE scores but very poor English skills. Discussions of replacing the GRE with other measures have been met with caution due to the lack of suitable alternatives. However, as Hall argues, “we needn’t wait until something else shows up in order to eliminate measures [such as the GRE] that we know are at best useless, and at worse biased against students from certain groups and those from lower SES backgrounds.”

Letters of recommendation appear to be a valuable metric by which to evaluate graduate student performance. One caveat, however, is that most letters tend to portray students in a positive light. In addition, the terms used to describe someone’s qualifications in letters are not consistent among faculty members. While there can exist varying degrees of positivity in their assessments, distinguishing students based on two seemingly equally positive letters of recommendations is difficult. Gender stereotypes can also bias how recommenders describe female compared with male applicants in letters, as recommenders used significantly more standout adjectives to describe male as compared with female candidates (Schmader et al.). Examining how meaningful the letter content is may provide better information when evaluating how the specific candidates may perform in graduate school.

The laboratory environment and the research advisor are clearly critical to graduate training. Studies tend to focus less on the impact of the advisor on graduate student progress toward graduation and their overall success in graduate school, even though the advisor holds probably the single most important influence on them.  For example, very good graduate students may have a low level of productivity due to being in a “toxic” lab environment. On the other hand, less promising graduate students could thrive in graduate school with an advisor who can motivate and elevate them to perform at a high level. The quality of training and the laboratory environment in general may also influence graduate performance.

Conclusions: Defining and measuring graduate student success are not trivial tasks, and there is much work to be done in figuring out when and by whom success should be evaluated. What should go into a holistic review of graduate school candidates can also be difficult to define. Admissions committees need to revise the values upon which they evaluate candidates. They need to look at both classroom-based criteria and personal traits, and not just rely simply for convenience on metrics that are flawed in predicting graduate student performance.

Acknowledgments: We thank Joshua Hall (UNC Chapel Hill) for providing us with quotes and insights on some of the obstacles in graduate school admissions.

References:

Bell SM, Blumstein J, Brose K, Carroll A, Chang J, Charles J, Haswell ES, Michelitsch M, Owens JC, Patil CK, Smith R, Tupy J, Walsh E, and Ware T (2014). Defining success in graduate school. Molecular Biology of the Cell, 25(13), 1942–1944. http://doi.org/10.1091/mbc.E14-03-0793

Forth, T Gender bias calculator [Accessed 22 March 2017]. http://www.tomforth.co.uk/genderbias/

Hall JD, O’Connell AB, and Cook JG (2017). Predictors of Student Productivity in Biomedical Graduate School Applications. PLoS ONE 12(1): e0169121. doi:10.1371/journal.pone.0169121

Miller C, and Stassun K (2014). A test that fails. Nature, 510, 303-304. doi:10.1038/nj7504-303a

Moneta-Koehler L, Brown AM, Petrie KA, Evans BJ, and Chalkley R (2017). The Limitations of the GRE in Predicting Success in Biomedical Graduate School. PLoS ONE 12(1): e0166742. doi:10.1371/journal.pone.0166742

Posselt JR (2016). Inside Graduate Admissions: Merit, Diversity, and Faculty Gatekeeping. Cambridge, MA: Harvard University Press.

Posselt JR (2016). “Behind the Doors of Graduate School Admissions Rooms” [Audio podcast]. Retrieved from

http://www.hup.harvard.edu/catalog.php?isbn=9780674088696

Sachs P (1997). Standardized testing: meritocracy’s crooked yardstick. Change, 29(2): 24-31.

Schmader T, Whitehead J, and Wysocki VH (2007). A Linguistic Comparison of Letters of Recommendation for Male and Female Chemistry and Biochemistry Job Applicants. Sex Roles, 57(7-8), 509–514. http://doi.org/10.1007/s11199-007-9291-4

Valantine HA, and Collins FS (2015). National Institutes of Health addresses the science of diversity. Proc Natl Acad Sci U S A. 112(40):12240-2. doi:10.1073/pnas.1515612112.

Weiner OD (2014). How should we be selecting our graduate students? Molecular Biology of the Cell, 25(4), 429–430. http://doi.org/10.1091/mbc.E13-11-0646

Weiner O (2014). Response to Bell et al. Molecular Biology of the Cell, 25(13), 1945. http://doi.org/10.1091/mbc.E14-04-0921