Skip to main content

How to Assess Resident Applicants' Core Competencies

By Julie McCoy, for HealthLeaders Media  
   January 07, 2011

From the day the Electronic Residency Application System (ERAS) opens, keeping up with applications is an overwhelming task for program coordinators.

“You come in one day and there are another 100 applications to download,” says Jill Philp, radiology residency training coordinator at University of Michigan Medical Center in Ann Arbor. 

It’s often up to program coordinators to prescreen ERAS applications, but filtering out the best candidates a huge undertaking. With limited interview spots available, efficiently reviewing applications is critical to ensuring that your program extends invitations for interviews to the most desirable candidates.  

Although coordinators typically have a list of criteria to look for on each application, reviewing ERAS applications is like comparing apples to oranges, says Lois Shuman, C-TAGME, radiology education coordinator at Indiana University School of Medicine in Indianapolis. This is because while the ERAS application is standardized, much of the information included in it is not. Inconsistencies include: 

  • Different grading methodologies depending on the medical school
  • Letters of recommendation that vary greatly in content and quality
  • Whether the medical schools offer membership to the Alpha Omega Alpha honor society

The United States Medical Licensing Exam (USMLE) score is the one apples-to-apples comparison you can make, but it only gives you insight into a candidate’s medical knowledge, Shuman says, adding that you don’t get an idea of whether he or she is a good fit culturally. 

To get a better sense of the candidates, Shuman and Philp developed a framework for evaluating candidates’ ERAS applications using the ACGME’s six core competencies. 

“You have to use the competencies to evaluate the residents once they’re in your program, so it makes sense to select candidates based on those competencies, too,” Shuman says.

Dig deeper into the ERAS application

The information candidates submit via ERAS contains many clues to their competency skills. 

“You can pull things out of ERAS if you just look,” Shuman says.

For example, you can evaluate the candidates’ professionalism by looking at the completeness and quality of their personal statement. 

You can also look at research experience listed on the CV as an indication of how well candidates will master practice-based learning and improvement and systems-based practice. The Medical School Performance Evaluation (MSPE) gives insight into all six of the core competencies, Philp says.

“This is like the 360-degree evaluation of the person before they come for the interview,” Shuman explains. “It doesn’t replace the interviews, but it may help you invite people you really want to meet so you don’t waste an interview slot.”

Score candidates on the competencies

Shuman, Philp, and their radiology coordinator colleagues Dora Thompson at Cedars-Sinai in Los Angeles and Linda Galante at Henry Ford Hospital in Detroit developed a competency score sheet to use when prescreening candidates for interviews or when ranking candidates before submitting the Match list.   

“This is something that’s customizable to each program,” Shuman says, adding that programs can place more emphasis on their most desired qualifications.  

Program coordinators should work with their program directors to develop a candidate score sheet that factors in the core competencies.

Assign a point value to each element of the ERAS application, such as: 

  • MSPE
  • Academic abilities
  • Letters of recommendation
  • Research experience
  • Publications
  • Personal statement
  • CV
  • Volunteer experience
  • Work experience

Rate candidates on each item by creating a scale that has specific objective anchors associated with each score. For example, if you want to score the practice-based learning and improvement competency by looking at research, the scale may include the following ratings and anchors:

  • 5 (highest): First author on published work
  • 4: Publications
  • 3: Posters, abstracts, papers
  • 2: Submitted research but not published
  • 1: No research

To measure professionalism, consider scoring the personal statement and CV. A five-point scale might include:

  • 5 (highest): Well-written and focused
  • 4: Very good
  • 3: Average
  • 2: Somewhat poor (grammatical errors)
  • 1: Poorly written

The score sheet can be modified to give higher weight to those competencies or qualities that are most important to your institution, specialty, or program, Shuman says. If the USMLE is important to your program and you want to give it more weight, create a seven-point scale based on USMLE score ranges so that those who scored high receive more points. 

Defining the anchors is critical to making the score sheet meaningful. Get input from your program director and selection committee members on what they look for in a candidate and what red flags they search for in MSPEs or letters of recommendation. 

Collaborate with other coordinators in your specialty or institution to create score sheets that reflect qualities common to physicians in your specialty or that fit with your institution’s culture, Philp says. 

Make the score sheet work for you

Most programs receive several hundred applications which makes the thought of completing this evaluation sheet for each and every candidate very daunting. 

Shuman suggests using the scoring feature included with ERAS that allows reviewers to enter scores for each component of the application. 

“You can put scores on different parts of the application, if you want, and it manages the scores and does the calculation for you,” Shuman explains. 

If you want to add scores for items not preprogrammed into ERAS, such as evidence of leadership or publications, create a data field using the Manage Composite Score section. Specify that the new field should be included in the composite score, Shuman says. You can also give certain scores more weight than others in this section.

As you review the applications in ERAS, click in the application and enter the scores. Use the filter feature to sort candidates by their scores to see which scored the highest. 

Tip: If you’re not using ERAS to calculate the competency scores, create the competency score sheet using a spreadsheet program. Write formulas or functions so that the program will tally the scores for you.

There is no right or wrong way to use the sheet, says Shuman. During the first year she implemented the sheet, she used it to score interviewed candidates and to break ties between them.   

Shuman and Philp agree that the competency score sheet has given them a more well-rounded view of candidates. 

“There are many people who look good on paper, but when they come [in for an interview] it’s a wrong fit for your department’s culture,” Shuman explains.

Additionally, the score sheet has made the selection process more streamlined and objective. 

In the future, Shuman and Philp hope to use the scores generated by the competency score sheet to:

  • Compare applicant scores with the scores of current residents and recent graduates
  • Compare competency scores with satisfaction of and about residents
  • Adjust weight given to competency criteria

By identifying specific items and assigning numeric values to each element of the ERAS application, coordinators have an easier time sorting applications, save time during the busy recruitment season, and are better able to select the best candidates based on their program’s values.

Tips for prescreening applications

Keep the following in mind when reviewing applications:

  • Account for all dates listed. If there is a break, look for the reason.
  • Note the applicants’ interests to get an idea of whether they work well in teams and have manual dexterity and problem-solving skills. Additionally, most physicians work and interact with multiple caregivers. Hobbies or interests involving team sports may indicate the ability of the applicant to work well with others.
  • Read letters of recommendation and MSPEs closely because they give insight into any problems or issues with authority, professionalism, or communication skills. 
  • Pay attention to who the letters are from—there should be letters from the department chair and faculty from the clinical specialty to which the applicant is applying.

Tagged Under:


Get the latest on healthcare leadership in your inbox.