Configuring the Holistic Score

What is holistic review? 

A holistic review approach identifies individuals whose unique strengths align with a program’s goals by considering multiple dimensions—such as personal attributes, experiences, skills, and academic performance—rather than focusing disproportionately on just one or a few factors like test scores.

What is the ResidencyCAS holistic score?  

To assist in the holistic review process, ResidencyCAS automatically calculates a holistic score for each applicant. This scoring model was developed by Othot, Liaison's data science team, which has over a decade of experience in the field. The model is:

• Research-based: developed through rigorous research, incorporating feedback from both the community and experts.
• Explainable: unlike some data science models that function as "black boxes," where it's challenging to interpret how they arrive at their decisions, this model's algorithm is designed to be comprehensible.
• Customizable: flexible and can be tailored to align with your program's mission, goals, and values. 

Where can I find the score?

The holistic score appears in several places throughout the Program Portal, including the application. When you click the score, a breakdown appears:

How can I use the score? 

You can choose to use the holistic score at your discretion. For example, you may choose to add the score to the applications grid so you can sort applicants by score and prioritize reviewing those with higher scores. Additionally, you may find that the holistic nature of the tool brings to your attention applications that align with your mission, goals, and values, and might have otherwise gone unnoticed.

When using the score, note the following: 

• This is a "surfacing tool."
  • Programs may find this tool useful in identifying an initial set of applications that should be prioritized for review.
  • Alternatively, you might want to identify specific subpopulations before reviewing the holistic score. The Program Portal allows you to focus on applicants based on various factors using the segments feature. 
• You can customize how the scoring components are weighted so they align with your program's values.
  • The weights represent the relative priorities of different elements of the application to your program and how applicants compare to other applicants within your cohort.
  • For a self-assessment to determine your weights, visit https://holisticreviewtool.com.
  • Note that while you can adjust overall weightings, you cannot add or create custom scoring elements, adjust the weights for individual applicants, or adjust any final calculated scores.
• This is not meant to be used as a precision instrument.
  • You should look at the score in relation to the other scores; the absolute score is less relevant.
  • For example, an 88.1 score is not meant to be far better than an 87.9 score. 
• This is not meant to replace scoring forms.
  • If you currently use scoring forms when assigning applications to your review team, you should continue to do so. 
• This model will continue to evolve as we collect program feedback. 

What are the key scoring concepts?

• The scoring model uses direct data to calculate scores. 
  • The model does not attempt to infer things, such as by counting words in a personal statement or using other less reliable data analysis methods.
  • Scores are derived from real data elements provided in the application or in official supplementary materials, such as USMLE scores, COMLEX, and the SLOE. 
• Each data element is calculated as a percentile score relative to other applicants to each program.
  • All data element scores are scaled from 0 to 100 (including binary variables).
• The scoring model organizes application data elements into domains and subdomains.
  • Domains are overarching categories, while sub-domains provide further granularity within a domain. 
• All scores and percentiles in the model are calculated within your program’s cohort of applicants, not against the general applicant pool.
  • As a result, each applicant receives a unique score for each program, reflecting their comparison to other applicants within the same program. 

How are scores derived? 

1. When viewing the score, notice there are four columns in the table.
   
   
    
2. On the left, we have the domains (i.e., categories of data elements). Domains such as Academic Metrics, Background and Lived Experience, and Values and Competencies can be expanded to show subdomains. 
3. The Original Score column lists a set of scores assigned to the applicant, ranging from 0 to 100. These scores are derived using a percentile calculation method.
   • For each data element included in a domain and subdomain, ResidencyCAS assesses how each applicant's performance compares to that of others applying to the same program. Once these percentiles are determined, they undergo a rescaling process, converting them from percentile values to a standardized 0 to 100 scale. This ensures a fair and consistent evaluation of applicants' standings within the applicant pool.
   • If a data element lacks a value, it is excluded from the calculation, and the weight is redistributed among the remaining domains and subdomains to prevent unfair penalization due to missing data.
4. The Weight column displays the weightings assigned to each domain and subdomain, which you can customize. 
5. The Weighted Score column displays the weighted score for each domain and subdomain, which is the multiplication of the original score by its weight. 

To calculate the Total Holistic Score, the weighted scores for all domains are summed together.