When evaluating applications, you may wish to implement a system that automatically scores applicants based on a rubric or set of criteria used by your program. This allows you to compare applications using a pre-established set of components and to pinpoint your strongest applicants. Outcomes offers several options to do this:
- Phase Calculated Fields: this feature is configured in the Phase setup and allows you to calculate the sum, mean, or median of fields included in your Evaluation Forms. You can add the resulting score to the Application Summary and use it in filters and exports. Use Phase Calculated Fields when you want to make a simple calculation using the results of your reviews, like averaging the responses submitted by different reviewers.
- Translation Tables: when configuring Exports or Calculated Properties, you can also apply a Translation Table to desired fields. This allows you, for example, to build a scoring system that converts the number found in a field to another number, based on the translations you have configured. Translation Tables also support ranges, allowing you to award a predetermined amount of points for test scores, GPAs, or other items that fall within set ranges of numbers.
- the Field Dictionary, which lists all fields available in the site, or
- the applicable Evaluation Forms, which display keys under each question.
- Default values are set for each category.
- A scale is established in each category to determine how many points should be awarded based on the option the reviewer selects.
- A total score is calculated by adding the five categories together.
- The let function establishes each variable (scored item) and sets the default value for each of them to 0.
- The switch function sets each category's score by establishing what amount to award based on the item the reviewer selects. The keys used here: rubric_gpa_scale, rubric_Degree_scale, rubric_work_exp_scale, rubric_letter_1_scale, rubric_letter_2_scale, are the keys created in each evaluation question (see Key in the image below). In this example, the reviewer manually selects which range the applicant's GPA falls into. The options available to the reviewer (represented here as case 1, case 2, case 3, case 4 and case 5) convert to a value of either 0, 2, 4, 6, or 8 points. This method of manually selecting a GPA range is used instead of capturing the GPA from the application, which may be a useful option when GPAs either aren't included on the application by default or aren't verified. The image below shows how the rubric_gpa_scale question is built on the Evaluation Form.
- The switch function is again used here, this time for the degree information. Similar to the previous category, this is designed for the reviewer to make a selection on a degree-related question. The reviewer's answer is here converted to a value of 0, 4, 6, or 8 points. The image below shows how the rubric_Degree_scale question is configured on the Evaluation Form.
- The switch function is again used to convert a reviewer's selection into a numeric score. This time, the function is converting the reviewer's response to a question about the applicant's work experience (rubric_work_exp_scale).
- Similar to the previous sections, the switch function is used here to award the application either 0 or 2 points based on how the reviewer responded to a question regarding the first letter of recommendation.
- The switch function here works the same as the one above, this time using the second letter of recommendation score.
- Lastly, the return function is used to calculate a final score and display its value. In this example, the score is calculated by adding the scores of each category (score_gpa, score_degree, score_work, score_letter1, and score_letter2). Other equations could be also be used here (e.g., multiplying the scores, subtracting values from each score, or dividing the scores by a value).
Working with Evaluation Calculated Fields
To configure an Evaluation Calculated Field:
- Navigate to Evaluation Forms in the Application Review area of the Settings menu.
- Build a new Evaluation Form or click on an existing one to edit it.
- After adding the fields you want reviewers to respond to, click Add Calculated Field at the bottom of the Evaluation Form.
Finding Evaluation Calculated Field Results
Once you've created your score using Calculated Fields, you can view the results in several ways.
To find the results of an Evaluation Calculated Field:
- Navigate to an application and click Completed Reviews from the Information Pane.
- Find your desired review and look for the results in the Calculated Scores area. You can also click Show Review Details to see the rest of the review.
- You can also find the results of an Evaluation Calculated Field in the Evaluations section of a PDF export.
- Alternatively, scores can be added to the Application Summary to feature them more prominently on each application record.
Calculating Scale Group Scores
When building an Evaluation Form, one of the Question Type options is Scale Group. This option allows reviewers to select from a scale of options to respond to your review questions. If you've implemented this option, you can use Phase Calculated Fields to calculate a sum, average, or median of the subscores from the Scale Group and display them. To do this:
- Build your Evaluation Form using the Question Type of Scale Group.
- Assign your Evaluation Form to a Phase.
- Add a Phase Calculated Field and select which subscores you'd like to calculate. Note that each Scale Group allows you to select one of the individual scoring components, the sum of each scoring component, or the average of each scoring component. For instance, as seen in the example below, you may have three components in your Scale Group, and want to add those components together. In this case, choose fieldName - Sum when building your Calculated Field.
- Once you select the field you're working with, you can then select how you want these fields to be calculated across different reviewers. For example, if you want to take the sum of the Scale Group scores calculated for each application, but then average them across all reviewers, select Average (Mean) in the second dropdown.
- In the example shown above, your Calculated Field will add up each reviewer's response to all three of the Scale Group questions (academicReview - Sum). It will then take that sum and average it with the sum calculated in other reviews of the same application. Alternatively, you could select each Scale Group question individually and have the software calculate the Sum, Average, or Median of each of those components across all reviews of the application. This results in three separate Calculated Fields.
- Complete the Calculated Fields area, adding a Label and Key for each field.
The results of Phase Calculated Fields are available to use in Exports and Filters.
Working with Calculated Properties
To configure a Calculated Property:
- Navigate to the Application Properties area under Application Setup of the Settings menu.
- Click Manage.
- The following page displays your existing Custom Properties and allows you to add more of them. Click Calculated Property from the menu on the left and drag it into your desired order on the right. This adds a new custom field, allowing you to configure it as desired.
- Once you've added a Calculated Property, the results for each applicant can be found by going to Application Properties in the More Options menu of an application record. You can also add scores to the Application Summary to feature them more prominently on each application record.
Using the Field Dictionary to Find Key Names
The Field Dictionary provides a database of every field available to you. Using this, you can determine what keys to use in your expressions. Keys that come from forms can also be found on the applicable form near the associated question.
Fields that aren't sourced from forms can be located in the Field Dictionary. To use this dictionary:
- Click Field Dictionary.
- From the Field Dictionary, click one of the Quick Links or use the Search bar to locate your desired field.
- The Field Dictionary provides information regarding the key to use (e.g., forms.standardized_tests.toefl.toefl_total_score), the data contained within the field, and the format the data appears in (e.g., String, Number, or Array). Review the information for the fields you want to use.
- Continue building on your expression or adding new Calculated Fields with different expressions.
- When ready, click Save Changes.