Rewarding Excellence in Instruction and Leadership
Practical Guidebook for Leaders: Components of the 2015-2016 REIL Score
What is the Purpose of the Practical Guidebook? The purpose of this Practical Guidebook is to provide you with information on how your REIL Score will be calculated for the 2015-16 school year. This guidebook answers the following questions:
How will observation data and student growth estimates be converted into performance scores to inform your REIL Score?
How will your final REIL Score be calculated based on the observation and student growth components?
How will your final REIL Score be used to determine your performance classification?
As shown in Figure 1, the 2015-16 REIL performance-based evaluation system has four major components: Leading Observation Instrument results (50 percent), school growth results (40 percent), student growth results of observed teacher cadre (5 percent), and district growth results (5 percent). Components of the REIL Performance Evaluation System
In general, as a building-level administrator, your evaluation results on each component are converted to a Performance Score on a 1 to 5 Scale. These performance scores are used to calculate a final REIL Score that is then used to assign a performance classification.
Page | 1
Convert LdOI and Student Growth Measures Leading Observation Instrument (LdOI) The LdOI is composed of five rubrics: 1. Setting & Communicating Direction 2. Building Relationships 3. Developing the Organization 4. Leading Instruction 5. Securing Accountability Across the five rubrics, a total of 22 unique elements are assessed and rated using a scale from 0 to 4. The majority of elements are rated on multiple occasions using evidence gathered through observations of your practice in different settings and through the review of documentation, including campus improvement plans, instructional support plans, educator goal plans, student achievement data, certified evaluator rating, management system artifacts and teacher/school observational data or artifacts. Over the school year, you will receive multiple ratings on the 22 elements within the LdOI’s five rubrics. At the end of the school year, data from the individual element ratings will be combined to determine a summative performance score on each of the rubrics. As shown in Figure 2, these rubrics individually contribute between 7 percent and 14 percent to a leaders’ REIL Score. LdOI Observation Components of REIL Performance Evaluation System
Page | 2
Converting LdOI Rubric Results to Performance Scores As previously noted, the individual element ratings for each LdOI rubric will be combined to determine a summative performance score for that rubric. This section details how performance scores are converted. Step 1: For each element in the rubric, average all ratings received within the element, rounding to the tenth decimal. This produces one final rating for each of the rubric’s elements. For example, let’s take a look at Leader A’s observation scores for the Setting & Communicating Direction rubric.
Rubric
Subject
Setting & Communicating Direction
Shared Purpose CIP: Goal Setting CIP: Action Plans School Resource Management
Ratings 2 3 3
3 3
3 3
4
3
2
3
4
Average of Ratings by Element 3 3 3 3
Step 2: The averaged element ratings are then combined to create one sum which determines the Total LdOI Points Earned for that rubric, again rounding to the tenth decimal. In this case, Leader A has earned 12 total points for the Setting & Communicating Direction rubric.
Rubric
Subject
Setting & Communicating Direction
Shared Purpose CIP: Goal Setting CIP: Action Plans School Resource Management
Ratings 2 3 3
3 3
3 3
4
3
2
3
4
Total LdOI Points Earned
Average of Ratings by Element 3 3 3 3 12
Step 3: Use the conversion chart (see chart below) to convert the total points earned on each rubric to a performance score on the Common 1 to 5 Scale.
Page | 3
Conversion of Total Rubric Points to REIL-TNG Performance Scores
Leader A received a performance score of 3 based on 12 Total Rubric Points earned for the Setting & Communicating Direction rubric. The same steps are used to calculate performance scores for the other rubrics. Below is Leader A’s performance scores for each of the LdOI rubrics.
Rubric
Total LdOI Points Earned
Performance Score
Setting & Communicating Direction
12
3
Building Relationships
9.5
4
Developing the Organization
13.5
3
Leading Instruction
12.5
2
Securing Accountability
14
4
Page | 4
Student Growth Measures Multiple student growth models are utilized in order to measure observed teacher cadre, school, and district-level growth across a wide range of teaching assignments. The Conceptual Guidebook provides details about the Value-Added Model & Categorical Growth Analysis which are used in a leader’s REIL Score. Basis Policy Research applies the statistical calculations to develop the growth estimates for each method. The following sections of this guidebook show how the growth estimates are converted into a performance score and the corresponding conversion charts. The calculations applied to your REIL Score are shown on the scorecard drilldowns in RDSS: Profile.
Observed Teacher Cadre Performance Score The Observed Teacher Cadre Growth component of the REIL Score for leaders is broken out into its component measures. The Composite Performance Score for each measure reflects the value that is entered on the REIL Scorecard.
*This table is a screen shot from RDSS: Profile of a leader’s Observed Teacher Cadre Growth Component drill down displaying each of the contributing student growth models used by the different members of a leader’s observed teacher cadre.
Step 1: Each teacher contributing to a particular component measure is weighted equally. For example, when looking at Leader A’s Custom Value-Added (VAM-C), there are three teachers and each contributes 33.33% to the overall Composite Performance Score to give Leader A score of 3.33 (see screenshot below). In other words, the Composite Performance Score Across Teachers represents a weighted average of each teacher’s individual score.
*This table is a screen shot from RDSS: Profile of a leader’s Observed Teacher Cadre Growth Component drilling into individual contributing teacher composite performance.
Page | 5
Step 2: The leader’s final Composite Performance Score is then calculated by applying the measure weights associated with each component measure. In Leader A’s example is shown below.
Growth Measure VAM-G CGA-G VAM -C
Composite Performance Score Across Teachers 3.67 4.27 3.33
Weights
Composite Performance Score
50.0% 25.0% 12.5%
(3.67 x .50) + (4.27 x .25) + (3.33 x .125) = 3.32
*Please note that all numerical values displayed in RDSS: Profile drill downs are rounded to two decimal places. In certain instances, when rounded to two decimal places for display purposes only, the numerical values displayed for the weights will not sum to 100%.
Leader A’s Observed Teacher Cadre performance score is 3.32.
School-level Growth Due to there being multiple student growth models, there are likewise multiple student growth models that affect a leader’s school-level growth. Just as the Student-level Growth component of the REIL Score is broken out into its component measures, so too is a leader’s School-level Growth Component of the REIL Score. For example, Leader A’s REIL Score, as shown below, is composed of Galileo Value-Added (VAM-G), Galileo Categorical Growth (CGAG), and Custom Value-Added (VAM-C). The Composite Performance Score for each measure reflects the value that is entered on the REIL Scorecard.
*This table is a screen shot from RDSS: Profile of a leader’s School Student Growth Component drilling into the different component measures affecting the leader’s REIL Score.
Step 1: Each tested subject contributing to a particular component measure is weighted in proportion to the number of students tested. When looking at Leader A’s Galileo Value-Added (VAM-G), there are two tested areas (Math and Reading). For example, Math and Reading have the same amount of students contributing to VAM-G; however, they would be weighted differently if the N counts were not equal. Using these weights, the two subjects are combined to create the Composite Performance Score. Page | 6
*This table is a screen shot from RDSS: Profile of a leader’s School Student Growth Component drilling individual contributing assessments administered within each Student Growth Component.
The leader’s final Composite Performance Score is then calculated by applying the measure weights associated with each component measure. In Leader A’s VAM-G performance score is shown below.
Subject Math Reading
Composite Performance Score Across Grades 3.50 4.00
Weights
Composite Performance Score
50% 50%
(3.5 x .50) + (4.0 x .50) = 3.75
*Please note that all numerical values displayed in RDSS: Profile drill downs are rounded to two decimal places. In certain instances, when rounded to two decimal places for display purposes only, the numerical values displayed for the weights will not sum to 100%.
District-level Growth District-growth makes up 5% of a leader’s REIL. When calculating district growth, the same process described previously as part of the school-wide growth applies and is completed for all schools within the district. Each of the students, who have data associated to a particular assessment, become part of your REIL Score.
*This is a screen shot from RDSS: Profile showing a leader’s District Student Growth.
Page | 7
Calculating the REIL Score & Performance Classification Scorecard Matrix Within the REIL Score model, numerous scorecards are developed in order to represent the various educator assignments across the REIL alliance. Shown below is the 2015-16 REIL scorecard matrix for leaders. 2015-2016 REIL Scorecards Group 1 Leaders LdOI Measures REIL Scorecard Code
Observed Teacher Cadre Growth
School Growth
Setti ng & Devel opi ng Bui l di ng Lea di ng Securi ng Communi ca ting the Rel a tions hi ps Ins truction Accountabi l i ty Di rection Orga ni za tion
VAM-Z
VAM-G
CGA-G
VAM-C
District Growth
Average of Observed Teachers' Individual Performance Scores
VAM-Z
VAM-G
LR_GR01_003
9%
7%
11%
14%
9%
25%
10%
5%
5%
3%
2%
LR_GR01_004
9%
7%
11%
14%
9%
20%
10%
5%
5%
5%
3%
2%
LR_GR01_038
9%
7%
11%
14%
9%
25%
10%
5%
5%
3%
2%
LR_GR01_136
10%
7%
12%
14%
7%
30%
10%
5%
LR_GR01_152
9%
7%
11%
14%
9%
6%
4%
25%
10%
5%
CGA-G
5%
*A larger version of the scorecard matrix is available is in the appendix.
The placement on a specific scorecard is driven by the observation and assessment data tied to a particular leader and observed teacher cadre. Going back to the example of Leader A, let’s look at what data have been connected.
LdOI Observation Observed Teacher Cadre Growth School-level Growth District-level Growth
Measures Setting & Communicating Direction Building Relationships Developing the Organization Leading Instruction Securing Accountability
Performance Score 3 4 2 3 4
Average of Observed Teachers’ Performance Scores
3.32
VAM-G CGA-G VAM-C VAM-Z VAM-G
3.75 5.00 3.36 2.75 3.70
Page | 8
Based on the measures calculated, Leader A would be placed on scorecard LR_GR01_038. The observation, observed teacher cadre growth, school-level growth and district-level growth measures are inputted on the scorecard. Since each measure has a specific weight, the performance scores will be multiplied accordingly in order to get the total points. Leader A’s REIL Score will now be calculated in order to demonstrate this final step. Scorecard: LR_GR01_038
Performance Score (1-5) Setting & Communicating Direction Building Relationships Developing the Organization Leading Instruction Securing Accountability
Observation Measures
Scorecard Weights
Points Earned
Possible Points
3
x
9
=
27
45
4
x
7
=
28
35
3
x
11
=
33
55
2
x
14
=
28
70
4
x
9
=
36
45
School-level Growth
VAM-G CGA-G VAM-C
3.75 5.00 3.36
x x x
25 10 5
= = =
94 50 17
125 50 25
Observed Teacher Cadre Growth
Average of Observed Teachers’ Performance Scores
3.32
x
5
=
17
25
District-levelGrowth
VAM-Z VAM-G
2.75 3.70
x x
3 2
= =
8 7
15 10
345
500
REIL Score *Points earned are rounded to the nearest whole number
Determine Performance Classifications The REIL Score also results in the generation of a performance classification. The state requires all classroom leaders to receive one of the four classifications: Highly Effective, Effective, Developing, or Ineffective. There may be further LEA-specific sub-classifications (e.g., Effective 1 and Effective 2). As shown in Table below, cut points are placed on the REIL Score continuum to identify the range of scores associated with each performance classification. For example, Leader A’s REIL Score was a 345 which places him/her in the “Effective” range. Table 7. Converting REIL Scores to Performance Classification REIL Score Range
Performance Classification
400 to 500
Highly Effective
300 to 399
Effective
200 to 299
Developing
100 to 199
Ineffective
Page | 9
Appendix Student Growth Measures Conversion Charts Benchmark
Measures
Population
VAM-G,C,E*
REIL/REIL-TNG
Student Growth Value-Added
Performance Scores
Growth
Method to Evaluate
1
1.5
2
below
-1.75 to
-1.25 to
-1.75
-1.26
-.76
2.5 -.75 to
3 - -.25 to
.26
+.25
3.5
4
4.5
5
+.26 to
+.76 to
+1.26 to
above
+.75
+1.25
+1.75
+1.75
* Galileo, Custom, and GOLD
Method to Evaluate Student Growth
Assessment
Categorical Growth
Galileo
Performance Scores
Growth
Benchmark
Measure
Population
1
2
3
4
5
CGA-G
REIL/REIL-TNG
1.0 - 1.4
1.5 - 1.7
1.8 - 2.2
2.3 - 2.5
2.6 - 3.0
Page | 10
