Assessing Bias Risk in a Multi-Rater, Continuous Performance Review System
Client Challenge
An organization implementing a redesigned performance review system sought to ensure that the new process did not unintentionally disadvantage women or people of color. The system represented a significant shift in both structure and mechanics: performance reviews moved from an annual cycle to quarterly check-ins, and ratings were no longer provided solely by a single manager. Instead, employees were permitted to request feedback from multiple higher-level colleagues, with final scores calculated as the average of all responses received.
While the intent of the new system was to promote richer feedback, broader perspective, and continuous improvement, leadership raised a critical equity concern. Prior research and internal discussion suggested that professional networks may differ by gender and ethnicity, potentially affecting who employees felt comfortable approaching for feedback and how likely those individuals were to respond. The organization wanted to understand whether the redesigned process produced equitable outcomes—or whether it introduced structural bias in access to feedback and resulting performance scores.
Key Question
The client asked a clear and high-stakes question:
Were employees receiving unbiased performance feedback regardless of gender and ethnicity, and were differences in performance scores—if observed—fair, explainable, and justified by objective performance indicators?
Approach
From Data to Action was engaged to conduct an independent, statistically rigorous evaluation of the new performance review system. The analysis was designed to assess both outcomes (performance scores) and process risks (representation and access to raters), using multiple complementary methodologies to ensure robustness.
The team partnered with HR, legal, and business stakeholders to define analytic guardrails, subgroup definitions, and interpretation standards, ensuring the work met both ethical and organizational requirements.
Analytical Framework
From Data to Action applied a multi-layered methodology, including:
-
Overall Performance Score Analysis
Examined average performance scores across gender and ethnicity to identify high-level differences and directional patterns. -
Score Distribution Analysis
Assessed the proportion of employees receiving high scores and low scores by demographic group, rather than relying solely on averages, to identify potential distributional bias. -
Representation Bias Testing and Access Analysis
Evaluated whether the gender and race/ethnicity distribution of those receiving high performing scores and low performing scores are similar to the gender and ethnicity population distribution—addressing whether a certain gender or ethnic group were disproportionately receiving high or low scores. -
Access Analysis
Evaluated whether employees across gender and ethnicity received feedback from a comparable number of raters and whether participation rates differed systematically—addressing concerns that network dynamics might influence access to evaluators. -
Subgroup Testing
Conducted analyses within more homogeneous subgroups defined by business sector and geographic location to reduce confounding factors and isolate potential bias effects. -
Justification Testing Using KPIs
For subgroups where score differences were observed, From Data to Action compared performance ratings against relevant business KPIs (e.g., productivity, quality, delivery metrics) to determine whether differences were aligned with measurable outcomes or indicative of potential bias.
This layered approach ensured that findings were not driven by a single metric or analytical lens.
Findings
The analysis produced a nuanced set of results. In several areas, the client observed strong and reassuring outcomes, including evidence that many groups were receiving equitable feedback and that score differences, where present, aligned with objective performance indicators. These findings increased leadership confidence in key aspects of the redesigned system.
At the same time, the analysis surfaced specific areas of concern, including pockets where differences in scores by gender or ethnicity could not be fully explained by performance KPIs alone. In these cases, representation patterns and access to raters suggested potential structural risks that warranted attention.
Recommendations
From Data to Action provided targeted, practical recommendations tailored to the organization’s context, including:
-
Refinements to rater selection guidance to ensure more consistent access to feedback
-
Calibration and governance mechanisms to reduce variability across teams and sectors
-
Monitoring metrics to proactively track equity outcomes over time
-
Communication and training strategies to reinforce fair feedback practices
Impact
By rigorously evaluating the new performance review system, From Data to Action enabled the organization to:
-
Validate elements of the system that were working as intended
-
Identify and address equity risks early, before they became entrenched
-
Strengthen the credibility and defensibility of performance outcomes
-
Reinforce leadership and employee trust in the review process
-
Embed equity considerations into the ongoing governance of performance management
This engagement demonstrated how data-driven analysis can move equity discussions from assumption to evidence—supporting fair, transparent, and sustainable people practices.
Next Case Study
Mapping the Workforce and Skills Landscape of the UK In-Silico Technologies Sector
From Data to Action anchored the research in advanced labor market analytics, drawing on large-scale job posting data, employer demand signals, occupational taxonomies, skills inference models, and longitudinal workforce trends.