Public Services: Correlates of Success in Performance Assessment

This project is funded by the ESRC Public Services Programme

The UK Government is placing ever greater weight on performance assessment of local public bodies as a policy tool; the devolved administrations (DAs) place some, but noticeably less, weight on it. In England, assessments are now an integral part of the policy of developing 'earned autonomy', which links some freedoms and flexibilities with a certain level of assessed performance. The Audit Commission publishes comprehensive performance assessment (CPA) scores for all principal local authorities in England (counties, districts, and unitary authorities). Its summary measure gives each authority an integer score between 1 ('poor') and 5 ('excellent'). The Healthcare Commisssion is now the body responsible for assessment of NHS bodies in England. Its summary measure gives each health authority a score of between 0 ('poorly performing') and 3 stars ('high performing').

Critics of this regime complain that:

  1. Any inspection regime is subject to "Goodhart's Law", viz., "When a measure becomes a target, it ceases to be a good measure".
  2. Smart inspectees like to live in smart places. Therefore there is reason to expect public bodies in rich places to perform better than public bodies in poor places. If so, rewarding good performance would tend to grant autonomy disproportionately to rich places, introducing inequity to the governance of the UK.
  3. Rewarding good inspectees with the promise of 'lighter touch' inspection is problematic. It implicitly concedes that inspection is unpleasant and onerous, and that it diverts public servants from service delivery to target-chasing, disproportionately in those authorities which need the most improvement and, it might be assumed, the least distraction.
  4. The scores may be unreliable and/or invalid. Reliability means that an identical subject should receive identical scores on repeated testing. Validity means that a score truly measures what it purports to measure: in this case, the relative performance of a local authority or health authority.

There is now a large volume of data on the comprehensive performance assessment (CPA) of public bodies in England: for instance all local authorities and all NHS trusts. In Scotland, local authorities are subject to CPA but health authorities are not. In Wales, neither local authorities nor health authorities are subject to CPA.

The Audit Commission has itself done some work on the correlates of success. We will model the performance of British public authorities using the best available data on predictor variables including class of authority, population, and indices of deprivation.

This research will address the following questions:

  1. Is performance correlated with the desirability of areas for living and working?
  2. How well does CPA work as a way of granting earned autonomy?
  3. What is the regulatory burden on authorities of light- and non-light-touch performance assessment?
  4. Are performance scores reliable?
  5. Are performance scores valid, both service-by-service and overall?
  6. What are the implications for the government's agenda of promoting New Localism through earned autonomy?

There will be three streams to the research: 1) quantitative analysis of the association between CPA scores and such predictors as the size, prevalence of multiple deprivation, and class composition of localities; 2) interviews with policymakers in all three countries to establish why they chose to rely on CPA, or chose not to rely on it; 3) interviews with participants in the inspection system where it exists, and with senior local authority and health authority staff where there is no CPA regime, to find out what costs and benefits participants find, and whether non-participants would like to be in a CPA regime.

We will use the early results to inform a larger application to the second round of the Public Services Programme, aimed at answering the larger question "Can governments avoid Goodhart's Law? If so, how?"