Monitoring and evaluation

Why it matters

Every project needs a strategy for measuring the performance of the building before and after a retrofit. How much was the property improved, and did we get the expected benefits? If the property is not performing as intended, then it is vital to understand why so that issues can be fixed and key lessons can be fed into future projects.

Monitoring and evaluation can help you to:

• Adapt your retrofit project as you learn more about how upgrades are working in practice
  
  
• Spot problems and take action to correct them
  
  
• Ensure that you continue to deliver value for money
  
  
• Transfer learning to future retrofit projects
  
  

Key steps to take

1. Choose your approach to monitoring and evaluation - Your approach will depend on which measures you will be installing and what the intended outcomes are. For example, if the primary intention is to reduce costs for residents, then this should be the main focus of the evaluation.
   
   
2. Set a timeframe - The timeframe should reflect the level of evaluation required. Basic evaluation is around 3 months post-occupancy (ie, after the works have been completed), whereas a more advanced evaluation may take 18 months - 2 years post-occupancy.
   
   
3. Engage with residents - You will need to gain resident consent for monitoring, whether you are asking them to provide energy data, complete surveys or have monitoring equipment in their homes.
   
   
4. Collect baseline data - Before the retrofit measures are installed, baseline data should be collected so it can be compared with the post-installation data. This may include an occupant’s questionnaire, energy usage, water usage, moisture levels etc.
   
   
5. Ensure that the correct monitoring devices are installed as part of the project: Make sure you have the right tools for the job, for example, smart meters, relative humidity sensors or internal temperature sensors.
   
   
6. Review the data: Following the monitoring period, the data should be reviewed and compared against the baseline data.
   
   
7. Decide on any appropriate actions: If the building is not performing as expected then a strategy should be put in place for rectification measures. This may be immediate rectification of defects or a medium-term strategy for future retrofit measures to improve the energy performance further.
   
   

Success factors

Integrate the strategy into the early stages of the project: Evaluation shouldn’t be an afterthought but should be fully integrated into your retrofit project from the early stages. This will ensure it can be appropriately planned in terms of resource, equipment, and timescales.

Focus on what matters: The evaluation strategy should respond to the intended outcomes of the project and retrofit measures to be installed. For example, there is no need to measure water consumption if you are only installing external wall insulation and new windows.

Think about your residents and how they use their property: Monitoring can only provide helpful information when the homes are in use, that is, when people are living there. This has important privacy and ethical implications. Your strategy should be GDPR compliant, ensuring that you have informed consent from the occupants, and should minimise disruption.

Assess a range of dwellings: Data will vary according to a number of factors including property size, location, orientation and occupancy patterns, so it is important to assess a range of properties as part of your evaluation.

Set a plan for how to integrate lessons learnt into future projects: You may need to review internal systems and processes to ensure that data from the evaluation doesn’t get lost and is used to help improve future projects. You could set up a data sharing platform or a focus group to allow space to discuss and record key issues.

Deep dive

Monitoring and evaluation strategy

A monitoring and evaluation strategy is needed before detailed work starts on the homes. The strategy should include the following elements:

1. Which elements should be reviewed? This might include energy usage, water consumption, humidity levels, resident satisfaction or changes in health and wellbeing.
   
   
2. How will the baseline data be collected and collated?
   
   
3. What equipment/methods are needed for monitoring?
   
   
4. How long will the monitoring take place for after the retrofit has been completed?
   
   
5. Who will carry out the evaluation?
   
   
6. How will the evaluation be used to inform future projects?
   
   

Methods of monitoring

There are many quantitative ways of gathering information on how well the retrofit is performing:

• Recording energy usage: This could be as simple as looking through energy bills to assess consumption; however, this method could be very time consuming. Smart meters will be able to digitally collect and send data to a central platform which can then be analysed.
  
  
• Measuring the internal environment: This may include temperature, humidity and air quality. Smart devices can be installed within the property to record this data remotely or can be measured manually using portable equipment.
  
  
• Monitoring moisture levels in the building fabric: This will generally be carried out by a surveyor using a moisture meter at different intervals of time post-occupancy. This level of monitoring would usually only be carried out if there has been a history of damp or rot at the property, or if there is evidence of condensation and mould following the retrofit.
  
  
• Thermographic surveys: This can be used to spot areas of poor insulation and cold bridging and should be done by a suitably qualified surveyor to correctly interpret the results.
  
  

There are also qualitative approaches to finding out how effective the retrofit project has been.

• Occupant questionnaires and interviews: This may consist of a simple questionnaire delivered to homes within 3 months of occupancy. The responses should be collated to assess overall feedback, any issues that have arisen and any patterns in responses. You may choose to interview selected occupants to gain a more detailed response on certain issues.
  
  
• Most Significant Change methodology: Using this approach, researchers gather “stories” from project beneficiaries. These are then discussed by project stakeholders to select the most significant, capture project impact and identify the value of these changes. The methodology is quite structured (see link to guidance below), but is helpful for monitoring project impacts on an ongoing basis.
  
  
• Evaluation forms: These can be used after any significant intervention to collect data on health, wellbeing, feedback on the process and outcomes. This creates a baseline which is then re-evaluated using a follow-up form to help the scheme monitor improvement (or deterioration) over time. As well as helping identify opportunities for improvement, this data will feed into an overall evaluation identifying whether the scheme has achieved its objectives and desired outcomes.
  
  

PAS 2035 approach

PAS 2035 requires a three-stage approach to monitoring and evaluation which should be carried out by a suitably qualified retrofit evaluator. The first stage (basic evaluation) should be carried out for every retrofit project, and if the intended outcomes have not been met then the intermediate stage should be carried out, moving to the advanced stage if required. The three levels are as below:

• Basic monitoring and evaluation: This should be applied to every domestic retrofit project and should include a questionnaire which is issued to residents. This level should be completed within 3 months of handover.
  
  
• Intermediate monitoring and evaluation: This method will be used if the basic evaluation highlighted that the project outcomes were not met, or that there were unintended consequences following the retrofit measures. This will include several activities such as a site visit, air tightness test, monitoring fuel usage, occupant interviews and recording internal temperature and humidity. This stage should be completed within 6 months of handover.
  
  
• Advanced monitoring and evaluation: This should be applied if the intermediate evaluation indicates that further evaluation is required to resolve issues. This stage will include a formal post-construction review, a formal post-occupancy evaluation, a thermographic survey of the dwelling, monitoring of internal conditions for one year, sub-metering new building services and investigation of defects. This stage should be completed within 2 years of handover.
  
  

Further resources

What do I need to consider for post-occupancy evaluation?

Home Quality Mark Technical Manual, BRE – There is clear and useful guidance on post occupancy evaluation within BRE’s Home Quality Mark standard document.

Visit Website (https://www.homequalitymark.com)

Post Occupancy Evaluation and Building Performance Evaluation Primer, RIBA – This guidance from the RIBA outlines a structured approach to carrying out post-occupancy evaluations.

Visit Website (https://www.architecture.com)

How can I monitor the health and wellbeing impacts of my project?

Affordable Warmth and Health Evaluation Toolkit, Department for Energy Security and Net Zero and CSE - This toolkit is designed to carry out impact evaluations for fuel poverty projects more easily and effectively.

Visit Website (https://www.gov.uk)

EuroQol - EuroQol is a standardised evaluation instrument which can be used to describe and value health-related quality of life. The tools can help you to compare health outcomes from different types of interventions, and can be used to assess effectiveness in terms of QALYs (quality-adjusted life years).

Visit Website (https://euroqol.org)

Where can I find out more about the Most Significant Change methodology?

The “Most Significant Change” (MSC) Technique – Rick Davies and Jess Dart, 2005 – This guidance introduces the Most Significant Change evaluation methodology.

Visit Website (http://www.mande.co.uk)

Get in touch

If you would like to discuss monitoring and evaluation strategies for your retrofit project, please contact the RISE support team rise@turntown.co.uk

We would love to hear about your experiences. What has worked for your housing association? What lessons have you learned? What documents, reports or tools have you found most helpful? Please contact rise@turntown.co.uk if you would like to share your experiences.