This is a new service – your feedback will help us to improve it.
Driving artificial intelligence adoption through procurement: How the UK is keeping its roads safe
Driver and Vehicle Standards Agency (DVSA), UK,
8 minute read
The UK’s DVSA have developed an approach that uses artificial intelligence (AI) to help identify and target garages that may not be testing vehicles properly
Governments around the world are at varying levels of readiness to procure, implement, and responsibly develop and use AI
New technologies need to work with existing technologies, processes and infrastructure, and be able to adapt to future demands
When exploring how best to use new technologies like AI, working collaboratively with the market can help to build internal capabilities, and save time and costs
The problem
Every year in the UK over 40 million vehicles receive a standards test called a ‘MOT’. These are carried out by more than 80,000 testers in around 23,000 garages across the country, generating over £1 billion for the UK economy.
However, some vehicles weren’t being tested properly, which endangers people’s lives. Inspection of authorised garages was resource intensive and there was limited knowledge to effectively target these inspections.
In support of the DVSA vision to help people stay safe on Britain's roads, the digital team wanted to explore data-driven approaches to help them conduct intelligent inspections of authorised garages carrying out MOTs, to ensure that vehicle standards are enforced.
To do this, the DVSA wanted to work with supplier partners to collaboratively deliver improvements as part of blended agile teams, and develop internal skills to use digital technologies.
Internationally, there’s increasing recognition that when developing and implementing AI systems, these need to be designed to be safe, fair and trustworthy. For example:
in May 2019, 42 countries signed up to the Organisation for Economic Cooperation and Development (OECD) Principles on AI
in June 2020, the Global Partnership on AI was created to support the responsible and human-centric development and use of AI, in a manner consistent with human rights, fundamental freedoms, and shared democratic values, as elaborated in the OECD Recommendation on AI
The approach
The DVSA proactively shares information early with the market ahead of starting their procurements. They do this by running 'meet the buyer' events to help potential suppliers find out about their work and plans.
Image showing a DVSA ‘meet the buyer’ event to share their digital journey since 2014; image source: the DVSA’s ‘Helping Digital Outcomes and Specialists suppliers to find out about us’ blog post
For this procurement, the DVSA hosted a supplier open day to explain the challenges they face, and gather initial ideas of how to address these and with which technologies.
They ensured that the invitation to tender (ITT) defined outcomes and clearly stated what challenges it wanted to overcome.
The ITT did not specify AI but focused on the use of technologies that would deliver the most effective outcomes. The aim of the procurement was to contract for digital services and capabilities that would help the DVSA team identify and deploy the right tools and systems, to address the delivery challenges of improving the inspection of authorised garages that conduct MOTs.
The project started with a set of mini discoveries, the insights from which would help the DVSA to:
improve the quality of MOTs by better supporting testers
know which garages presented the greatest risks of testing poorly
identify those applying to be involved in MOT that may present risks to the integrity of the MOT service
The DVSA didn’t have labelled data so used unsupervised machine learning, where a computer finds patterns in data without any prior information about what it should be looking for. In collaboration with supplier partners, the DVSA applied a ‘Local Outlier Factor’ (PDF; 226KB) clustering model against garage test data from a three-month period.
Image showing examples of different shapes of data clusters; image source: the ‘Part 1: Exploring data patterns using clustering’ blog post
The clustering model grouped MOT-authorised garages based on the behaviour they show when conducting MOTs, such as the test duration, time of test and result of inspection (compared to the expected result).
The DVSA created a risk (of testing incorrectly) score for each garage, which allowed them to rank garages and their testers and helped it identify regional trends.
The model was validated against those who had been identified as doing things incorrectly, ensuring that the model could learn what behaviours are good indicators of underperformance or potential MOT fraud.
An important consideration was the ability to explain the outcome of the risk rating without losing the integrity of the test. Having a human in the loop who interrogates and decides to take action on the risk score was crucial to make the use of AI successful.
All the data used for the AI system was data that was already collected by the DVSA and it didn’t include lots of sensitive data.
The results
Clustering techniques offered new insights that help the DVSA make predictions, and now support a more targeted approach to inspections at garages and testers with the highest risk scores. By identifying areas of concern in advance, examiners’ preparation time for enforcement visits has fallen by 50%.
Image showing a tester in a garage; image source: DVSA
There’s also been an increase in disciplinary action against garages, meaning standards are now being better enforced. As more garages are delivering better MOT standards, there are more cars on the road that comply with roadworthiness and environmental requirements.
“The DVSA digital team worked with our supplier partners to deliver an approach using machine learning. This helped build our capability, and, once they’ve finished working with us, will leave us with the skills and ability to develop this model. We’ll then be able to apply this to other areas of our business and support other parts of the Civil Service.” Neil Barlow - Head of MOT Policy and the MOT Service Manager at DVSA, responsible for the service that records MOT results
“Make use of innovative procurement processes to acquire AI systems - encourage collaboration between different bidders”
It was important to rely on a team of suppliers for project delivery, rather than just one supplier. Partnering with multiple suppliers and asking them to deliver the project in collaboration ensured that all relevant skills were available, and checks and balances were in place. One supplier developed the AI model and another supplier helped to test it.
“Focus on developing a clear problem statement, rather than on detailing specifications of a solution”
The requirements in the ITT focused on outcomes rather than the means of how to achieve those outcomes. This gave suppliers flexibility to select the technology that best fit this purpose, and ensured solutions were innovative and effective.
“Work with a diverse, multidisciplinary team”
The DVSA worked actively on upskilling internal teams and recruiting experts into the team where needed. This helped them to become a better customer for AI systems.
Project delivery was supported through close collaboration with the suppliers. Key to this was thinking as a single team and as partners, not contractors.
At a practical level, this meant being open about the problems that needed to be solved, the challenges that different solutions may present, and the costs of different options. This experience showed that openness makes things better, bringing real reward in getting value from the partnerships.
“Engage vendors early and frequently throughout the process”
Extensive pre-market engagement helped to better target potential AI system providers. The DVSA asked shortlisted suppliers to present their proposed approaches, which helped them to evaluate the different delivery options.
The challenges
The DVSA found that some of the WEF ‘AI Government Procurement Guidelines’ were harder to implement:
“Support an iterative approach to product development”
As elements of delivery might shift due to the agile nature of the work, it’s important to ensure this is reflected in the ITT and how it’s scored, and not only focus on lowest price.
“Consider during the procurement process that acquiring a tool that includes AI is not a one-time decision; testing the application over its lifespan is crucial”
Lifecycle management of the tool wasn’t fully considered upfront and became a challenge once the technology was developed. When the DVSA team identified this issue they worked with supplier partners to put together a plan to further develop the skills of the continuous improvement team. This ensured that the system continues to work effectively and meets users’ needs, as well as technical support that addresses issues related to hosting and live service failures.
Next steps
The DVSA is now using the AI approach to further develop its risk ratings data model to help identify where errors might be being made in the MOT, including a ‘predictive vehicle failure model’.
This will fine-tune their approach and help officers to better understand the likelihood of a vehicle passing or failing its MOT. They’ll then use this to target testers who repeatedly record results contrary to the prediction.
The DVSA is confident that their innovative use of AI is the right approach. It allows them to be flexible, and to develop approaches to risk rating that keeps getting better. They’re more able to respond to what they learn and any changes in tester behaviours.
