Skip to the content

Image recognition and Augmented Reality

Schneider Electric – Circuit Breaker Identification App

Schneider Electric is a global specialist in energy management and automation with operations in more than 100 countries. They offer integrated energy solutions across multiple market segments.

Schneider Electric is a global specialist in energy management and automation with operations in more than 100 countries. They offer integrated energy solutions across multiple market segments.

Challenge

They wanted to demonstrate the value add that Image Recognition (IR) and Augmented Reality (AR) provides to the end user experience. The project formed part of their long-term strategic goal to deliver disruptive technology and value added services to their clients.

Solution

Hack & Craft utilised cutting edge technologies in Machine Learning multi-object detection algorithms to create an Image Recognition (IR) framework that identifies Circuit Breaker models from panel installations via real-time AR video

Hack & Craft’s mobile experts combined the IR framework with ARCore to create a unique tool that allows users to scan a room of Circuit Breaker panelboards, which are then identified and tracked in 3D space.

How We Did It

We formed a project team that included Schneider Electric stakeholders from the Innovation Department and machine learning and AR mobile experts from Hack & Craft.

The project team was able to leverage both Hack & Crafts extensive experience in computer vision, machine learning and mobile development, as well as Schneider Electrics deep product and customer knowledge.

Build

Hack & Craft used a unique four phase agile methodology (Discovery, Development, Roll Out and Continuous Delivery & Support), allowing incremental refinements based on stakeholder feedback to ensure best market fit.

During Discovery the team conducted an initial workshop to identify the core user stories, integrations and data sources that were required. Our UX Team then delivered a series of interactive UX/UI prototypes based on those user stories.

At the end of each sprint Hack & Craft deployed a working App, allowing Schneider Electric stakeholders to track deliverables and provide feedback to help steer development.

In parallel the Data Science team used our bespoke Machine Learning & AI toolkit to supplement the image training set by generating additional images that simulated features such as different lighting conditions, being out of focus, rotation, different backgrounds etc.

Tech

The App uses the ARCore framework to provide virtual object placement and tracking within the AR view that is displayed to the end user when identifying Circuit Breakers.

Hack & Craft’s bespoke Machine Learning & AI Toolkit was used to provide a multi-layered Image Recognition framework based on state of the art YOLOv3 real-time multiple object detection algorithms and TensorFlow.

Additional layers included a ‘width detection’ layer that calculates the distance of each circuit from the end user’s position, by using the known width of each Circuit Breaker Model after objects are identified. This resolved one of the key challenges with current AR technologies, due to the lack of accurate depth detection sensors on the current generation of phones. A further OCR layer provides validation of Circuit Breaker models and allows the reading of device settings.

Impact

As part of the Maintenance & Operations workflow, the App improves end user efficiency by providing time savings when identifying and documenting the model and layout of circuit breakers in existing or new installations.

The Image Recognition framework also provides a platform that can be used in the wider Schneider Electric business, for applications such as sales or customer support when devices need to be identified.

Circuit Breaker Identification App UX/UI

Step 1 – Detecting a vertical plane

 

Step 2 – Analysing the Installation, detecting circuit breakers

 

Step 3 – Identifying detected circuit breakers

 

Step 4 – All detected circuit breaker identified, analysis completed

 

Saved installation

All case studies

Close
See all case studies (17)

Outro

Science and technology are the principal drivers of human progress. The creation of technology is hindered by many problems including cost, access to expertise, counter productive attitudes to risk, and lack of iterative multi-disciplinary collaboration. We believe that the failure of technology to properly empower organisations is due to a misunderstanding of the nature of the software creation process, and a mismatch between that process and the organisational structures that often surround it.