About the client
Schneider Electric is a global specialist in energy management and automation with operations in more than 100 countries. They offer integrated energy solutions across multiple market segments.
Challenge
They wanted to demonstrate the value add that Image Recognition (IR) and Augmented Reality (AR) provides to the end user experience. The project formed part of their long-term strategic goal to deliver disruptive technology and value added services to their clients.
Key goals
- Empower field engineers with access to the organisation's knowledge.
- Reduce the time requirements for creating building specifications
- Provide a single source of truth for product data
- Drive engagement with the wider Schneider Electric digital ecosystem
Solution
Hack & Craft utilised cutting edge technologies in Machine Learning multi-object detection algorithms to create an Image Recognition (IR) framework that identifies Circuit Breaker models from panel installations via real-time AR video.
Hack & Craft’s mobile experts combined the IR framework with ARCore to create a unique tool that allows users to scan a room of Circuit Breaker panelboards, which are then identified and tracked in 3D space.
How we did it
We formed a project team that included Schneider Electric stakeholders from the Innovation Department and machine learning and AR mobile experts from Hack & Craft.
The project team was able to leverage both Hack & Crafts extensive experience in computer vision, machine learning and mobile development, as well as Schneider Electrics deep product and customer knowledge.
Build
Hack & Craft used a unique four phase agile methodology (Strategy, Development, Roll Out and Continuous Delivery & Support), allowing incremental refinements based on stakeholder feedback to ensure best market fit.
During Strategy the team conducted an initial workshop to identify the core user stories, integrations and data sources that were required. Our UX Team then delivered a series of interactive UX/UI prototypes based on those user stories.
At the end of each sprint Hack & Craft deployed a working App, allowing Schneider Electric stakeholders to track deliverables and provide feedback to help steer development.
In parallel the Data Science team used our bespoke Machine Learning & AI toolkit to supplement the image training set by generating additional images that simulated features such as different lighting conditions, being out of focus, rotation, different backgrounds etc.
Tech
The App uses the ARCore framework to provide virtual object placement and tracking within the AR view that is displayed to the end user when identifying Circuit Breakers.
Hack & Craft’s bespoke Machine Learning & AI Toolkit was used to provide a multi-layered Image Recognition framework based on state of the art YOLOv3 real-time multiple object detection algorithms and TensorFlow.
Additional layers included a ‘width detection’ layer that calculates the distance of each circuit from the end user’s position, by using the known width of each Circuit Breaker Model after objects are identified. This resolved one of the key challenges with current AR technologies, due to the lack of accurate depth detection sensors on the current generation of phones. A further OCR layer provides validation of Circuit Breaker models and allows the reading of device settings.
Impact
As part of the Maintenance & Operations workflow, the App improves end user efficiency by providing time savings when identifying and documenting the model and layout of circuit breakers in existing or new installations.
The Image Recognition framework also provides a platform that can be used in the wider Schneider Electric business, for applications such as sales or customer support when devices need to be identified.