Hololens, Azure AD and cognitive services
HackWeekend in Acando is targeted to experiment with new technology, concepts and ideas to be seen. Many of us have often read posts about new technologies and tools but rarely took time to go hands on and try them. HackWeekend is the meeting place where we can experiment new tools and technologies with enthusiastic colleagues. In addition, we try to come up with projects and tasks that can be real use cases for possible customers.
One of our projects in the last HackWeekend was called: “TreasureHunt”, the idea behind it is to be able to capture images using Microsoft HoloLens and get information from people captured by using Microsoft Cognition Services (Face API). We could then use this information to find a Treasure based on clues registered to a specific person face.
The concept behind TreasureHunt is simple, we capture images by using the Air Tap gesture from the HoloLens and then send the image taken to the Face API service to get a match and then return data from the person face object found. Cognition services gives you the ability to recognize interesting info from the picture by using AI, like for example presence of water or trees or mountains, or maybe guess the age or gender of a person, or even detect if there might be nudity content.
Training Face API
We tried to go a little bit further and included the possibility to train Face API with faces from all users in Acando Azure AD tenant. For this purpose we used Acando Active Directory where most of our colleagues have updated their profile pictures. The workflow goes like this:
1. Run a console app that connects to our Azure AD via Microsoft Graph and gets all users.
2. For each user we get the picture, full name and email address. (You can get more info if you have higher permissions to the Azure AD).
3. Train Face API with each person found in Azure AD and assign some extra info (name and email).
Figur 2 FaceAPI training
Capturing with HoloLens
Once face API is updated and trained we proceeded with the creation of a simple interface for the HoloLens by using Unity3D. The interface consists of a Canvas scaled to fit the users vision, an Image object and some textboxes to show the data. We followed a very interesting Post from Windows Apps Team to implement the interface.
Figur 3 Unity interface
We included a property in the canvas object (InfoPad) to save the URL of the azure function to call for face recognition and match against Azure AD users.
Figur 4 InfoPad object properties
Calling Face API service from HoloLens
In order to make calls from HoloLens to any web service endpoint we use the WWW class found in the Unity framework. We get first the image captured and then send it to and Azure Function to be processed.
Figur 5 Analyse Image
From the Azure Function we proceed to get all faces from the image and send them to Face API in order to get a match from the face database trained. The list of matched people is then returned and shown in the Hololens interface.
In this post, you were able to see how to feed cognition services with data proceeding from Azure AD in order to achieve extra functionality. In our case the idea was to save extra information to one person’s face in order to have some kind of clue for the user to find the Treasure.
If you have any questions or comments, just contact me.
The full solution can be found here.