July 29, 2020
Thoughts on AR/AI Integration
Here’s a great article by Adobe that covers some of the ways that they are using an integrated AI/AR approach.
Another good starting point is this article on “Combining artificial intelligence and augmented reality in mobile apps”. It’s somewhat of a sales pitch for Fritz.ai, but it does contain a lot of useful links – some of which I have incorporated.
Other applications of AI that are useful in AR: * Object Detection – finding the boundaries of objects * Image Classification – this can be used to identify known objects in a scene and make a correlation to an object in the digital world * Pose Estimation – determining position of hands to control movement * Text Recognition – determine text (not always horizontally aligned) and convert to actionable content * Audio Recognition – voice commands to control movement
Moving AI to mobile devices
Before AI models were available to be run on mobile devices, most applications followed something along these lines:
- Grab data on the device
- Move it to a storage location
- Trigger some operation (possibly store the results as well)
- Respond to the device with the results
To be “believable”, we need better performance to combine with reality.
Animation from https://heartbeat.fritz.ai/combining-artificial-intelligence-and-augmented-reality-in-mobile-apps-e0e0ad2cfddc
Question???
Are there any performance guidelines available for maintaining a realistic application? Something for response times or update rate?
Mobile AR Development Frameworks
There are frameworks available for AR in both the Apple (iOS) and Google (Andriod) ecosystems. Apple’s offering is AR Kit, while Google provides AR Core. These frameworks are mostly for basic AR development, but they do provide some broad applications of AI. Walk through each link to highlight AI integration
Mobile AI Frameworks
Similar to the AR frameworks, there are also AI development frameworks available for both Apple and Google. In this case, Apple provides Core-ML while Google provides TensorFlow Lite.
Both frameworks appear to be ‘inference only’ by using a minimized network optimized to their particular hardware platform. This would be useful if you needed a more specialized implementation of an AI technique and wanted to integrate it yourself.
And then there’s React+TenforFlow.js
Thoughts going forward
One thought is to take our application of image recognition for trail camera use and apply it to a mobile platform. We may need some help getting a basic iOS and Android application created though. Maybe a cross-group project somewhere?