The API uses machine learning to label pixels from outdoor scenes into 11 classes and expose confidence. See applications, limits and privacy.
Direct answer
ARCore's Scene Semantics API runs a machine learning model on top of the camera and classifies outdoor scene pixels into 11 concepts, including sky, building, tree, road, sidewalk, vehicle, and person. It also provides confidence and prevalence by class. The feature can guide occlusion and contextual behavior, but it was designed for outdoors in portrait mode and has different quality between classes; should not be used alone for security decisions.
Semantics complements geometry
A mesh tells you where there is surface; the class indicates whether that region looks like a road, tree or building. Together, these layers enable more relevant visual rules.
Trust must control the experience
Large classes tend to be more reliable than smaller details. Set thresholds, uncertainty states, and fallback to not react to every pixel as true.
Outdoor and portrait use are product limits
The documentation does not promise quality indoors or in landscape orientation. The journey design needs to fit the conditions for which the model was built.
Person and vehicle urge additional caution
Dynamic labels can support occlusion, but they do not replace sensors or safety procedures. Avoid individual identification and retain only what is necessary for the function.
Nexus Reading
Semantics makes AR more contextual, but the business value lies in the rule that each class uses. A pilot must measure accuracy by scenario, visual impact, consumption and fallback rate.
FAQ
Does the API recognize indoor environments?
Not supported usage; it was designed for outdoor scenes.
Does it identify a specific person?
No. The return is a per-pixel semantic class, not individual identity.
Can it guide critical navigation?
Not as a single source. Signage, maps, sensors and human validation remain necessary depending on the risk.
Essential guides to delve deeper into the decision
This editorial analysis was produced by Nexus from the official sources below, consulted on September 15, 2026. The text is original and interprets practical implications for companies.
- Google ARCore — Scene Semantics API: classes, trust, compatibility, orientation, and supported uses.
- Google ARCore — What's new in ARCore: Official API availability history on Android, iOS, and Unity AR Foundation.
Date reported by main source: ARCore documentation and history consulted on September 15, 2026.