MARRS: Multimodal Reference Resolution System

*= All authors listed contributed equally to this work

Efficiently dealing with context is crucial for any dialog understanding process. This context possibly be conversational (counting on earlier consumer queries or system responses), visible (counting on what the consumer sees, for instance, on their display screen), or background (based mostly on alerts reminiscent of a ringing alarm or enjoying music). On this work, we current an summary of MARRS, or Multimodal Reference Decision System, an on-device framework inside a Pure Language Understanding system, chargeable for dealing with conversational, visible and background context. Particularly, we current totally different machine studying fashions to allow handing contextual queries; particularly, one to allow reference decision, and one to deal with context through question rewriting. We additionally describe how these fashions complement one another to kind a unified, coherent, light-weight system that may perceive context whereas preserving consumer privateness.