RPA AI Recorder
Re-imagined the Power Automate Desktop recorder using GPT-V to combine UI automation with voice instructions, creating a more intuitive and accessible desktop flow creation experience
Product Designer
Microsoft
2023
Overview
Back in September 2023, OpenAI announce a new AI model (GPT-V) that is multi-modal. In other words, this new model can take various data such as an image, text, sound as an input and generate either an image or text or code as an output. I collaborated closely with a cross-disciplinary team across Paris, Athens and Seattle to explore how we could re-imagine UI automation and disrupt the RPA market.
Key responsibilities
- Re-imagined the PAD recorder to leverage user interface and cursor movement with voice instructions
- Reconciled different recording experiences (classic recorder, UI element picker, task coach)
- Ensured consistency across interaction and visual design
- Created end-to-end video demonstration for Ignite conference
- Collaborated with cross-disciplinary (Data science, PM, DEV) team across multiple locations
- Designed innovative UI automation approach using GPT-V multi-modal capabilities
Impact
The RPA AI Recorder represents a significant advancement in automation technology:
- Lowered barrier of entry for desktop flow creation
- Combined multiple input modalities (UI, cursor movement, voice) for enhanced user experience
- Positioned Power Automate as a leader in AI-first automation
- Selected as a key feature for Microsoft Build conference
- Demonstrated innovative use of GPT-V technology in enterprise software
- Created seamless integration between voice instructions and UI automation
Project Videos (3)
Power Automate Desktop AI Recorder
Introducing the AI-powered recorder for desktop automation
Project Gallery (19)
