Skip to main content
Back to Works

Design and AI

·

Autodesk · 2018–2020

Taming the AI/ML Hype Cycle

Yesterday it was machine learning and chatbots. Today it's AI and agents. How do we explore and experiment without giving up judgement?

Autodesk Assistant chatbot iterations showing the move from humanoid avatars to abstract iconographyAutodesk Assistant chatbot iterations showing the move from humanoid avatars to abstract iconography

The Challenge

In the information age, there is always something new to get excited about. That excitement is fueled and rationalized by the seemingly exponential increase in the pace of change and the need to keep up. Even if the hype is warranted, a sober reflection on the promise, and cost, of that change is critical.

With the first wave of machine learning-powered chat bots, tech companies saw an opportunity to dramatically improve the efficiency of the support experience. The relatively finite knowledge space around a particular product paired with the faux-conversation capabilities of early ML-powered chatbots promised untold savings… if only we could crack the puzzle of customer engagement. Cue the advent of personified chatbots.

Vendors popped up faster than weeds in a garden. They offered engaging conversation-ish models that, paired with a humanoid chatbot with just the right appeal, would encourage customers to pour out their support and pre-sales questions. Contracts were signed.

At Autodesk, the cost of continuing to pursue this avatar-centric approach was getting expensive. That was enough to get some attention for voices that had been skeptical of the premise. But how could the truth be sorted out?

Otto chatbotAva chatbot v1Ava chatbot v2Autodesk Assistant logo

Approach

  1. 1.

    Academic Research Survey

    The first clues had already been readily available from numerous published studies in the field of Human-Computer Interactions (HCI). These included not just the Uncanny Valley notion, but more specific findings showing that text-only interaction produces more disclosure and more positive impressions than cue-rich alternatives. These studies also showed that avatars specifically have negative effects on user comfort. The importance of this kind of research early on cannot be understated. With examples few and far between and our own studies still underway, existing academic information can be critical in assessing the early hype.

  2. 2.

    Competitive Study

    Since this was as much a race to market as an experiment with new technology, we knew that understanding what competitors were doing would be important. There was some early attention to this, but much was mediated by vendor interaction and, at that stage, there weren't that many examples. We embarked on a quick, but thorough review of similar experiences in our space. The results pretty clearly showed a move away from anthropomorphic imagery.

    Competitive avatar comparison
  3. 3.

    UX Studies

    The first two caches of information allowed us to pursue a new, non-humanoid approach to what was now called Autodesk Assistant. Since a large contract renewal was upcoming with the original avatar vendor, we had to work fast. Generating ideas with internal stakeholders, we quickly set them in front of users to gauge reactions and impact on usability. The approach was a mix of brand-building creative and remote usability rolled into a single effort. The usability results helped sell the customer-chosen version internally and we had a winner… just in time to avoid signing the contract.

Outcomes

Over $1M saved

Return to focus on the information customers need

More consistent and cohesive branding