Users found the bot efficient, well-integrated, and easy to use, though some anticipated a learning curve.
Copilot received a SUS score of 67. Our chatbot received a SUS score of 89.5. That's a 33% improvement :)
The system at Window Nation was functional but inefficient. Without obvious issues, we worked hard to identify ways to improve it without disrupting the user experience.
This documentation helps Window Nation implement the bot using tools like Copilot Studio.
Survey Data + Average Scores
Design • Usability Testing
KEY POINT
100% of the users said that they would use this bot over the bot provided by Microsoft Power BI. They felt that the new bot was more nuanced, more in-tune with the business, and better at being proactive.
It's really hard to get work done when the problem is unclear.
AI bot's content was shaped by design insights + gaps, influencing its conversation & persona.
33% increase in usability!
Reflections • Future steps
Enterprise products don’t always offer the same user experience as consumer products, but the focus should be on functionality. The design doesn’t need to be liked, as long as it works effectively for the users.
It would be super cool to have a BI bot that can understand a users preferences and adapt to them. For example, if you ask the same questions when you come into office every morning, the bot could start anticipating your needs and have the answers ready before you even ask. It could also suggest insights based on your past queries, making your workflow smoother and more efficient.
In the future, we should aim for a more refined BI bot, that can learn a users behavior.
You don't have to like the design, as long as it works for the users.
Copilot received a SUS Score of 67. We strove to turn it's limitations into opportunities.
First, we tried a design to fit the Window Nation design system.
Inspired by modern chatbots like Google Learn About, we updated the design system with a fresh look.
Users found the design out-dated.
Stakeholders were okay with it.
Design • Usability Testing
CoPilot does not analyze data correctly, since it's vocabulary is different to that used at the company.
CoPilot can create useful artifacts like reports, but nothing else.
Make prompts tailored to marketing use cases.
Use visuals effectively, incorporate familiar graphs and charts.
Incorporate more multimodal exportable items to be used in meetings.
Sync team vocabulary with that used in the chatbot.
Case 1: Addressing a major change in data.
Example statement: "Why are sales in Atlanta down today?"
Notification proactively informs user of changes in data.
AI explicitly refers to data + displays source, allowing user to double check.
For example:
Notification
Launch in sidebar
AI refers to data
Answer
Click
notification
Type or
choose
prompt
Crafting customized flows to fill in gaps for marketing team.
Copilot suggested prompts are too generic for this use case.
Copilot does not use visuals effectively.
Text in WN Blue
Light pink background
Buttons are distinguished using shadows
Basic icons in WN iconography used to visually indicate common prompts
Secondary button in WN Blue with underline
Primary button with white text and WN Blue for button color
Elements of WN design system in our designs
= Limitations of Copilot
= Opportunities for our product
A quick usability test helped us understand that users wanted a new looking dashboard. Stakeholders still wanted us to stick to the old design system to some extent.
I focused on creating a Data Analysis card component that would visually support the textual data analysis.
More highlights of the final high-fidelity designs, based on user opinions:
Mid-Fi Explorations
High-Fi Design
User
We tested 4 users on 3 key tasks, noting areas of ease, excitement, struggle, and confusion.
Users loved the bot’s capabilities but suggested improvements:
Comparing data across time
Exporting videos for recap meetings
Displaying more monthly data for better analysis.
Explanation of how BI bot works
Standardized tag for title
Concise text for explanations
Ability to refer to historical data across weeks, months years.
Hyper-specific prompts to user
Easy data comparison through drop-downs
Future: allow for more types of data comparisons
STAKEHOLDER QUOTE
“Users shouldn’t use the tool blindly, the tool should teach them how to analyze data”
USER QUOTE
"I want the dashboard to be quicker at giving me general info."
USER QUOTE
“It’s hard to compare data because
things are on different pages”
USER QUOTE
“CoPilot has too much text, I have
to use ChatGPT to summarize”
USER QUOTE
”There’s so much going on, I forget what happened at the start of the month”
Underperforming Market
According to recent reports, Atlanta is experiencing severe weather, which may be causing postal-related delays.
Data Analysis
Analyze similar data
Go to source
Has this happened before?
How was the data analyzed?
Data Analysis Source
The analysis I provided was extracted from “Total Leads over Time by Source Type.” The data reflects direct mail conversions during the week of 5/17 to 5/23.
You can find the original visualization used for this analysis here.
Data Analysis
According to recent reports, Atlanta is experiencing severe weather, which may be causing postal-related delays.
Go to source
Analyze similar data
Data Analysis
Lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.
Go to source
Analyze similar data
Data Analysis
Go to source
Analyze similar data
Lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.
DATA ANALYSIS
1
1
Lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.
Lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.
Button 1
Button 1
Button 1
Source of data is useful
Allows user to double-check the AI analysis
Standardized tag for title
Makes the issue glanceable
Continuing prompt buttons
Helpful to have directly on the card since they are about the data
Especially useful since we are only designing for a finite set of questions
Expandable visuals
Don’t overwhelm the AI sidebar with visuals but are a helpful visual aid
Description of how data was analyzed
Allows user to re-analyze data + serves as a pedagogical tool
Icons are useful
Users are familiar with icons, glanceable
Source of data is useful
Allows user to double-check the AI analysis
Low contrast between items on the page
Difficult to skim information
Lack of visuals in bot
Users mention the need for visuals to feel engaged with bot
Notification text is too long
Difficult to skim quickly
Standardized notification text is faster
The Critique of WN Design System
I felt the design fell short, though Window Nation users seemed happy with it.
I learned that enterprise products often prioritize function over user experience.
Usability testing confirmed that users found the new chatbot valuable!
Always up for a coffee (no sweetener).
Let's connect!
This is the end of the case study!
Thank you for reading!
What we achieved ↓
MY CONTRIBUTION
I structured the ideation process, pushing the team to justify ideas due to their unfamiliarity with our research. I ideated 7 of the 11 ideas that Window Nation selected to move forward with.
How do you find a solution to an unclear problem?
Throwing out some good ideas hurt,
but we had to narrow down.
Users wanted a better AI bot, they just didn't know it before.
We finally had a clear direction: refine the existing Copilot bot to be more context-specific.
…maybe reverse the process!
Ideate first, then ask users to validate.
We conducted a design workshop that generated 50+ ideas
The process:
Window Nation's UX Team gave us insight on the ideas they wanted to prioritize. Many members of the UX team gravitated towards the same ideas, so it was clear to us that certain ideas were standout.
Implementing all 11 ideas in our timeframe was simply infeasible. We chose 5 narrowed down ideas and sketched out the way we saw these integrating with the dashboard.
And that's why our new research process was so effective!
Once users saw the possibilities, they became strongly focused on improving the AI bot. They also mentioned that they found the existing Co-Pilot bot difficult to use.
Copilot is Power BI's AI ChatBot which claims to streamline data analysis by generating insights, reports, and visualizations through natural language queries.
50+ ideas
11 narrowed ideas presented as sketches
We categorized our findings and brainstormed extensively as a team.
1
We refined these ideas individually, giving everyone time to think.
2
Concept testing with Window Nation to align ideas with their needs and constraints.
3
Drawings and annotations for this sketching activity by me!
User
Design Direction + Product Goal
Ideation • Problem Solving
CONTEXTUAL INQUIRY
INTERVIEWS
How has the introduction of the daily business intelligence (BI) dashboard changed the team’s approach to market analysis?
COMPETITIVE ANALYSIS
PRODUCT TEARDOWN
In what ways can areas within the greater umbrella of AI (e.g. ML) be integrated into this system?
COMPETITIVE ANALYSIS
LITERATURE REVIEW
In what ways can predictive modeling be connected to Power BI tool?
Our research questions were very broad. We explored a few methods based on these questions.
Research uncovered various issues — dashboard layout, navigation, terminology, and more — but no single, dominant problem emerged across methods, leaving us uncertain about where to focus.
Users find issues with the dashboard, but none of them are make or break.
Despite all our research, we still couldn’t pinpoint a central problem to address.
Research • Exploration
CONTEXTUAL INQUIRY
Users are generally satisfied with the dashboard.
Users find it frustrating to switch between dashboards on different tabs.
Different members of the marketing team use different pages on the dashboards, sometimes with no overlap.
Users like the UI of the dashboard. It is familiar, and an upgrade from the way they were doing things before.
COMPETITIVE ANALYSIS
Dashboards that are familiar to users are preferred.
AI able to lead the user to a solution without the user having to think about a detailed prompt are the most useful.
AI can use multimedia to "show" info.
PRODUCT TEARDOWN
AI results are badly integrated with the dashboard - very few useful insights generated.
Inconsistencies in date ranges, vocabulary, context, make it difficult to navigate quickly.
User
There was minimal overlap between the findings from each method.
Data-driven notifications.
AI pulls from external and internal data to notify users of trend changes in data.
Actionable suggestions.
AI provides suggestions for strategies to solve problems. AI outlines historical strategies to mitigate similar problems.
Easy data comparison.
AI supports user in comparing data faster than navigating between pages and visualizations.
Chatbot that focuses on useful marketing flows, understands the language of the dashboard, and creates useful assets.
Solution
Reverse the research process. Ideate first, then validate concepts with users to see what they will find most useful. Design, then validate again! That led to our design solution:
The marketing team is able to use the current dashboard, but slowly.
The dashboard may not effectively answer critical questions about marketing channels, customer segments, and budget optimization.
Examples of questions:
"Why did radio do badly in Austin this month?"
"Apply percentage change formula to ATL and LOS for today."
But, while leadership saw AI as a way to enhance insights, marketers didn’t express a strong need for it or major frustrations with the current setup.
This left us with an interesting challenge— how do we make their workflow faster when no glaring issues exist?
… and, data-driven AI is becoming a bigger trend in industry.
Problem
The Challenge
*content redacted for NDA
How do we find the right direction to improve efficiency when users aren’t expressing strong needs?
Context-Specific AI Bot Design
Improving efficiency of marketing team by building a nuanced AI Chatbot to address the key challenges in their daily workflows.
33%
INCREASE IN USABILITY AS MEASURED BY SUS SCORES
Market
Source Type
Broadcast Week
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Filters
BI Bot
i
Ask me anything...
Underperforming Market
According to recent reports, Atlanta is experiencing severe weather, which may be causing postal-related delays.
Data Analysis
Analyze similar data
Go to source
Has this happened before?
How was the data analyzed?
Data Analysis Source
The analysis I provided was extracted from “Total Leads over Time by Source Type.” The data reflects direct mail conversions during the week of 5/17 to 5/23.
You can find the original visualization used for this analysis here.
What should I do?
This week, you should invest 20% more into digital leads for Atlanta.
Generate a report
Contact a team member
Context
Industry Sponsored Project
Aug - Dec 2024
Team
Students:
Disha / Rachit / Natalie / Tim
Window Nation:
2 Designers / 1 UX Head /
1 Business Strategy Manager
Responsibilities
Business Strategy
Product Design
Contextual Inquiry
Product Teardown
Design Workshop
Tools
Figma
Power BI
FigJam
Microsoft Suite
The TLDR ↓
Process starts here ↓