Summary
Context
The product was a Korean music streaming service named "Melon", a mobile application.
My team had just renovated the recommendation system of the app, and the business goal was to increase the streaming counts of the recommended songs.
However, data scientists and engineers in the team concerned about "why" view counts of the music recommendation page were not growing despite their efforts. That was the observed problem that they initially wanted me to resolve.
Challenges
The product was new to me, and I was the first and the only UX researcher in the team without any UX resources.
I chose qualitative approach to answer "why" rather than "how many/much". I also decided to find ways to maximize efficiency of the research practice given a scarcity of resources.
Heuristic evaluation: I used this method to get the gist of major UI and UX issues in the app.
Information architecture: I created an information architecture based on the product to figure out possible bottlenecks in user flows.
Journey mapping: Relying on the outcomes from previous qualitative methods, I analyzed the interaction flow of streaming recommended music by creating a journey map.
Results
A complicated and unorganized UI prevented both short and long-term users from accessing the music recommendation page. Short-term users used up all cognitive resources before reaching the page. Long-term users created their own detour to listen to music, which prevent further music discovery.
Hence, I suggested three solutions: (1) Reducing the number of features in each page, (2) simplifying UI, and (3) developing a new user journey that facilitates new music discovery.
There were two unexpected findings: (1) User groups obviously preferred "already known" songs over totally new songs in the recommended list, and (2) such a behavioral tendency became stronger when they were looking for music right before doing specific activities like driving and jogging.
Impact
Based on the findings, I got an insight that recommending music with a good level of familiarity to users could be an effective way to increase streaming counts, as users who were searching for songs right before carrying out daily activities would gravitate to what's familiar to them.
The findings led me to do further investigation on the impact of familiarity in streaming songs. With strong supports of the team, I conducted a log data analysis and an online A/B test for this further investigation, and by incorporating the insights from it, the number of streaming counts increased by 6%.
Research Process
1. Heuristic evaluation
In this kind of evaluation, two or three trained UX researchers evaluate the usability of products or services.
This is relatively faster and more efficient resource-wise, but it does not take into account the voices of actual users.
In this stage, I focused on the method's strengths rather than its shortcomings because I needed to quickly sketch out the most obvious UI/UX issues of the product.
Adapting "10 Usability Heuristics for User Interface Design" by Jacob Nielsen, I operationalized criteria for a comparative analysis across three major pages in the product.
As a result, there were common UI issues such as a lack of visual saliency, a high level of redundancy, and too many functions and features packed in a limited space.
In terms of UX, customization was unavailable and visual intuitiveness was lacking.
2. Information Architecture
Information Architecture works like a blueprint showcasing possible interactions and user flows before diving into developing product or service development. I should note that an IA may guide the design but would not predict the real interactions that users will make.
I quickly drew an IA of the app's major pages (because there was none before) to figure out how the UI issues above would affect interactions and user flows.
I used Omnigraffle to create an IA, and then from heuristic evaluation found issues that might cause friction in a user flow (pink arrows in the picture).
The biggest friction was found between the "recommended songs" lists and a black streaming bar at the bottom layer of the page, which is shown in the figure on the right side.
This finding was critical because streaming counts of songs from the "recommended songs" lists were the most important KPIs for the business goal.
The page seemed to confuse users when they navigated through the flow amongst packed, indistinguishable items in each page.
One interesting point, however, was that several "shortcut" flows were connecting across features in different pages. I wanted to keep an eye on it to see if users would recognize those.
3. Journey map
figuring out how UI issues involve in user journeys
A journey map is also a blueprint that reveals how users achieve their goals and what they intend, feel, and think during the "journey" toward the goals.
Though there was initially neither a journey map nor an IA, I developed one as a scaffold to evaluate which stages of a user journey would suffer from the friction point.
Based on user scenarios and expectations, three phases in the user journey seemed to be affected by friction in the UI:
(1) Selecting a song in recommended list
(2) Adding recommended songs to the streaming bar
(3) Tediously repeating these two phases several times
Such points of friction in a user journey could be a barrier to short-term users.
But what about long-term users?
Could they have found a detour using shortcuts?
4. User test
Testing to learn how real users respond to "known" issues
I needed to carry out the former three steps to independently determine the UI/UX issues within 2 weeks of joining the team. Finally it was time to invite users.
Although there was no recruiting pool or system, I had built rapport with most of 50 team members and they were willing to volunteer the user test.
To determine the number of users for each group, I followed the famous rule of thumb by Nielsen & Norman, "Five are enough."
This rule does not work like a panacea but is highly useful to detect obvious usability problems, as long as a researcher acknowledges the result may not be generalized to represent the majority of users.
I could recruit a total 15 internal participants within a week, including five "short-term (up to six-month use)" users and five "long-term (up to 5-year use)" users. All of them were back-end and data engineers who were not engaged in the production level of development.
The age range of participants was from 20 to 30, and none of them had any movement disorders. Each session took 30 minutes maximum, including both a 15-minutes usability test and a 15-minutes semi-structured interview about their experiences.
The usability task was to look for songs they want to listen to (1) in the moment, (2) in specific situations like driving or jogging.
While performing the task, users were asked to use it as similar as in daily lives, following think-aloud protocol.
Results
1. Behaviors of short-term users differed from those of long-term users
All five short-term users hesitated or touched the wrong button when they had to go back and forth between recommendation lists and the streaming bar, as the IA and journey map indicated.
On the other hand, five long-term users developed their own ways to access their favorite songs by adopting several "shortcuts." However, long-term users were also confused when they were asked to break those habits.
2. Music recommendation page was hiding in a gap between two behavioral patterns
12 out of 15 users were not aware of the music recommendation page, or simply did not use it. Specifically, short-term users already used up all cognitive resources in order to discover songs they wanted to listen at the moment, and then to figure out how to streaming them.
In terms of long-term users, they were satisfied with using shortcuts and not motivated to try out newly added features.
Both user groups exactly mentioned UI issues found in heuristic evaluation and information architecture: The recommendation page was too dense, and the interacting between playlists and the streaming bar was awkward.
3. But both user groups liked the recommendation page, especially when...
When they were asked to look at the recommended songs, spending a few minutes in "select" phase of the user journey, both groups expressed interest.
One interesting and unexpected finding was that familiarity in songs enhanced likability (the mere exposure effect worked here!). Both user groups obviously preferred "already known" songs over totally new songs in the recommended list.
4. Contexts shaped users' behavioral patterns
Another interesting factor that shaped users' behavior was situational contexts.
Both user groups tended to omit the "repeat" phase in the user journey when asked to find songs for specific activities like driving or jogging, unlike when asked to search for songs at the moment.
They selected a large chunk of songs at a time, streamed them right away, and stuck to the chosen playlist. Selected songs fell mostly within their favorites and the novel ones were excluded.
It seemed that users' searching strategies became more conservative when they were ready to engage with other activities, unlike when they were simply asked to find music without any situational context.
A summary
1. Why don't users use music recommendations?
The recommendation page's UI was too dense and did not support full flow of interactions.
These issues prevented both short and long-term users from discovering the music recommendation page and attempting to incorporate songs from it into their listening habits.
Hence, I suggested three solutions:
(1) reducing the number of features in each page.
(2) simplifying UI.
(3) developing a new user journey that facilitates new music discovery.
2. Any other insights?
According to the user test, most users developed first impressions of the music recommendation page during the "select" phase in the journey map. They preferred familiar songs over new songs. Also, it seemed users' searching strategies became more conservative when they were ready to engage with other activities.
Combining these two results, recommending music with a good level of familiarity to users could be an effective way to increase streaming counts and lower churn rate. Users who were searching for songs right before carrying out daily activities would most likely gravitate to what's familiar to them.
This result was also against a common idea in the domain of recommendation system "recommended items should be diverse to prevent users from being annoyed by the lack of variety."
3. What was the impact?
The results of this project successfully convinced stakeholders, data scientists and engineers. During the presentation, I got a feedback like "Now I am sure why a UX expert is needed for improving a recommendation system." (which was delightful for me!)The insights from this project led me to a further investigation on the impact of familiarity on streaming songs. Ahead of it, I tweaked the question to convince my stakeholders who had the opposite belief: "Do users really prefer more diverse playlists?"
With the strong support of data scientists and engineers, I conducted a log data analysis and an A/B test in this investigation, and by incorporating the results of it, view counts of new users increased, and also streaming counts increased by 6%.