Checkout Flow A/B Testing - Vacasa
Note: the above video is from a company presentation delivered roughly halfway through this project.
My Role
Product Manager for Vacasa.com leading a scrum team of 6 SWE and 1 UX.
Background
Following a re-org that saw our team temporarily reduced in size, our team was forced to re-evaluate our roadmap in light of our reduced capacity. We decided to focus our efforts on a single major project (in addition to KTLO work) for the rest of the quarter until we were able to backfill the rest of the team. After a thorough analysis of the funnel, we decided that the Checkout flow was the best candidate for our optimization efforts.
Despite Checkout being at the bottom of the funnel, past the ‘shopping/consideration’ phase, we still had a surprisingly low conversion rate through this experience: only 25% of users who started the checkout process completed it.
Project
Working closely with our UX designer, we ran a series of user interviews on UserTesting.com to gather feedback about our Checkout process. Users were asked to walk us through the checkout process, narrating their internal thoughts. Then users were asked to interact with a series of alternative checkout mock-ups and provide the same off-the-cuff feedback.
Through this process, we identified a number of pain points that might be causing friction in our Checkout flow:
Information overload - interviewees were bewildered by wall-of-text sections like our House Rules and Rental Agreement. Some got bogged down reading these sections, while others skipped past despite the importance of the information.
Lack of unit and booking information - the first page of the current checkout flow did not contain any information about the home or the booking (e.g. basic house information like number of bedrooms, basic booking info like dates and total price). We found that many users bounced back to the previous unit page to review these details, exiting the checkout flow — and often not returning.
Too many steps — our three-page checkout flow was over-long and required too many clicks. This was a particular problem on Mobile devices, where all information needed to be presented in a single vertical column.
Poorly presented trip insurance add-on - we offered guests to add trip insurance as an extra charge on their trip, but the verbiage and presentation were confusing.
In addition to the above, we identified a number of other justifications for redeveloping our Checkout flow:
The Checkout flow had not yet been redesigned with our new component design system, and therefore 1) used styles that did not match with the rest of the site, and 2) did not benefit from the maintainability and consistency of the new design system.
The Checkout flow unlike some other sections of the site was not hooked up to a CMS, meaning even small text changes required an engineer to deploy.
Implementation
From the very start, we framed the Checkout project as less of a single ‘redesign’ than a series of A/B experiments where we could test, learn, and iterate by monitoring how user behavior changed with each new update. The tests would be run via Optimizely full-stack, with traffic split 50/50 between test and control. Within each test, we would monitor ‘micro-conversion’ metrics like form completion, step completion, etc. and compare test vs. control.
It was good that we took this iterative approach from the get-go, as our first few iterations failed to move the needle vs. control — after several weeks of testing we had yet to reach a statistically significant result. However, we were able to observe how behavior changed, and came to some interesting insights — the test version performed better on some of the micro-conversion metrics, but worse on others. This information directed our focus on subsequent iterations, so that we continued to keep the things that worked, while test new approaches to those that didn’t.
At the end of the testing (roughly a quarter) we were able to find a version that converted about 15% better than the original control. In addition to the CVR bump, we saw an even larger bump in AOV due to the increased adoption of travel insurance.