AI / LLM User-Submitted Review Processing
My Role: Product Manager leading the product team (4 engineers, 1 designer) for post-booking applications, in this case the review form.
Background:
As the PM for Vacasa.com, my main directive was to drive conversions (bookings) through our site. Guest reviews had a clear impact on conversion rate, and ‘mature’ homes with more reviews and higher ratings converted significantly better than homes with low or no reviews. Corresponding to the CVR effects were the impact on homeowner churn: new units that had not yet received reviews were much less likely to be booked, and when a new unit went too long without a booking, the owner was significantly more likely to offboard (the math was very clear cut, if a home had not received a review within 30 days of launch on the platform, the home was almost guaranteed to churn). Therefore, maximizing review aquisition was critical to the business.
Taking up this challenge, my team partnered with marketing and embarked on a series of A/B experiments within our review solicitation flow to drive review acquisition. While we were successful (increasing review rate by ~20%), we noticed that reviews on the site had not moved to nearly that degree. Why? Because Vacasa’s Reputation Management team read through each and every review, as reviews could contain critical guest feedback for our property management teams — for example, if a guest told us that the refrigerator was broken, we knew we needed to fix that ASAP before the next guest arrives, and often this information only came through our reviews. The RM team was relatively large (at least 30 reps) but was still unable to keep up with the influx of new reviews, and so a backlog of 30,000+ unread reviews were sitting in the queue unprocessed, the oldest of which could be more than a month old. This bottleneck was preventing us from publishing reviews on the site, and from getting relevant feedback to our property management teams to fix issues with the home in a timely manner.
Working closely with RM, my team built a new in-house application for review processing that significantly improved RM’s efficiency and allowed them to cut through the backlog of 30,000 reviews and reduce the average time to publish a review to < 5 days. However, given the sheer volume of reviews, particularly during high season, the RM team would still sometimes fall behind.
A year later, with the rise of LLM’s, we identified several use-cases for AI that would make the RM team even more efficient.
Project
We identified three pain points in the review processing flow where AI could have a major impact:
Auto-publishing 5-star reviews
Review summarization and issue detection
Maintenance ticket and Response drafting
Auto-publishing 5-star reviews
The majority of Vacasa’s guest reviews received a 5-star rating. However, 5-star reviews could still include actionable information for our property management team despite being overall positive (e.g. ‘the light is out in one of the bedrooms’). So RM agents were forced to read through the text of every 5-star review, despite the majority of these not requiring any action.
That gave us the idea of using LLMs on our 5-star reviews to detect whether or not the review required the attention of an agent or could be published automatically.
We tested this by building an API connection with ChatGPT, where all reviews (with Personal Information masked) were submitted to the LLM, and through a series of prompts asked the LLM to identify any issues with the guest’s stay, in particular calling out common issues with vacation rentals. If the LLM identified any issues, the review would continue through the typical processing flow. But if the review was a 5-star rating and did not contain any issues, we could automatically publish it without involving any humans. We tested this process out with auto-publishing turned off at first, then compared the AI’s verdict to whether or not the RM agent had had to ‘work’ the review (e.g. respond to the guest or write a maintenance ticket to the prop management teams). Our goal was to maximize the number of reviews auto-published while keeping the false-positive rate near 0 (a false positive being a review that the AI would have published but an agent ended up working). After a few weeks of testing and tweaking our prompts, we were able to auto-publish roughly 50% of 5-star reviews while keeping the false-positive rate <2%. Once we turned auto-publishing on, given the share of 5-star reviews, a 50% reduction in 5-star review processing translated to a 30% reduction in overall review processing. We had saved our RM team from having to process a third of all reviews.
Review summarization and issue detection
While autopublishing cut down on 1/3rd of RM’s backlog, agents still had to spend their time reading through 1-4 star reviews and writing responses and maintenance tickets. The main challenge here was that each review could have multiple category comments (e.g. cleanliness, condition etc.) and reviews could be quite long and poorly formatted, resulting in quite a bit of reading. Agents also did not know which reviews to prioritize at a glance, and instead would have to read through each and every one in the order they came in.
Our solution was to collect the LLM’s responses from our auto-publishing flow and display those in the Review Manager UI, so that when an agent opened up a new review, they’d see both a short summary of the guest’s sentiment along with a bulleted list of issues that had been identified. This allowed agents to quickly scan a review and determine if it required their attention.
Maintenance ticket and Response drafting
Finally, if an review did require work — either a response to the guest or a maintenance ticket to property management or both — agents would still need to write these out for each review. Instead, when an issue was detected in the review, we asked ChatGPT to draft both a guest response and a maintenance ticket. Then the agent would only need to read the draft and tweak it before submitting.