Vacasa -Review Manager App
My Role: Product Manager leading the product team (4 engineers, 1 designer) for post-booking applications, in this case the Trip Manager web application.
Background:
Guest reviews are crucial to Vacasa’s business. Like all online shopping experiences, reviews help shoppers make decisions, and the more and better reviews an offering has, the higher its conversion rate will tend to be. In Vacasa’s business model in particular, reviews were critical to not only rental conversion rates on our own website (Vacasa.com), but also our performance on other vacation rental channels where we listed our homes, like Airbnb and VRBO. Reviews were also a signal to our homeowners that Vacasa was maintaining the home and service levels, and there was a correlation between good reviews and a homeowner’s likelihood of staying with Vacasa. Therefore, it was important that we 1) collect as many reviews as possible, and 2) publish those reviews to the site as fast as possible, two KPIs that fell under my team.
Beyond conversion rate, reviews were also a key communication channel between guests and Vacasa, alerting us to issues that might exist with a home, e.g. “the dishwasher was broken” or “the guest bedroom didn’t get cleaned” or “the WiFi was weak.” When such an issue is reported, it’s crucial that the message gets passed along to the local management teams so that they can fix the issue as soon as possible, reducing the likelihood that the problem persists for future guests.
The Review Processing Bottleneck
In Spring of 2022, my team ran a series of A/B experiments with our review flow (part of the Tip Manager application) in order to increase review rate. While we saw reviews per reservation steadily increase as a result, we quickly identified that the real problem was downstream, in review processing. We could drive more reviews, but those reviews were getting held up in processing, and not getting published on the site. Instead, these were sitting in a processing backlog, sometimes for weeks or months. At it’s height, there were over 30,000 reviews in the processing backlog.
Review processing happened via an internal product, and thus wasn’t under our guest-facing team’s purview. However, we found that due to the constant reorganization typical of a tech startup, review processing no longer had a product team’s support, and so we were able to convince the business to allow my team to take over the full review lifecycle.
Reputation Management
Each and every Vacasa review passes through the Reputation Management team, a ~30 member division of Customer Experience, for processing. RM reads through each review and takes action based on the following:
1) Content standards - a review must not include lewd or inappropriate language, nor mention anyone specifically by name. It must be on topic, and related to the home or the guest’s experience with Vacasa management. Reviews that do not meet content standards are Suppressed, meaning they won’t be published to the site.
2) Issuing Maintenance Tickets - If a review mentions an actionable issue with the home (e.g. “the dishwasher is broken”) the RM agent should catch this and issue a Maintenance Ticket out to the local service team so that it can be fixed as soon as possible. After a ticket has been submitted, the RM agent moves the review into a new queue, where it will await maintenance completion. Once maintenance has been completed, the RM agent writes a review response (to be published alongside the review) notifying the guest (and future guests) that the problem has been fixed.
Problems with the Review Processing tool
After speaking with the Reputation Management team, it quickly became evident that the reason they weren’t able to work their way through the huge backlog of reviews in a timely manner was the limitations of the current tool they were using. The tool lacked critical features that could streamline processing and make the team more productive.
Worse, the basic workflow of the tool was built around the goal of publishing the best reviews as soon as possible. While this original goal made sense in a certain way, the problem was that it de-prioritized the second mission of the Reputation Management team: issuing maintenance tickets when guests notify us of problems with the home.
The original Review Processing tool, called “Reviews Draft”. This is the only view inside this tool. You’re able to see only one review at a time and have no access to the review backlog.
The problems with auto-assignment
When an agent would open up the old Review Processing tool (pictured above), they would automatically be assigned a review from the queue of unprocessed reviews. This auto-assignment prioritized reviews by rating, meaning that the best reviews would be assigned out to agents first, with the goal of publishing good reviews faster. But with the huge backlog of reviews, this meant that bad reviews — those most likely to contain actionable feedback about problems with the home — sat unprocessed for weeks or months before maintenance tickets could be issued and addressed. In effect, problems were more likely to persist from one stay to the next.
Agents had no ability to access the actual queue of unprocessed reviews — everything happened by auto-assignment. This meant that the team could not differentiate into specialized roles, as it wanted to — ideally, seasoned agents would take on “bad” reviews that required more training, while new agents (the team was in part flexible between RM and standard CX duties) could be assigned the easier, “good” reviews.
Inability to be proactive
RM wanted to take a more proactive role in identifying issues in a home, but the old tool made this difficult. The agent could see only one review and one score at a time. Was a single 3-star WiFi rating indicative of a real problem that would require the local team to investigate, or a one-off issue with a certain guest’s device? RM agents had no way to tell if an issue was recurring, and no easy way to search for a home’s previous reviews.
Problems with maintenance tickets and tracking
A third major issue with the old Review Processing tool was how it handled maintenance tickets and tracking maintenance work.
Agents were not able to create maintenance tickets directly from the tool. Instead, they were forced to click through to a different, all-purpose maintenance tool. There, they had to spend minutes of their time adjusting the settings of the maintenance ticket to tailor it to the review, including copying and pasting one of two templates (saved in Google Docs) used for review feedback.
After a maintenance ticket was submitted, the review was forced to go through a convoluted workflow. Since an agent must either publish or suppress each review before being auto-assigned the next, reviews awaiting maintenance would be suppressed, meaning marked to not show on the site (but not permanently so, as the suppressed status typically indicated). The agent would need to save that review in a separate, shared team Google Sheet, so that the team knew that the review was not in fact to be permanently suppressed but was awaiting maintenance before publishing. Then, agents would have to periodically check on that spreadsheet, click through to the maintenance ticket, and see if the local team had completed maintenance yet. Once maintenance was completed, the RM agent would need to go and find the review in an entirely different tool, write a response, and switch it from “Suppressed” to “Published”.
Review Manager
Based on extensive conversations with the Reputation Management team, my team built a new Review Manager application from the ground-up.
The New queue and Assignment
The first step in building a new workflow for review processing was to expose the queue of unprocessed reviews to the Reputation Management team. We called this the “New” view:
The New view allows agents to prioritize reviews waiting in the queue by sorting and filtering based on age and rating. The view also provides a summary of how many reviews of each rating are waiting in the queue. The queue is sorted by oldest reviews first by default, since the new goal for this product would be to process all reviews faster. From here, agents can assign reviews out to themselves, or, as team managers, assign them to other agents. The flexibility would allow agents to assign reviews out to specialists, e.g. team leads could handle the most difficult reviews.
Assigned view
Once a review is assigned to an agent, it enters the Assigned view. Here, agents can view their backlog of assigned reviews (by making sure the “My Work” toggle in the upper-right hand corner is selected) or all Assigned reviews by de-selecting the “My Work” toggle.
The Assigned view is where an agent will spend most of their day, moving from one review to the next. They’re able to sort and filter their queue based on age and rating. The new “Assigned On” column indicates the date of assignment, and can let agents and their managers know when a review has been sitting in an agent’s backlog for too long a period of time.
Review Detail view
By clicking into a review row, an agent is able to access the Review Detail view, which displays the review summary, comments, and ratings, and is where the agent actually processes the review.
At the top of the review detail view, the agent sees a summary of the reservation, along with links to the unit information and a link that can be used to view all previous reviews for the unit.
The category comments and ratings are largely the same as the previous Reviews Draft tool, with the addition of a new 90 day average column, indicating a home’s performance in each category over the past three months. This can alert agents to recurring issues with a home, for example persistent low WiFi scores which may instigate a maintenance ticket.
From here, an agent has three options to proceed:
1) The agent can publish a review, with an optional response, in the right-hand sidebar.
The agent may choose to publish the review while hiding the comment so that it doesn’t display on the site (in circumstances that violate our content policy).
2) The agent may suppress the review entirely by toggling “Show Rating” and then clicking “Suppress”
3) The agent may decide to issue a maintenance ticket. The maintenance ticket menu is opened by clicking the ticket icon in the right sidebar.
There are two types of Maintenance Tickets:
1) Response Required - this is mandatory maintenance that the local team needs to fix. The review won’t be published until the issue is fixed.
2) Feedback FYI - for less urgent issues, the agent can surface comments to the local team, who can use their discretion before taking any action.
By selecting either type, the tool autofills the Maintenance ticket template, assigning it to the appropriate local team member:
The new tool connects to Vacasa’s internal Maintenance Ticket API, so once the agent selects “submit” a new maintenance ticket is pushed off into that system. A summary of the ticket appears in the sidebar:
From here, if the ticket was “Response Required”, the review will exit the Assigned queue and enter the Response Required queue — more on that in a minute.
Finally, another addition to the Review Detail view is the Case History. Previously, agents were not able to track a review over its processing lifespan — who did what and when with the review. It was a black hole that made keeping track of changes impossible. Now, the case history logs each change made to a review, and also leaves an option for the agent to leave a note for the future.
Response Required view
If a review is awaiting an associated Response Required maintenance ticket, it moves out of the Assigned queue and into the Response Required queue. This view allows agents to quickly see which review are awaiting maintenance, and how long the maintenance is taking.
The Response Required view works a little differently from the others: its purpose is to remind agents to check in on maintenance tickets on a regular cadence. Check-ins typically involve asking the local team if maintenance has been completed, and reminding them of the task if not. Reviews due for a check-in (the soonest “Follow Up Date”) are sorted to the top.
Team View
Reputation Management team leads wanted a quick way to track their team’s progress — how many pending reviews an agent had assigned, how many Response Required tickets they had assigned, and what their oldest assigned review was. This was useful not only to keep the team accountable and work distributed evenly, but also to ensure efficient processing if an agent was pulled away on another task or off on PTO.
Review Search
Another common use-case that wasn’t available in the previous tool was search. Reputation Management Agents wanted to be able to look up and read reviews that had already been processed, to identify recurring issues and if any actions or outcomes had previously been taken. In addition this feature was useful to more than just Reputation Management agents, but to a plethora of roles at the company. A search feature would make reviews far more accessible and digestible to everyone, from local managers to customer service agents to software engineers, and thus the search feature was opened up to a wider audience than the review processing features.
Reviews could be searched for by a variety of attributes, including Unit, Reservation, Assignment and more. The results would display the high-level information about the review, including when it was last updated, who it was assigned to, and its current status.
By clicking in to a result, one could view the full review, and re-open and re-assign it if necessary.
Bulk Upload
One last additional feature we added into the Review Manager was a Bulk Upload feature. Vacasa often acquired units with previous reviews (used at the owner’s permission) that we would upload to display on Vacasa.com. These reviews would typically be collected in a CSV, and then directly uploaded to the database. There were two problems with the previous system:
Direct uploads to the DB required an engineer or other technical role, thus Reputation Management agents were not able to upload reviews themselves
There were no guardrails or data validation in place, so formatting could be inconsistent and error-prone, with incomplete required data
To make things as easy as possible, agents were able to download a blank CSV template for the upload directly from this view (via “Download Excel Template”).
Helper text provided a reminder of the necessary formatting:
To upload a CSV, the agent would select “Browse CSV file”. They also had the option to add a description for the upload, useful for tracking each batch of new reviews.
Lower on the page, the agent would see a list of previous upload batches, with information including the count of reviews, the status, file name, description, name of agent, and date of upload. The agent could also download the uploaded file.
