As part of our Test Bench 1.0 update, we completely reformatted our previous Maneuverability test with a more rigorous testing methodology that better simulates a real-world environment. After all, manufacturers have added more and more features that are meant to improve adaptability across a wide range of obstacles, from combined navigational sensors for identifying things like mirrors or chair legs to retractable LIDAR turrets for sneaking under couches and coffee tables or even chassis lifting systems so they can climb over thresholds or thicker rugs. With all that said, robot vacuum navigation still isn't a solved problem, and this test illustrates the differences between good and great navigational performance.
When It Matters
You could argue that the main appeal of a robot vacuum is keeping a clean house, even while you're at the office or running a few errands. Consequently, an appliance that won't constantly get stranded on a rug, slam into your nightstand, or careen into a floor-length mirror is worth its weight in gold.
At RTINGS.com, we've made it our business to test robot vacuums based on cold, hard data, but that isn't always practical, as is the case here. Instead, we've developed a subjectively-scored testing methodology that considers a wide variety of parameters, all the while being consistent and applicable across all manner of robot vacuums.
While there are certainly similarities between this test and our Obstacle Avoidance test, also introduced in our Test Bench 1.0 update, the goal of the Household Adaptability test is to demonstrate how effectively a vacuum will deal with large obstacles, whereas the former is meant to evaluate performance regarding smaller hazards like charging cables, slippers, and yes, even pet waste. Together, both tests provide a cohesive overview of how well a robot vacuum can perform in a real-world environment.
Our Tests
We place the robot vacuum within our testing environment, with its base station positioned against the back wall, which is where it begins the Household Adaptability test.
We place a variety of furniture pieces within this space, which are selected to approximate the sorts of things a robot vacuum will encounter while cleaning. We place these items in predetermined spots to ensure that results across all robot vacuums we test are comparable. These obstacles are as follows:
- A pair of sled-base leather chairs in the middle of the room. These chairs don't have traditional legs but rather a single support that vacuums can either get stuck on or climb over. This obstacle also shows how thorough a vacuum will be when cleaning under furniture.
- A TV stand. This is meant to illustrate a vacuum's ability to clean under lower-lying furniture and how close a vacuum can get to cleaning around the TV stand's legs.
- A coat rack. Many robot vacuums can bump into this obstacle with enough force to jostle it or even knock it over. We evaluate performance here not only based on how much the vacuum bumps up against the rack but also on coverage around its legs. It's worth noting that the thin, steeply rising legs are also a hazard unto themselves, acting as a ramp that can lift the drive wheels of some robot vacuums off the ground and stop them in their tracks.
- A tasseled rug. Lightweight rugs are traditionally a pain point for many robot vacuums, as tassels can easily get wrapped around their brushrolls. In the case of more powerful robot vacuums, the entire rug may get sucked into its suction inlet. We deduct points here if we need to manually intervene to free the vacuum from being stuck on a rug.
- A floor-length mirror. Mirrors are another big bugbear for robot vacuums, specifically those that use LIDAR to navigate. A mirror will bounce return signals off its reflective surface, causing it to think that there could be another room beyond the mirror's surface. This, in turn, can cause some robot vacuums to collide with a mirror. Some newer models use camera sensors to properly identify mirrors for what they actually are and to disregard erroneous mapping data. We'll deduct points if the vacuum bumps into the mirror itself.
Video
The product tester positions the robot vacuum in the prepared testing environment, as outlined above. Then, they allow the vacuum to run a cleaning job within the confines of the test room, intervening as necessary if it gets stuck on an obstacle.
The tester will watch the recording of this process, paying particular attention to a vacuum's ability to negotiate around obstacles and its actual room coverage performance. They will then use a rubric to assign the vacuum a score based on these factors.
If recent robot vacuums are anything to go by, it seems the new frontier for robovac manufacturers is general maneuverability. Gee-whiz features like deployable arms for dealing with small hazards tend to take up a lot of air, but more subtle improvements like solid-state LIDAR sensors and lifting systems can do even more for a robot vacuum when it comes to effectively moving through a space without getting stuck.
That said, while robot vacuums with more sophisticated mapping and navigational sensors do quite well on this test, there isn't necessarily a 1:1 correlation between their general sophistication and performance. Case in point: the iRobot Roomba i4, with a comparatively simple vSLAM mapping system, achieves an impressively high score in this test not through fleet-footed maneuverability but rather its diligent room coverage and slow, careful navigation around furniture.
We're always looking for feedback about what we do, so if you've got any criticisms or concerns (or simply want to reach out and say hi!), contact us through feedback@rtings.com, on our forums, or on Discord.