Verti-Bench
[Paper] [Video] [Dataset] [Code]
About
Recent advancement in off-road autonomy has shown promises in deploying autonomous mobile robots in outdoor off-road environments. Encouraging results have been reported from both simulated and real-world experiments. However, unlike evaluating off-road perception tasks on static datasets, benchmarking off-road mobility still faces significant challenges due to a variety of factors, including variations in vehicle platforms and terrain properties. Furthermore, different vehicle-terrain interactions need to be unfolded during mobility evaluation, which requires the mobility systems to interact with the environments instead of comparing against a pre-collected dataset. In this paper, we present Verti-Bench, a mobility benchmark that focuses on extremely rugged, vertically challenging off-road environments. 100 unique off-road environments and 1000 distinct navigation tasks with millions of off-road terrain properties, including a variety of geometry and semantics, rigid and deformable surfaces, and large natural obstacles, provide standardized and objective evaluation in high-fidelity multi-physics simulation. Verti-Bench is also scalable to various vehicle platforms with different scales and actuation mechanisms. We also provide datasets from expert demonstration, random exploration, failure cases (rolling over and getting stuck), as well as a gym-like interface for reinforcement learning. We use Verti-Bench to benchmark ten off-road mobility systems, present our findings, and identify future off-road mobility research directions.
Off-road Features
Verti-Bench is based on Project Chrono, a high-fidelity multi-physics dynamics engine with a platform-independent open-source design implemented in C++ with a Python version, PyChrono. Compared to other commonly used robotics simulators (e.g., Gazebo, Unreal Unity, PyBullet, MuJoCo, and IsaacGym with well-known physics limitations especially for differential-drive mobile robots), Chrono is especially suitable to simulate complex off-road vehicle-terrain interactions involving suspension, tire, track, and terrain deformation, varying terrain contact friction, vehicle weight distribution and momentum, motor, powertrain, transmission, and wheel torque characteristics, aggressive vehicle poses with all six Degrees of Freedom (DoFs), etc. In Chrono, vehicle systems and terrain properties are made of rigid and flexible/compliant parts with constraints, motors and contacts, along with three-dimensional shapes for collision detection.
Geometry
The geometry of Verti-Bench environments is represented as 2.5D elevation maps created by SWAE and real-world elevation data. To be specific, we physically construct vertically challenging terrain with boulders and rocks and use a Microsoft Azure Kinect RGB-D camera to create elevation maps of different real-world terrain surfaces. We then use SWAE, a scalable generative model that captures the rich and often nonlinear distribution of high-dimensional data, as a feature extractor to reduce the dimension of the real-world elevation maps while preserving the original elevation information in a latent space, from which samples can be drawn to generate new elevation maps that resemble real-world vertically challenging terrain. To further introduce diversity and quantification of Verti-Bench geometry, we scale the output of the trained SWAE to 30%, 60%, and 100% and denote them as low, medium, and high elevation level (Fig. top). Each Verti-Bench environment is generated with 1/3 probability of each elevation level. Fig. bottom shows the histogram of elevation values of all three levels of terrain geometry. High elevation environments also have the largest variance (most rugged terrain), while low elevation environments are smoother.
Semantics
We design seven rigid and three deformable semantics classes with different textures and distributions of physics parameters as shown in figure. To be specific, the seven rigid semantics classes, i.e., grass, wood, gravel, dirt, clay, rock, and concrete, associate with a normal distribution of friction coefficient. When a pixel is sampled to be a certain terrain type, its friction coefficient is sampled from the corresponding distribution. We fix the restitution coefficient to 0.01 for all rigid semantics classes. For the three deformable terrain classes, i.e., snow, mud, and sand, we adopt the deformable Soil Contact Model (SCM) based on the Bekker-Wong model to simulate terrain deformation after wheel interaction: SCM presents the underlying terrain by a 2D grid and assumes each cell can only be displaced vertically and does not maintain any history other than the current vertical displacement. We hard-code three sets of physics parameters, including cohesive effect, soil stiffness, and hardening effect, for three different deformability levels, i.e., soft, medium, and hard.
Obstacles
Off-road obstacles, like large boulders or trees, exist in real-world off-road environments, which are simply beyond vehicles' mechanical capabilities and hence need to be avoided. We also include natural obstacles in Verti-Bench to pose challenges to obstacle avoidance systems. For example, a large boulder triple the size of the vehicle is completely non-traversable, while a steep hill as part of the terrain may or may not be ascended with the right maneuver. We add natural obstacles as instances of the former. To further promote variation, we randomly sample the locations and types (different sizes of boulders or trees) of 10, 20, and 40 obstacles to place on each 129$\times$129 Verti-Bench environment, denoted as sparse, medium, and dense for obstacle distribution.
Vehicles
We also provide a set of vehicle platforms in Verti-Bench, with the possibility of adding new customized ones in the future, so that different off-road mobility systems can be evaluated on standardized vehicles. Compared to simplified vehicles in existing simulators, the Verti-Bench vehicles are more sophisticated and articulated, including engine/motor, drivetrain, transmission, suspension, steering mechanism, and wheel torque, whose responses to complex terrain interactions are simulated. To be specific, Verti-Bench provides nine types of off-road vehicles, which are sourced from Project Chrono, open-source real and simulated research platforms, and custom-created vehicles using 3D scanning and modeling (with a Creality CR-Scan Raptor 3D scanner) of real-world scaled vehicles. Those vehicles vary in terms of scale (1/10th, 1/6th, and full scale), chassis (4-, 6-, and 8-wheeled and 2-tracked), suspension (single- and double-wishbone, multilink, toebar leaf-spring, and special tensioning), steering (pitman-arm, rack-and-pinion, toebar, bellcrank/rotary arm, and differential), and tires (rigid and handling, excluding FEA-based models due to significantly reduced simulation speed).
Evaluations
We evaluate ten off-road mobility systems using Verti-Bench, ranging from purely classical, end-to-end learning, and hybrid systems.
Purely Classical
- PID: A controller that takes a local goal 10 m away from the robot on the global path and minimizes the error angle between the desired and vehicle heading by regulating the steering and maintaining a 3 m/s speed;
- Elevation Heuristics (EH): A controller that splits the elevation map in front of the current robot pose to five regions and drives toward the region with the most similar mean to the current terrain patch and lowest variance;
- MPPI: An MPPI-based planner that uses a 2D bicycle model for trajectory rollout and obstacle avoidance.
End-to-End Learning
- RL: A RL policy learned from trial and error;
- MCL: A RL policy learned from a manually designed curriculum;
- ACL: A RL policy learned using Automatic Curriculum Learning.
Hybrid (classical and learning)
- WMVCT: A planner based on a decomposed 6-DoF kinodynamic model (bicycle model for x, y, and yaw, elevation map for z, and neural network prediction for roll and pitch);
- MPPI-6: An MPPI-based planner with a learned full 6-DoF kinodynamic model for trajectory rollout;
- TAL: An MPPI-based planner with a 6-DoF kinodynamic model that learns to attend to specific terrain patches;
- TNT: An MPPI-based planner that samples based on traversability and then unfolds 6-DoF kinodynamics.
Gallery
Links
Contact
For questions, please contact:
Dr. Xuesu Xiao
Department of Computer Science
George Mason University
4400 University Drive MSN 4A5, Fairfax, VA 22030 USA
xiao@gmu.edu