ICRA 2025 BARN Challenge

About

Designing autonomous robot navigation systems has been a topic of interest to the robotics community for decades. Indeed, many existing navigation systems allow robots to move from one point to another in a collision-free manner, which may create the impression that navigation is a solved problem. However, autonomous mobile robots still struggle in many scenarios, e.g., colliding with or getting stuck in novel and tightly constrained spaces. These troublesome scenarios have many real-world implications, such as poor navigation performance in adversarial search and rescue environments, in naturally cluttered daily households, and in congested social spaces such as classrooms, offices, and cafeterias. Meeting these real-world challenges requires systems that can both successfully and efficiently navigate the environment with confidence, posing fundamental challenges to current autonomous systems, from perception to control. Therefore, the Benchmark Autonomous Robot Navigation (BARN) Challenge aims at creating a benchmark for state-of-the-art navigation systems and pushing the boundaries of their performance in these challenging and highly constrained environments.

At ICRA 2024, The Third BARN Challenge in Yokohama had six teams competed in the simulation competition and top four teams in the physical finals. We congratulate LiCS-KI, MLDA_EEE, and AIMS for winning the 1st, 2nd, and 3rd place of The BARN Challenge at ICRA 2024. We published the results and findings about how the state-of-the-art navigation systems work in highly-constrained spaces in IEEE Robotics and Automation Magazine.

At ICRA 2023, we successfully continued with our second BARN Challenge in London. Ten teams competed in the simulation competition and the top six teams advanced into the physical finals. We congratulate KUL+FM, INVENTEC, and University of Almeria for winning the 1st, 2nd, and 3rd place of The BARN Challenge at ICRA 2023. We published the results and findings about how the state-of-the-art navigation systems work in highly-constrained spaces in IEEE Robotics and Automation Magazine.

At ICRA 2022, we successfully launched the first BARN Challenge in Philadelphia. Five teams competed in the simulation competition and the top three teams advanced into the physical finals. We congratulate UT AMRL, UVA AMR, and Temple TRAIL for winning the 1st, 2nd, and 3rd place of The BARN Challenge at ICRA 2022. We published the results and findings about how the state-of-the-art navigation systems work in highly-constrained spaces in IEEE Robotics and Automation Magazine.

The Challenge

The BARN Challenge will take place primarily on the simulated BARN dataset and also physically at the conference venue in Yokohama.

The BARN dataset comprises 300 pre-generated navigation environments, ranging from easy open spaces to difficult highly constrained ones, and an environment generator that can generate novel BARN environments. The task is to navigate a Clearpath Jackal robot from a predefined start to a goal location as quickly as possible without any collision. The Jackal robot will be standardized with a 2D LiDAR, a motor controller with a max speed of 2m/s, and appropriate computational resources. Participants will need to develop navigation systems which consume the standardized LiDAR input, run all computation onboard using the provided resources, and output motion commands to drive the motors. Participants are welcome to use any approaches to tackle the navigation problem, such as using classical sampling-based or optimization-based planners, end-to-end learning, or hybrid approaches. The following infrastructure will be provided by the competition organizers:

• The 300 pre-generated BARN environments
• The BARN environment generator to generate novel environments
• Baseline navigation systems including classical (default DWA, fast DWA, and E-Band), end-to-end learning (e2e RL and LfLH), and hybrid (APPLR) approaches
• A training pipeline running the standardized Jackal robot in Gazebo simulation with Robot Operating System (ROS) Melodic (in Ubuntu 18.04), with the option of being containerized in Docker or Singularity containers for fast and standardized setup and evaluation
• A standardized evaluation pipeline to compete against other navigation systems

Competition Rules

During the competition, another 50 new BARN evaluation environments will be generated, which will not be accessible to the public. Each participating team is required to submit the developed navigation system as a (collection of) launchable ROS node(s). The final performance will be evaluated based on a standardized metric that considers navigation success rate (collision or not reaching the goal count as failure), actual traversal time, and environment difficulty (measured by optimal traversal time). Specially, the score $s$ for navigating each environment $i$ will be computed as

During the in-person conference in London, physical obstacle courses will be set up at the conference venue using cardboard boxes (see example below). A physical Clearpath Jackal robot will be provided by the competition sponsor, Clearpath Robotics, with the same standard sensor and actuator suites as in the simulation. However, a sim-to-real gap may still exist due to factors such as friction coefficient between physical wheels and venue floor, electric motor impedance and resistance, real-world sensor noise distribution, etc. The top three teams will be invited to compete in the physical competition. In case a team is not able to attend the conference in person, the organizers will run their submitted navigation stack on their behalf on the physical robot at the conference. The team who achieves the highest collision-free navigation success rate and shortest traversal time wins the competition.

Due to the overhead of the simulator in the simulation phase, we do not specifically constrain computation. We will use a computer with Intel Xeon Gold 6342 CPU @ 2.80 GHz to evaluate all submissions. However, during the physical competition, robot onboard computation will be limited to an Intel i3 CPU (i3-9100TE or similar) with 16GB of DDR4 RAM. GPUs are not available considering the small footprint of the LiDAR perception data.

Starting this year, we will introduce dynamic obstacles in our evaluation pipeline, both in the DynaBARN simulation environments and in physical test courses with iRobot Create robots acting as dynamic obstacles. As the first competition with dynamic obstacles, we will only reward good performance when facing dynamic obstacles in terms of extra bonuses, but not penalize poor performance. More details to come soon.

Leaderboard

Real World

2025

2024

2023

2022

Simulation

The scores and ranking have been updated using the new scoring metric (previous $4\textrm{OT}_i$ vs. current $2\textrm{OT}_i$, moving the previous performance upper bound from $0.25$ to the current $0.5$). See The BARN Challenge 2023 Leaderboard for the previous scores using the old metric.

Participation

Citing BARN

If you find BARN useful for your research, please cite the following paper:

Schedule

Organizers

Xuesu Xiao
GMU

Zifan Xu
UT Austin

Saad Abdul Ghani
GMU

Aniket Datar
GMU

Daeun Song
GMU

Peter Stone
UT Austin
Sony AI

Clearpath Robotics
Competition Sponsor

Links

• BARN Dataset
• BARN Paper
• BARN Video
• BARN Presentation at SSRR 2020
• Source Code to Customize BARN
• Source Code to Run Simulation Trials
• Example Simulation Trial Results
• The BARN Challenge 2024 Report
• The BARN Challenge 2023 Report
• The BARN Challenge 2022 Report

Contact

For questions, please contact:

Dr. Xuesu Xiao
Assistant Professor
Department of Computer Science
George Mason University
4400 University Drive, Fairfax, VA 22030, USA
xiao@gmu.edu