Predicting the performance of intelligent multi-robot systems is advantageous because running physical experiments with teams of robots can be costly and time consuming. Controlling for every factor can be difficult in the presence of minor disparities (i.e. battery charge). Access to a variety of environmental configurations and hardware choices is prohibitive in many cases. With the eminent need for dependable robot controllers and algorithms, it is essential to understand when real robot performance can be accurately predicted. New prediction methods must account for the effects of digital and physical interaction between the robots that are more complex than just collision detection of 2D or physics-based 3D models. In this paper, we identify issues in predicting multi-robot performance and present examples of statistical and model-based simulation methods and their applicability to multi-robot systems. Even when sensor noise, latency and environmental configuration are modeled in some complexity, multi-robot systems interject interference and messaging latency, causing many prediction systems to fail to correlate to absolute or relative performance. We support this supposition by comparing results from 3D physics-based simulations to identical experiments with a physical robot team for a coverage task.
C. Melhuish, M. Wilson and A. Sendova-Franks, “Patch Sorting: Multiobject Clustering Using Minimalist Robots,” Proceedings of 6th European Conference on Advances in Artificial Life, Springer, 2001.
A. T. Hayes, A. Martinoli and R. M. Goodman, “Comparing Distributed Exploration Strategies with Simulated and Real Autonomous Robots,” Proceedings of the 5th International Symposium on Distributed Autonomous Robotic Systems (DARS’00), Springer Verlag, Berlin, 2000.
A. Boeing, S. Hanham and T. Braunl, “Evolving Autonomous Biped Control from Simulation to Reality,” Proceedings of International Conference on Autonomous Robots and Agents (ICARA’04), 2004.
K. Easton and A. Martinoli, “Efficiency and Optimization of Explicit and Implicit Communication Schemes in Collaborative Robotics Experiments,” Proceedings of the IEEE International Conference on Intelligent Robots and Systems, 2002. doi:10.1109/IRDS.2002.1041693
“SIMPAR,” 2008. http://www.simpar.org/.
R. Brooks, “A Robust Layered Control System for a Mobile Robot,” IEEE Journal of Robotics and Automation, Vol. 2, No. 1, 1986, pp. 14-23. doi:10.1109/JRA.1986.1087032
M. Brady and H. Hu, “Software and Hardware Architecture of Advanced Mobile Robots for Manufacturing,” Journal of Experimental and Theoretical Artificial Intelligence, Vol. 9, 1997, pp. 257-276. doi:10.1080/095281397147112
T. Balch and R. C. Arkin, “Behavior-Based Formation Control for Multirobot Teams,” IEEE Transactions on Robotics and Automation, Vol. 14, No. 6, 1999, pp. 926- 939. doi:10.1109/70.736776
M. J. Mataric, “Issues and Approaches in Design of Collective Autonomous Agents,” Robotics and Autonomous Systems, Vol. 16, 1995, pp. 321-331. doi:10.1016/0921-8890(95)00053-4
M. J. Mataric, G. S. Sukhatme and E. H. Ostergaard, “Multi-Robot Task Allocation in Uncertain Environments,” Autonomous Robots, Vol. 14, No. 1, pp. 255-263, 2003. doi:10.1023/A:1022291921717
E. H. Ostergaard, M. J. Mataric and G. S. Sukhatme, “Multi-Robot Task Allocation in the Light of Uncertainty,” IEEE International Conference on Robotics and Automation, Washington DC, 11-15 May 2002.
N. Jakobi, P. Husbands and I. Harvey, “Noise and the Reality Gap: The Use of Simulation in Evolutionary Robotics,” Proceedings of 3rd European Conference on Artificial Life, Springer-Verlag, Berlin, 1995.
M. Mataric and D. Cliff, “Challenges in Evolving Controllers for Physical Robots,” Robotics and Autonomous Systems, Vol. 19, No. 1, 1996, pp. 67-83. doi:10.1016/S0921-8890(96)00034-6
L. Meeden, “Bridging the Gap between Robot Simulations and Reality with Improved Models of Sensor Noise,” In Proceedings of the Third Annual Genetic Programming Conference, 1998.
N. Y. Ko, D. J. Seo, G. J. Kim, Y. Moon and Y. Bae, “Simulation of Mobile Robot Motion Considering Uncertainties in Robot Model,” IEEE International Conference on Industrial Informatics, Daejeon, 4 December 2008.
D. J. Seo, N. Y. Ko, G. J. Kim, Y. Moon, Y. Bae and S.-W. Lim, “Performance Evaluation of Robot Motion Incorporating Uncertainties in Sensors and Motion,” Next-Generation Applied Intelligence, Springer, Berlin, 2009, pp. 271-280. doi:10.1007/978-3-642-02568-6_28
J. Go, B. Browning, and M. Veloso, “Accurate and Flexible Simulation for Dynamic, Vision-Centric Robots,” Proceedings of International Joint Conference on Autonomous Agents and Multi-Agent Systems, New York, 23 July 2004.
R. A. Brooks, “Artificial Life and Real Robots,” In Proceedings of the 1st European Conference on Artificial Life, MIT Press, Cambridge, 1992.
E. Gat, “Towards Principled Experimental Study of Autonomous Mobile Robots,” Autonomous Robots, Vol. 2, No. 3, 1995, pp. 179-189. doi:10.1007/BF00710855
P. Husbands, I. Harvey and D. Cliff, “An Evolutionary Approach to Situated Ai,” Prospects for Artificial Intelligence: Proceedings of the 9th Conference of the Society for Artificial Intelligence and the Simulation of Behaviour, IOS Press, Amsterdam, 1993.
B. Jung and G. S. Sukhatme, “Tracking Targets Using Multiple Robots: The Effect of Environment Occlusion,” Autonomous Robots, Vol. 13, No. 3, 2002, pp. 191-205. doi:10.1023/A:1020598107671
T. M. Smith, “Blurred Vision: Simulation-Reality Transfer of a Visually Guided Robot,” Proceedings of the 1st European Workshop on Evolutionary Robotics, Springer Verlag, Berlin, 1998.
A. Rosenfeld, G. A. Kaminka and S. Kraus, “Adaptive Robot Coordination Using Interference Metrics,” The 16th European Conference on Artificial Intelligence, Valencia, 23-27 August 2004.
M. Anderson and N. Papanikolopoulos, “Implicit Cooperation Strategies for Multi-Robot Search of Unknown Areas,” Journal of Intelligent and Robotic Systems, Vol. 53, No. 4, 2008, pp. 381-397. doi:10.1007/s10846-008-9242-5
B. Balaguer, S. Carpin and S. Balakirsky, “Towards Quantitative Comparisons of Robot Algorithms: Experiences with Slam in Simulation and Real World Systems,” IROS Workshop on Performance Evaluation and Benchmarking for Intelligent Robots and Systems, San Diego, 2 November 2007.
S. Balakirsky, S. Carpin, G. Dimitoglou and B. Balaguer, “From Simulation to Real Robots with Predictable Results: Methods and Examples,” Performance Evaluation and Benchmarking of Intelligent Systems, Springer, Berlin, 2009. doi:10.1007/978-1-4419-0492-8_6
B. P. Gerkey and M. J. Mataric, “Are (Explicit) Multi- Robot Coordination and Multi-Agent Coordination Really So Different?” Proceedings of the AAAI Spring Symposium on Bridging the Multi-Agent and Multi-Robotic Research Gap, Palo Alto, 22-24 March 2004.
J. Guerrero and G. Oliver, “Physical Interference Impact in Multi-Robot Task Allocation Auction Methods,” Proceedings of IEEE Workshop on Distributed Intelligent Systems, Beijing, 26 June 2006. doi:10.1109/DIS.2006.58
K. Lerman and A. Galstyan, “Mathematical Model of Foraging in a Group of Robots: Effect of Interference,” Autonomous Robots, Springer, Berlin, Vol. 13, No. 2, 2002, pp. 127-141.
A. Martinoli and F. Mondada, “Probabilistic Modelling of a Bio-Inspired Collective Experiment with Real Robots,” Proceedings of the 3rd International Symposium on Distributed Autonomous Robotic Systems, MIT Press, Cambridge, 1997.
A. Rosenfeld, G. A. Kaminka and S. Kraus, “A Study of Scalability Properties in Robotic Teams,” Springer, Berlin, 2005.
T. Lochmatter and A. Martinoli, “Understanding the Potential Impact of Multiple Robots in Odor Source Localization,” 9th Symposium on Distributed Autonomous Robotic Systems (DARS ’08), Tsukuba, 17-19 November 2008.
P. E. Rybski, S. A. Stoeter, M. Gini, D. F. Hougen and N. Papanikolopoulos, “Performance of a Distributed Robotic System Using Shared Communications Channels,” IEEE transactions on Robotics and Automation, Vol. 18, No. 5, 2002, pp. 713-727. doi:10.1109/TRA.2002.803460
K. Lerman, C. Jones, A. Galstyan and M. J. Mataric, “Analysis of Dynamic Task Allocation in Multi-Robot Systems,” The International Journal of Robotics Research, Vol. 25, No. 3, 2006, pp. 225-241. doi:10.1177/0278364906063426
B. P. Gerkey, R. T. Vaughan and A. Howard, “The Player/Stage Project: Tools for Multi-Robot and Distributed Sensor Systems,” Proceedings of International Conference on Advanced Robotics (ICAR), Coimbra, 30 June-3 July 2003.
R. Vaughan, “Massively Multi-Robot Simulation in Stage,” Swarm Intelligence, Vol. 2, No. 2, 2008, pp. 189-208. doi:10.1007/s11721-008-0014-4
R. Smith, “Open Dynamics Engine,” 2007. http://www.ode.org/.
E. Kokkevis, D. Metaxas and N. I. Badler, “User-Controlled Physicsbased Animation for Articulated Figures,” Proceedings of Computer Animation, Geneva, 3-4 June 1996.
D. Floreano, P. Husbands and S. Nolfi, “Handbook of Robotics,” Chapter 61, Springer, Berlin, 2008, pp. 1423- 1447.
N. Koeing and A. Howard, “Player Project,” 2004. http://playerstage.sourceforge.net/index.php?src=gazebo.
S. Carpin, M. Lewis, J. Wang, S. Balakirsky and C. Scrapper, “USARSim: A Robot Simulator for Research and Education,” Proceedings of the IEEE International Conference on Robotics and Automation, Roma, 10-14 April 2007.doi:10.1109/ROBOT.2007.363180
O. Michel, “Cyberbotics ltd-Webots: Professional Mobile Robot Simulation,” International Journal of Advanced Robotic Systems, Vol. 1, No. 1, 2004, pp. 39-42.
S. Dawson, B. L. Wellman and M. Anderson, “The Effect of Interaction on Robotic Sensor Network Experiments,” Proceedings of 2nd International Conference on Sensor Networks and Applications (SNA), Las Vegas, 8-10 November 2010.
S. Dawson, B. L. Wellman and M. Anderson, “Using Simulation to Predict Multi-Robot Performance on Coverage Tasks,” Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Paris, August 2010.
B. Yamauchi, “Frontier-Based Exploration Using Multiple Robots,” Proceedings of the Second International Conference on Autonomous Agents, New York, 10-13 May 1998. doi:10.1145/280765.280773