In this paper, we present a passivity-based decentralized approach for bilaterally teleoperating a group of UAVs composing the slave side of the teleoperation system. In particular, we explicitly consider the presence of time delays, both among the master and slave, and within UAVs composing the group. Our focus is on analyzing suitable (passive) strategies that allow a stable teloperation of the group despite presence of delays, while still ensuring high flexibility to the group topology (e.g., possibility to autonomously split or join during the motion). The performance and soundness of the approach is validated by means of human/hardware-in-the-loop simulations (HHIL).