Riyad as-Salihin

'Amr bin Shu'aib on the authority of his father and grandfather reported: Messenger of Allah (ﷺ) said, "It is not permissible for a man to separate between two [people] without their permission."

[Abu Dawud and At-Tirmidhi].

And in a version narrated by Abu Dawud "He should not sit between two men without their permission."