On the world map you can simply outrun them, since your speed is slightly faster then theirs. Beware however that terrain affects speed, yours and theirs both so choose right path.
On the battle map, you can hit retreat button in the upper right corner. Brothers who are standing at the very border of the map will get away unharmed. Those who don’t, will get injured. Therefore it is preferable to get all your brothers to the edge of the map first. It shouldn’t be difficult given raiders have the same APs and movement speed as you, so unless you do something stupid, they shouldn’t catch you.
Retreating from battle also cost morale, so beware. One retreat on it’s own will not affect much, but retreating while morale of your group is already very low can cause desertions.
.