As I have seen there is some mechanic that determines if the enemy pack will follow you and try to attack or simply flee. It depends on your partys relative strenght compared to the enemy. The weaker you are the more likely they will want to destroy you. Now to be definitely on the safe side you need to break line of sight. In my experience even the following armies will stop after a time, tho did not have a chase for a long time.
Few things might help you, like crossing a caravan path to throw off your followers, or running to the castle or watchtowers and engaging at their location drawing in the guards at your side. You can even force 3 way battles if you draw in bandits and orcs(they will fight not only you but between themselves). Probably all factions do fight each other. If the enemy is weak you can try to use the settlements militia, or a bit stronget guards by the city.
