I notice this too.
Its just caravan self-preservation instinct.
They try to avoid danger and move around it.
I have similar situation. The caravan move backward then try to move trough mountains but after moment it come back to road.
Just Pay attention to the escorted caravan.
Bonus pic of 4 my guys(3 militia men plus one hunter) fighting lonely fight against overwhelming number of bloodthirsty savege orcs.