I think it is a matter of time AND distance.
Tried out just camping next to brigand fort that I attacked (killed about half of them and then retreated). A week went by but nothing changed, they were still down to the survivors of the attack.
Reloaded to just after the attack, took my company a good ways away (about half a day travel-time on steppe-terain), waited another two days, went back and the fort had a new garrison.
I also remember one time I was on my way back from some mission, completely packed full of loot and passed by a ruin that had a bunch of wiedergängers.
I went on to the nearest town, sold what I wanted to sell and went back to the ruin but now it was populated by ancient legonaries and some honor guards.
The trip back there had not taken more than a day. Was kinda weird but I guess the game just rolls a new garrison for a location everytime you enter a certain radius around it.
