If you take on contracts from a settlement and/or house, certain random events can give you the opportunity to betray the contract or do something else to piss them off. If this kind of thing happens more than once within a short time period, relations might become sufficiently hostile for you to have the option of attacking caravans and militias originating from that settlement (although you won’t be able to attack buildings, nor even enter the settlement at this point).
Other random events (e.g. a wildman or prizefighter/brawler from your company assaulting the locals as you exit the settlement) can result in a worsening of relations. Simply abandoning contracts without any betrayal will also worsen relations.
On other occasions, you might get into altercations with troops from a royal house – e.g. if you hire a thief from one of the settlements under that house’s patronage, a random event might trigger where you have to give the thief up or fight (or try to trick or scare the troops so that there’s no fight). If you piss off a house enough, presumably you’ll be barred from all of their settlements and have the option to attack any troops or caravans originating from them.
