It seems to me that the friendship and enmity of settlements both wither away at the same rate (i.e. somewhat too quickly). I defied two noble houses and fought their troops and gave their shields to my frontline to wear as trophies, and all it took was a few missions for friendly settlements under their overlordship before they went back to neutral from threatening.
