You do get unique weapons and armours in towns with the relevant industry from time to time, but the armours are rarer than the weapons.
You will be led by tavern rumours on odd occasions, but the tavern will probably need to be in a town that’s near to a large area of wilderness.
Sometimes you get an event where an old man tries to sell you a map. Once, I bought it and it turned out to be useless. Another time, I had a thief in my party and I was given the option to get the thief to steal the map, and we got some loot including a rare item.
Basically, the further into the wild you go, the more likely you’ll get something rare, but you do occasionally get rare things from the most innocuous contexts.