I think the answer is balance. If you buy more brothers but put them in to the fight underequiped, they are just going to die. Decent armor is important to have less bumpy start in this game. My recommendation would be to recruit new brother only if you can give him some sort of armor, head protection and shield. Shield is absolute must if your brother is melee (unless he is pike). For head and body protection I tend to buy something with 30-40 durability. Aketon or Open Leather Cap would be my choice for head and Thick Tunic for body, but I settle for Hood/Cowl and Leather Tunic too. That’s about 150 crowns. Given early game recruits costs about 100-250, you need about 300-400 per recruit (not counting weapon and shield).
I tend to go for 8 brothers right at the start, before battle with Hogart. Then I tend to work on bringing them up to the above standard at last. Then I recruit new brothers as funds allow.
I too recommend what namespace said. Battle salvaged armor is my main way to upgrade my equipment to higher standard. To get armor as a loot after battle, it shouldn’t have being destroyed during battle. Many bandits comes with light head protection, or outright without. In such cases flail types of weapons works very well with their special attack. They hit exposed head and they leave armor intact. Knives and Daggers are the other option with their special attack which bypass armor and deals direct damage only. I tend to buy few knives right at the beginning of the game and make sure at last few brothers have them in bags. Getting farm hand recruit with flail is great too at the beginning and recruits with nets are also welcome since nets help to immobilize your target, giving your brothers time to knife victim in to the death.
Just don’t press too much with “armor hunting”, early combat is deadly and it’s not worth to put your brothers to unnecessary risk just to get that armor suit. Cost of the recruit and any equipment which might get lost (usually his armor) usually outweighs price of armor you can obtain. Using special attacks cost lot of fatigue and switching weapons cost AP, so do it only when tactically convenient or when you have gained upper hand in combat already. The safes way is to surround last enemy standing. If he is not fleeing already, bring club or two to keep him stunned and stab him with your other brothers to death. Beware however that there is some small random chance that armor will not appear in the loot anyway. It’s fairly small in my experience, but it happens from time to time.