Actually I find this idea very interesting. It would definitely allow more customization and more distinct characters.
Currently all your brothers are distinct mainly by perks and some difference in numbers.
With basic stats and skill system some perks or attacks skill could be possible to use only if basic stats or skills have some minimum value.
Like Strong character with some decent axe skill have access to special chop attack or high dexterity character would have access to lunge skill(or whatever).
Currently all of brothers are jacks of all trade on start with no distinct likes or proficiency towards weapons. Sword masters are not better with sword than with axe or mace or pike. Or that one guy who just love axes but dislike using spears.
Sadly we are probably stuck with current system – simple and working but without flavour.
