Exactly, it gets kinda boring as it takes quite long to get a brother up to your party level, so there should be some workaround, maybe even leave a brother at a training hall for a week or so and come back and he gained a level or simply get to hire higher level brothers later in the game.
