Or bring kite shields with you when you face them. Bringing some bows with you, ideally along with brothers with bow specialty helps too.
If you find bandit marksmen OP, I wonder what’s your opinion about mercenary or faction crossbowmen who come with actual armor and heavy crossbows.
Bandit marksmen have a good skill and perks, but they lack armor. They also come with the mixture of lv.1 and 2 bows and crossbows. Moreover half the bandits don’t use shields. That is major weakness which can be exploited. Withdraw your troops so that they are either out of the range of enemy missiles or form protective shield wall. Pelt his unshielded troops with missiles while your ranged units are out of the range of his. If you use crossbows, you can use tactics where you step forward to fire, step back and reload next turn, firing every second turn. At some point enemy will either charge you or bring his ranged unit forward, exposing them to your ranged. When they charge, draw them out, preferably somewhere where you can use terrain advantage for your ranged. Then counter charge. At that point they are usually stretched out and can be dealt piecemeal. Send one or two brothers with shields around the flanks to threaten his ranged unit if he still have some and situation permits.
Also utilize bushes for hiding your ranged inside. If they can’t see you, they can’t hit you. Another thing to look for is terrain protection given by tings like trees or rocks. If you place your ranged next to it, it will provide cower protection from certain angles while allowing you to shoot in there. It does come with blind angle but it’s protective radius is much wider -basically full 180 degrees.
Another thing is that AI is reluctant to shoot at targets which are engaged and obscured by AI own troops. So once you charge, make sure you close distance fast.
You shouldn’t underestimate ranged defense, both stat wise and perk wise. This is easy to omit because many factions in game does not utilize ranged. Then when you come against force which use lot of ranged, like goblins, you are for a rude awakening.
And the last tip related to enemy crossbows. Crossbows and other armor penetrating weapons are reason why you shouldn’t underestimate health stat thinking thick armor will do. Crossbow have damage of 40-60, of which 20-30 ignores armor. This means that it can basically two shot your brother if you did not invest in his health stat. If it hits head, it can one shot brother with health under 45. Heavy crossbow is even more dangerous.
Also remember, everything which you think is OP about bandit marksmen is true for your own ranged units too, if you bothered to develop them decently. Even more so since you can use better equipment. So it’s fair.