It does mention in the perk description that if you put your 2-handed weapon away you’ll lose all stacks, and I would argue that it makes sense in both balance (exploit example: successfully cleaving all target with a 2-handed gives you 3 stacks, with Quick Hands you get to equip your shield for free and if could keep those stacks that char will be untouchable) and realism (you’re keeping the enemy at bay with your superior reach, no matter how fast you can switch weapons, they’ll close distance the moment your weapon is not in the way).
