Well you can always do 2 attacks for less fatigue and use last 3 AP to get closer to new target or just rest
Not using all your APs in a turn won’t help. You’re removed of 15 points of fatigue every turn, no more and no less, regardless it you do something or do nothing.
I do not know how strong this build will be, so wanted to know opinion of someone who used it.
It will be very strong against particular type of the enemy, like nobles, hedge knights or even Orc warriors. Weak against everything else.
Of course you can always switch weapons. Fact that you have dagger specialization does not mean that you can’t use any other weapon. Only disadvantage of this build is that you are using up one perk point.