Three things:
1. Armor damage is dealt first, and THEN the current armor’s 10% is taken into consideration.
2. The direct damage does NOT take away from armor damage.
3. Crossbow perk further increases the ignore armor efficacy by a 15% to a total of 65% ignores armor.
With your simplified 60 damage, it would look like this instead on first shot:
War bow deals 39 (60*0.65) damage to armor, then 0 damage to HP. 60*0.35-(250-39)/10
Heavy crossbow deals 45 (60*0.75) damage to armor, then 9.5 damage to HP. 60*0.5-(250-45)/10
BTW, are you the same GlyphGryph from Cataclysm? Just wondering :P