Yes and no. It’s big part of the difficulty, but there is other one, which is to properly manage your resources. Knowing whom to hire, what to buy and when is something which if you fail to manage properly, can ruin your game as much as a picking up a wrong fight.
Moreover unless the first crisis you get is noble war (which you can ignore), you need to make sure that when greenskins or undead invade, you are ready to face them. It seems beginning of the crisis can be postponed if you develop at the slower pace, but it will eventually kick at about day 100. If you are still running around with most of your mercs being rookies, no good sergeant and raider class gear, you will have very hard time to handle crisis.