Ok. Lets get some things straight here.
There are fixed locations on the map and there are quest spawned / faction spawned locations.
The fixed locations are places that are guarded by toughest enemies regardless of what level you are. I also think they may get different sizes depending on your stats later on (i cant be 100% sure but i think ive had locations that displayed different amounts and compositions of troops at different times i visited them, i believe some of them spawn parties that roam around).
The spawned locations, and in particular the quest spawned locations, will, depending on different factors, be scaled to your power. Naturally the missions that spawn them will also have different amounts of skulls, so if you have 6 men party, you wont be sent to kill 14 raiders, in a 1 skull mission at least.
So when it comes to the particular location you ran into, its not one that you are supposed to kill that early on. It will not scale down to your size, you are basically wasting time there. You should not be wandering off road until you have at the very least 12 men party that are at the very least ~ level 6. What you should be doing is roaming from town to town and picking up easy quests that net 1000-2000 gold.
For example my current level 11 party that is customized to kill Orc Warriors would smash through that location probably without anyone even being hit (archery is useless on skeletons, they have like 95% immunity to it, so the trick is to have 12 melees and attack during the night so their archers are useless; since each one my 12 melees can kill multiple skelies in a 1 v 1 its like hot knife through butter, although you have to prioritize the skelies with 2 handed weapons (and 2 tiles reach weapons), those can be nasty).
