The Seven Kingdoms were surrounded by impenetrable barriers on all sides.
First, they were located on a large peninsula. All the oceans were ringed with perpetual storms just 3 miles off the coastline, all the way around the peninsula. Even near the coast, the waves were huge. This limited the utility of even local shipping; most trade happened via the highways and rivers. There was no way around the storms, they never let up, and no one was ever known to survive passing through them in either direction.
The land border of the kingdoms was a massive range of mountains called the Venn. This was a rugged, inhospitable forest at low altitudes and a rugged, inhospitable alpine environment higher up. Both parts were filled with giant beasts of all kinds. The deeper parts of the Venn, called the High Venn, were known to be the homes of great beasts and even various monsters.
There were no known passes through the Venn, and no trade. Explorers of the High Venn only returned if they turned back before reaching the second range of peaks. None ever came through from the other side.
So the Seven Kingdoms simply established the Warders to keep watch on this border and keep the population of giant monsters under control then ignored the Venn.