"The Desolate Wall has been breached by the Otherworldly Ghosts. Black Castle, East Sea Watch, and Shadow Tower, all three Night's Watch strongholds have fallen. The army of Otherworldly Ghosts has already reached Fireplace City!"
The people of Winterfell were panicking because Fireplace City was nearly the last line of defense against the invasion of the North by the Otherworldly Ghosts.
Once Fireplace City is breached, cross the Mo River, traverse Long Lake, and head south, and you'll reach Winterfell. As for Terror Castle to the southeast, it simply can't stop the Otherworldly Ghosts from marching south.
In the short term, the only places that could remain safe might be distant islands like Skaggs Island and the Iron Islands.
But in the long run, once the mainland is taken, the surrounding islands will undoubtedly fall to the army of Otherworldly Ghosts sooner or later.
