The most important raw material for making glass is sand, and the sand for the City of Glass comes from the Yalahar Desert directly to the south of the city. However, making glass requires more than just sand, and one of the main reasons Mikaela Ironworks City became the "City of Glass" is that it had an abundance of lime mines locally.
The servants, attendants, and workers who were moved were not driven by the Grey Glove to work in the mines; instead, they were relocated to the Hensinger Mine Area on the South Shore of Green Lake. This area is the largest village north of the City of Glass, and the Grey Glove found a small farmhouse at the end of a country road far from the village. Even if Shard didn't come to investigate, those who had cleared their names would soon regain their freedom anyway.
