Castle Williams was turned into a formal military prison, and served as such until 1966. During WWI, Walt Disney was held there for going AWOL. During WWII, the boxer Rocky Graziano was also held for going AWOL. Later, the FBI used the prison to hold Mafia and other criminal informants.

It was designed and built by Lieutenant Colonel Jonathan Williams.