Mercury poisoning affects the nervous system, with dementia a common symptom. Daily direct contact with the metal made hat-makers particularly vulnerable to the affliction.
The phrase appears in "The Clockmaker" (1817) by Thomas Haliburton but surprisingly not directly by Lewis Caroll in either 'Alice in Wonderland' (1865) or 'Through the Looking Glass (1871).
Of course there is "A Mad Tea-Party" in Alice. This is where Hatta the Hatter and the March Hare are initially referred to as "both mad" by the Cheshire Cat. It is the zany antics of Hatta & the Cheshire Cat that are being referenced in modern usage.