What is a smoking gun? Where does the phrase come from?

smok·ing gun  

Noun
A piece of incontrovertible incriminating evidence
Used to mean indisputable evidence 'smoking gun' was first used in the Sherlock Holmes story, The Gloria Scott (1893). 
We rushed into the captain's cabin . . . there he lay with his brains smeared over the chart of the Atlantic . . . while the chaplain stood with a smoking pistol in his hand.
William Safire the identifies the first contemporary use as during the Watergate scandal in 1974. The phrase was then heavily used in the controversy regarding weapons of mass destruction in Iraq.

The Adventures of Sherlock Holmes (Kindle Edition $0.89)
The Memoirs of Sherlock Holmes