Born in 1916, Khan hailed from a family of musicians. His ancestors were court musicians who used to play in Naqqar Khana (an orchestra pit in Mughal-era monuments) in the princely state of Bhojpur, Bihar. His father was a well-known shehnai player who performed in the court of Maharaja Keshav Prasad Singh of Dumraon Estate, Bihar.
At the age of six, he moved to Uttar Pradesh's Varanasi, where he received his formal training under his uncle, Ali Bux 'Vilayatu', a shehnai player attached to Vishwanath Temple.
He performed at the Red Fort on India's Independence Day and also on the first Republic Day ceremony on January 26, 1950.
Since then he was part of every Independence Day ceremony and his performance was shown on national television immediately after the Prime Minister's address to the nation.
Initially, Khan turned down numerous invitations to perform outside India, owing to his fear of flying.