At fourteen he worked as an usher at the NYC Paramount Theatre. His father was an electrician who played guitar and his mother taught piano. Damone attended PS 163 and sang in St. Finbar's choir and later attended the Alexander Hamilton Vocational High School and then Lafayette High School in Brooklyn. He left school at sixteen to support his family, but returned to graduate from Lafayette in 1997. Damone won first prize in an Arthur Godfrey talent scouts contest in 1945. His first night club appearance at the LA Martinique Club was set up by comedian Milton Berle. He was drafted and served in the army from 1951 to 1953. After he was discharged from the army he married actress Pier Angeli, whom he later divorced. Damone was later married to Becky Ann Jones from 1974 to 1982 and Diahann Carroll from 1987 to 1996. He married Rena Rowan, fashion designer and co-founder of Jones New York, in 1998. In 1999, he received a certificate of advanced study from Philadelphia University.
