Mathis was born in Gilmer, Texas, and grew up in San Francisco, California. He began singing publicly at school and church events shortly after his father discovered his talents. His father found him a voice teacher when he was about thirteen years old. He remains one of the very few popular singers who received years of professional voice training that included opera. He also was an athletic talent, earning four athletic letters in high school and a subsequent athletic scholarship to San Francisco State College; while there he was a basketball teammate with future Boston Celtics great Bill Russell. Mathis also remains a large part of San Francisco State College's sports history—in 1954 he broke Russell's high jump record by jumping six feet five inches (1.96 meters). At the time only four Olympic athletes had managed to clear this height. While training to become a teacher, he was talent-spotted at a concert and signed by Columbia Records. His most difficult decision was deciding whether to go to the Olympic tryouts he had been invited to or go to make his first recordings which were released in 1956. While Mathis opted for a recording career, he has never completely abandoned his enthusiasm for sports. An avid golfer who has completed a minimum of five holes-in-one, he has hosted several tournaments in his name in the USA and the United Kingdom.
Although frequently described as a romantic singer, his vast discography includes jazz, traditional pop, Brazilian and Spanish music, soul/R&B, soft rock, and Broadway/Tin Pan Alley standards. He enjoyed some early cinematic visibility when he sang one of his first hits, "It's Not for Me to Say", in the film, Lizzie, in which he also had a small acting role. Mathis also remains highly synonymous with holiday music, having recorded six Christmas albums. He has recorded over 100 albums and sold more than 100 million albums and singles worldwide. He has the distinction of having the longest stay of any recording artist on the Columbia Record label, having been with the label from 1956 to 1963 and from 1968 to the present. He is one of only a few recording artist whose career has spanned six calendar decades.
Some of his hit songs include "Chances Are, "It's Not for Me to Say," "Wonderful! Wonderful!," "The Twelfth of Never," "Wild Is the Wind," "Heavenly," "Misty," "Small World" "A Certain Smile," "Gina," "What Would My Mary Say," "On a Clear Day (You Can See Forever)," "I'm Coming Home," "When a Child Is Born," "Too Much, Too Little, Too Late" (with Deniece Williams), "The Last Time I Felt Like This" (with Jane Olivor, and "Friends in Love" (with Dionne Warwick.
Mathis has had much more success as an albums artist. His albums achieved success in part due to their reputation as an accompaniment to lovemaking. Some of his celebrated early albums include "Heavenly," "Faithfully," "Open Fire, Two Guitars," "Warm," "Swing Softly," "Johnny's Mood," and "I'll Buy You a Star."
Mathis continues to perform and record regularly and his latest album, 2005's Isn't It Romantic: the Standards Album, has been enthusiastically received by critics.
A 1982 article in Us Magazine quotes Mathis as having had a sexual relationship with a male saxophone player; however, despite a multiplicity of other, more lengthy articles, no other source documents Mathis as being other than heterosexual.