Griffith was born in Mount Airy, North Carolina, the only child of Geneva (née Nunn) and Carl Lee Griffith. At a very young age, Griffith had to live with relatives until his parents could afford to get a home of their own. Without a crib or a bed, he slept in drawers for a few months. In 1929, when Griffith was three years old, his father took a job working as a carpenter and was finally able to purchase a home in Mount Airy's "blue-collar" southside.