Razaf was born in Washington, D.C. His birth name was Andriamanantena Paul Razafinkarefo and he was the son of Henri Razafinkarefo, nephew of Queen Ranavalona III of Imerina, and Jennie (Waller) Razafinkarefo, the daughter of John L. Waller, the first African American consul to Imerina. The French invasion of Madagascar left his father dead, and forced his pregnant 15-year-old mother to escape to the United States, where he was born. He was raised in Harlem, and at the age of 16 he quit school and took a job as an elevator operator at a Tin Pan Alley office building. A year later he penned his first song text, embarking on his career as a lyricist. During this time he would spend many nights in the Greyhound bus station in Times Square and would pick up his mail at the Gaiety Theatre office building which was considered the black Tin Pan Alley.