According to Carmen, he first wrote the solo part of the song, writing four bars at a time, eventually completed the interlude after two months.[7] He needed to put this into a song, and after listening to Rachmaninoff's 2nd piano concerto, a piece famously used to underscore the 1945 British film Brief Encounter, he adapted the melody of its second movement to write the verse.[7] Rachmaninoff's music was in the public domain in the United States at that time, so Carmen thought no copyright existed on it, but it was still protected outside the U.S. subsequent to the release of the album. He was later contacted by the Rachmaninoff estate and informed that it was protected.[8] An agreement was reached in which the estate would receive 12 percent of the royalties from "All by Myself" as well as from "Never Gonna Fall in Love Again", which was based on the third movement from Rachmaninoff's Symphony No. 2.[9][10]
Carmen stated that he also incorporated part of another melody into this song. The melody of the chorus was taken from his previous hit with the Raspberries, "Let's Pretend".[7]