In 1860 he moved to Colorado to invest in mining. He was one of the founders of the City of Denver, Colorado, and founded the First National Bank of Denver in 1865. Chaffee entered politics and helped organize the Colorado Territory, serving in its first legislature as speaker. He was the territorial delegate to the United States Congress starting in 1870.
In 1876, after Colorado was admitted to the Union, Chaffee was elected to the United States Senate. He served for the duration of his term, until 1879, but did not seek reelection due to poor health.
In 1884, Chaffee was elected state chairman of the Colorado Republican Party.