Earlier efforts to provide space for the Senate had included the construction of the Russell Building and the Dirksen Building. By 1967 the Senate began to experience a strain on its existing office facilities and initiated the process that led to the creation of the Hart Building. In 1972 the Senate Office Building Commission authorized Architect of the Capitol George M. White to commission John Carl Warnecke & Associates to prepare studies. In addition to satisfying space and design requirements, the architects were required to preserve the neighboring 19th-century Belmont House.

In August 1974, the Senate Office Building Commission and the Senate Committee on Public Works approved a proposed nine-story extension to the Dirksen Senate Office Building (the extension would later be named the Hart Senate Office Building). The design included suites for 50 senators, with over one million square feet of interior space, including three floors of garage and service facilities, eight floors of offices, and a mechanical equipment floor at the top. A central atrium provides offices and corridors with light in an energy-efficient manner. To allow flexible office space design, Warnecke introduced a two-story "duplex suite," consisting of a Senator's office with traditional 16-foot ceilings and two staff levels that can be easily rearranged by the use of demountable partitions.

Excavation began in December 1975, and in August 1976 the building was named in honor of former Senator Philip A. Hart. The first occupant, Majority Leader Howard H. Baker, moved into the building in November 1982.

Installed in 1986 in the building's atrium was the sculpture Mountains and Clouds by Alexander Calder, creator of the mobile. The matte black aluminum clouds, the largest of which weighs one ton, are suspended from the roof, revolving above the 39-ton steel mountains.