Gibbs was born on his father’s estate, now within Astoria, Long Island, New York. He was the son of mineralogist Colonel George Gibbs and grandson of Oliver Wolcott Jr., Secretary of the Treasury in the administrations of Presidents George Washington and John Adams. His older brothers were George Gibbs and Oliver Wolcott Gibbs.

Gibbs attended school in White Plains, New York, and Dartmouth College before he was appointed to the United States Military Academy at West Point, New York. He graduated from West Point forty-second of fifty-nine in the class of 1846. Only one reference cited below was found to mention Gibbs marrying or having children. A Gibbs genealogy privately published in 1933 shows Gibbs was married and had two sons. He married Peggy Forshee Blair of Richmond, VA December 27, 1855. They had two sons: Alfred Wolcott Gibbs (1856–1922), a railway mechanical engineer, and Dr. John Blair Gibbs (1858–1898), an assistant surgeon, U.S. Navy.