Elijah Alexander Brown was a businessman and the son of former Georgia Governor Joseph Emerson Brown. He was born on September 4, 1857 in Canton, Georgia, to Joseph Emerson Brown and Elizabeth Grisham Brown. His siblings were Julius Lewis; Mary Virginia; Joseph Mackey; Franklin Pierce; Charles McDonald; Sally Eugenia; and George Marion. As a child, Brown moved to Milledgeville, Georgia and then to Atlanta, Georgia, where he graduated from Boys' High School. The Oglethorpe University Chapter of Chi Phi Fraternity initiated Brown in 1874 and he became a member of Chi Phi at University of Georgia. In 1879, Brown graduated as a Phi Beta Kappa from University of Georgia. He was a director in the Georgia Savings and Trust Company, in the Dade Coal Company, and treasurer of the Joseph E. Brown Company. Brown married Henrietta Law McBride in 1894 and had two children: Elijah Alexander Brown, Jr. and Frances Brown. After the death of his first wife, Brown married Harriet Elizabeth Johnson on June 26, 1901. Three children were produced from this union: Penelope H. Brown, Marcus J. Brown, and Joseph E. Brown, II. Brown was also a member of Sons of Colonial Wars, the Second Baptist Church and a road overseer in the 973 District in Gordon County, Georgia.

Brown and his father, Joseph E. Brown, ran a family farm, Brownwood, located on the Coosawattee River in Gordon County, Georgia. The farm produced corn, peas, wheat, and cotton.

Joseph E. Brown was born on April 15, 1821 in South Carolina. Although he is best known as Georgia's Civil War governor, serving in that capacity from 1857 to 1865, his career includes many other impressive positions: State Senator for the 41st District from 1849 to 1850, Georgia Presidential Elector for 1852, Superior Court Judge for the Blue Ridge Circuit from 1855 to 1857, Chief Justice of the Georgia Supreme Court from 1868 to 1870, U. S. Senator from 1880 to 1891, and president of the Western and Atlantic Railroad from 1870 to 1890. Joseph Brown died on November 30, 1894 and is buried in Oakland Cemetery in Atlanta.

On January 28, 1926, Elijah A. Brown, Sr. died suddenly of a heart attack. He was sixty-eight years old. Elijah Brown was interred in Oakland Cemetery in Atlanta. Addistional biographical information about the Brown family has not been determined.