Magnificent Constructions: The Role of Environment on the Stellar Mass Growth of Massive Galaxies