Seems queer to argue brain differences are purely due to environment when other areas of the body are not controversial to say differ due to genes across groups
http://www.cell.com/current-biology/abstract/S0960-9822(15)00671-5
https://bmcevolbiol.biomedcentral.com/articles/10.1186/1471-2148-11-16
If anything, the brain differs more than other parts of the body, including pigmentation. And cultural/societal differences further compounded this, making us evolve rather quickly in a period of time due to differential fertility. Some societies, being smart was the best way to have the most surviving kids. Others, not so much
https://www.ncbi.nlm.nih.gov/pubmed/18087044
https://www.ncbi.nlm.nih.gov/pubmed/25748943
https://en.wikipedia.org/wiki/The_10,000_Year_Explosion
 
https://en.wikipedia.org/wiki/G_factor_(psychometrics)#cite_ref-RJ2010_31-1
 
https://www1.udel.edu/educ/gottfredson/30years/Rushton-Jensen30years.pdf
 
https://medium.com/@houstoneuler/the-cherry-picked-science-in-voxs-charles-murray-article-bd534a9c4476
 
http://laplab.ucsd.edu/articles2/Lee2010.pdf
 
John Baker, Race (London: Oxford University Press, 1974), pp. 360-400.
 
"The Persisting Racial Chasm in Scores on the SAT College Entrance Examination," Journal of Blacks in Higher Education, Autumn, 2009, p. 85.
 
Microcephalin, a Gene Regulating Brain Size, Continues to Evolve Adaptively in Humans, Science, September 9, 2005: Vol. 309 no. 5741 pp. 1717-1720, Patrick D. Evans, Bruce T. Lahn, et. al.
 
Antonio Regalado, Head Examined: Scientist's Study of Brain Genes Sparks a Backlash, Wall Street Journal, June 16, 2006.
 
Rushton, J. P. & Ankney, C. D. (1996). Brain Size and Cognitive Ability: Correlations with Age, Sex, Social Class, and Race. Psychonomic Bulletin & Review, 3, 21-26.
 
Gottfredson, L. S. (2006). Social consequences of group differences in cognitive ability (Consequencias sociais das diferencas de grupo em habilidade cognitiva). In C. E. Flores-Mendoza & R. Colom (Eds.), Introducau a psicologia das diferencas individuais (pp. 433-456). Porto Allegre, Brazil: ArtMed Publishers.
 
Wolff, Henry. "Eye Size, Brain Size, and Intelligence." American Renaissance 22.9 (2011): n. pag. Web. http://www.amren.com/ar/2011/09/index....
 
Cochran, Gregory, and Henry Harpending. The 10,000 Year Explosion: How Civilization Accelerated Human Evolution. New York: Basic, 2009. Print.