Researchers show glare of energy consumption in the name of deep learning

Deep learning
Credit: CC0 Public Domain

Wait, what? Creating an AI can be way worse for the planet than a car? Think carbon footprint. That is what a group at the University of Massachusetts Amherst did. They set out to assess the energy consumption that is needed to train four large neural networks.

Their paper is currently attracting attention among tech watching sites. It's titled "Energy and Policy Considerations for Deep Learning in NLP," by Emma Strubell, Ananya Ganesh and Andrew McCallum.

This, said Karen Hao, artificial intelligence reporter for MIT Technology Review, was a life cycle assessment for training several common large AI models.

"Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data," said the researchers.

What is your guess? That training an AI model would result in a "heavy" footprint? "Somewhat heavy?" How about "terrible?" The latter was the word chosen by MIT Technology Review on July 6, Thursday, reporting on the findings.

Deep learning involves processing very large amounts of data. (The paper specifically examined the model training process for natural-language processing, the subfield of AI that focuses on teaching machines to handle human language, said Hao.) Donna Lu in New Scientist quoted Strubell, who said, "In order to learn something as complex as language, the models have to be large." What price making models obtain gains in accuracy? Roping in exceptionally large computational resources to do so is the price, causing substantial energy consumption.

Hao reported their findings, that "the process can emit more than 626,000 pounds of carbon dioxide equivalent—nearly five times the lifetime emissions of the average American car (and that includes manufacture of the car itself)."

These models are costly to train and develop—-costly in the financial sense due to the cost of hardware and electricity or cloud compute time, and costly in the environmental sense. The environmental cost is due to the carbon footprint. The paper sought to bring this issue to the attention of NLP researchers "by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP."

How they tested: To measure environmental impact, they trained four AIs for one day each, and sampled the throughout. They calculated the total power required to train each AI by multiplying this by the total training time reported by each 's developers. A was estimated based on the average carbon emissions used in power production in the US.

What did the authors recommend? They went in the direction of recommendations to reduce costs and "improve equity" in NLP research. Equity? The authors raise the issue.

"Academic researchers need equitable access to computation resources. Recent advances in available compute come at a high price not attainable to all who desire access. Most of the models studied in this paper were developed outside academia; recent improvements in state-of-the-art accuracy are possible thanks to industry access to large-scale compute."

The authors pointed out that "Limiting this style of research to industry labs hurts the NLP research community in many ways." Creativity is stifled. Good ideas are not enough if the research team lacks access to large-scale compute.

"Second, it prohibits certain types of research on the basis of access to financial resources. This even more deeply promotes the already problematic 'rich get richer' cycle of research funding, where groups that are already successful and thus well-funded tend to receive more funding due to their existing accomplishments."

The authors said, "Researchers should prioritize computationally efficient hardware and algorithms." In this vein, the authors recommended an effort by industry and academia to promote research of more computationally efficient algorithms, and hardware requiring less energy.

What's next? The research will be presented at the Annual Meeting of the Association for Computer Linguistics in Florence, Italy in July.


Explore further

Chip design dramatically reduces energy needed to compute with light

More information: Energy and Policy Considerations for Deep Learning in NLP, drive.google.com/file/d/1v3Txk … yRTTFbHl1pZq7Ab/view

© 2019 Science X Network

Citation: Researchers show glare of energy consumption in the name of deep learning (2019, June 9) retrieved 23 August 2019 from https://techxplore.com/news/2019-06-glare-energy-consumption-deep.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
217 shares

Feedback to editors

User comments

Jun 09, 2019
This is not a shortcoming of computing. This is a shortcoming of electrical generation
and fuel choices.

Jun 09, 2019
This is not a shortcoming of computing. This is a shortcoming of electrical generation
and fuel choices.


It's a shortcoming of the AI model/theory that requires the equivalent of thousands of human lifetimes to learn any simple thing that a human learns by one example. That puts them at a significant energy disadvantage compared to the real deal, and for any practical real-world applications for advanced and continuously learning AIs. For example, you can't afford a self-driving electric car to consume 20% of the battery just to run its brain. That would be unreasonable.

The human brain works at around 25 Watts of power. That's a sandwich a day. Even a small computer cluster easily gobbles up 25 kiloWatts - and that's going to be a barrier for the usefulness of AIs in general unless they become significantly more efficient and, you know - intelligent.

Jun 09, 2019
Programs consume energy because the underlying hardware consumes energy.

Period.

The hardware consumes exactly the same energy regardless of which program is running. Implying otherwise by describing the program as using the energy rather than the hardware executing it is silly.

I am not one of those that often criticizes either the content or writing of articles published in this site, but this one in particular is simply ridiculous.

Jun 10, 2019
so far so good
said the
couch potato

Jun 10, 2019
"...Creating an AI can be way worse for the planet than a car? ..."

What a STUPID comparison!
Does an AI do the job of a car? No.
Does a car do the job of an AI? No.
So neither one is an alternative for the other so it is NOT a case of "AI versus car" and thus this comparison is STUPID!

And computer technology is improving all the time and its just a question of when, not if, spintronics will replace convetional computer circuits and then the carbon footprint of AI will go massively way down. (And I am speaking here as an AI expert)

Jun 10, 2019
-and why do they pick specifically on deep learning AI rather than AI in general or even computers in general? Using their same 'logic' we shouldn't be having ANY computers! A computer generally makes a carbon footprint whether its AI or not and a large super computer will make about the same carbon footprint whether its a deal learning one or not. They make absolutely NO valid point here.

"authors recommended an effort by industry and academia to promote research of more computationally efficient algorithms, and hardware requiring less energy. "

Yes, but that is what scientists and experts are working on ANYWAY without your advice, thanks. Your advice makes no difference because we are doing that ANYWAY because of the OBVIOUS need for it.

Jun 10, 2019
The hardware consumes exactly the same energy regardless of which program is running. Implying otherwise by describing the program as using the energy rather than the hardware executing it is silly.


My computer uses different amounts of energy depending on which program is running, because the program uses different parts of the actual circuitry for different amounts of time, and when no operation is necessary it switches to a power saving mode to turn some circuits off.

The hardware doesn't do anything without the software, so it's the software using the hardware, so it's the software using the energy. Otherwise the machine would be in a suspend mode consuming nothing.

Jun 10, 2019
and why do they pick specifically on deep learning AI


Because deep learning AI as it currently is requires extensive training using very large data sets, as opposed to things like expert systems which aren't computationally taxing to "train".

Every time you need to teach your deep-learning AI a new trick, like how to drive a car around a new type of intersection, you have to simulate the reality of the situation to it and run it through the simulation multiple times, and because the models of deep learning AI are so hard at learning they need to run it through millions of times with varying examples so it doesn't latch on to some easy but stupid solution.

advice makes no difference


It puts the hype mongers down a notch to remind everyone that these things aren't absolutely perfect yet - regardless of what the "experts" say.

spintronics will replace convetional computer circuits


Yes, and we'll have flying cars by year 2000.

Jun 10, 2019
The issue is that "deep learning" as it is currently understood in AI is actually just exhausting the problem spaces by brute force - the algorithms are said to be intelligent when they produce the desired output for the right input, but how you get there is of little concern. You just throw more data and computing power at it until it does what you ask of it, and if you run out then tomorrow you'll have more to throw at it - hopefully.

But, if your "human-equivalent" board game algorithm has to run through simulations that would last 800 years non-stop if actually played at a human pace, there's obviously something missing from the picture, and it's going to be a big problem in trying to extrapolate the AI out of simple board games and into the real world where the complexity of the situation grows exponentially.

Jun 10, 2019
Eikka

Spintronics have yet to come of age but technology is improving all the time and it is OBVIOUS it is just a matter of when, not if, spintronics will replace convetional computer circuits. Why wouldn't this happen? There has already been some advances in spintronic research and its obvious there will be more and more advances in it until it finally comes of age. There is nothing in the laws of physics nor any practical consideration from which you can deduce its impossible and/or impractical; its just a matter of doing all the research. Already some working spintronics have been made in the lab thus giving proof of the principle of it. Its definitely can be and thus will eventually be done.

https://en.wikipe...ntronics


Jun 10, 2019
An impressive example of progress in spintronic research;

https://www.eurek...2019.php
"...Development of nonvolatile spintronics-based 50uW microcontroller unit operating at 200MHz ..."

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more