Stochastic Parrots study questions large language models
A study by a group of researchers exploring the risks of large language models resulted in the dismissal of a number of high-profile Google employees and raised questions about Google's values, culture and governance . It also prompted heated discussion about the role of ethics in technology decision-making and its effective 'privatisation' by commercial interests. Written by linguist Emily Bender and then Google ethicists Timnit Gehru and Margaret Mitchell, the 'Stochastic Parrots' study assessed the financial, social and environmental risks of large language models such as Google's BERT and OpenAI's GPT-2 , and set out a series of recommendations for minimising these risks . Amongst other things, the paper referenced a 2019 University of Massachusetts, Amherst, study that had concluded that the energy consumption and carbon footprint of large language models had massively increased since 2017. The study also found that the training of a single model emits over 626,000 pounds of carbon dioxide equivalent, which is nearly five times the lifetime emissions of the average American car, including its manufacture. The Information reported in May 2023 that OpenAI had incurred losses of USD 540 million during 2022 as it developed GPT-4 and ChatGPT, underscoring the huge costs of training its models. System 🤖 Google BERT 🔗 GPT-3 Operator: Alphabet/Google Developer: Alphabet/Google Country: USA Sector: Technology; Multiple Purpose: Generate text Technology: Large language model (LLM); NLP/text analysis; Neural networks; Deep learning; Machine learning Issue: Bias/discrimination - race, ethnicity; Ethics/values; Employment; Environment
- Date it happened
- 2020-12-01
- Organisation involved
- Product, system or model
- BERT; GPT-3; GPT-2
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.