New community provides an open-source framework to assess large language models for capability, energy efficiency and safety
25 February 2025, London – The GSMA Foundry, the GSMA’s innovation hub, today announced the launch of GSMA Open-Telco LLM Benchmarks, an open-source community aimed at improving the performance of large language models (LLMs) for telecom-specific applications. The community provides an industry-first framework for evaluating AI models in real-world telecom use cases and is supported at launch by Hugging Face, Khalifa University, The Linux Foundation and a host of leading mobile network operators and vendors.
As AI adoption in telecoms accelerates, LLMs have demonstrated significant shortcomings in handling technical telecom knowledge, regulatory compliance and network troubleshooting. In recent tests, GPT4¹ scored less than 75% on TeleQnA² ³, a comprehensive dataset tailored to assess the knowledge of LLMs in the field of telecommunications, and less than 40% on 3GPPTdocs Classification⁴, a dataset based on 3GPP standards documentation. Microsoft's Phi2⁵, a much smaller model, scored only 10% on MATH500⁶ ⁷, a benchmark of 500 general maths questions.
These results highlight the current limitations of AI models in addressing telecom-specific queries. GSMA Open-Telco LLM Benchmarks will address these gaps by providing transparent, open evaluations of AI models across capabilities, energy efficiency and safety.
“Today’s AI models struggle with telecom-specific queries, often producing inaccurate, misleading or impractical recommendations,” said Louis Powell, Head of AI Initiatives, GSMA. “By creating an industry-wide set of benchmarks, we’re not only improving model performance but also ensuring AI in telecoms is safe, reliable and aligned with real-world operational needs.”
The mobile network operators supporting the launch of GSMA Open-Telco LLM Benchmarks include Deutsche Telekom, LG Uplus, SK Telecom and Turkcell and technology vendor, Huawei.
The GSMA Open-Telco LLM Benchmarks community enables mobile network operators, AI researchers and developers to submit use cases, datasets and models for evaluation. A standardised benchmarking framework ensures that all AI models are evaluated against real-world challenges in areas such as telecoms domain knowledge, mathematical reasoning, energy consumption and safety. The resulting benchmarks will be hosted on Hugging Face to ensure transparency and encourage community engagement.
Mobile network operators, vendors, startups and researchers are now encouraged to contribute, by submitting interest and LLM telcos use cases, to aiusecase@gsma.com and for more information visit www.gsma.com/get-involved/gsma-foundry/gsma-open-telco-llm-benchmarks.
Quotes from partners:
Hugging Face
Jeff Boudier, Head of Product and Growth, Hugging Face, said: "Hugging Face is the leading open platform for AI builders, and we're thrilled to support and host the GSMA Open-Telco LLM Benchmarks to advance telecoms AI adoption and innovation."
Khalifa University
“Academia plays a crucial role in advancing AI for telecommunications by ensuring rigorous benchmarking and scientific integrity. At Khalifa University, we are proud to support the GSMA Open-Telco LLM Benchmarks initiative. This effort will drive innovation and enhance the reliability of AI models in real-world telecom applications," said Prof. Merouane Debbah, Director, 6G Research Center, Khalifa University.
LG Uplus
Sangyeob Lee, Chief Technology Officer, LG Uplus, said: “We stand at a turning point, heading for human and AI agent coexistence, and the telcos will play a vital role in establishing safe, autonomous connections between them. LG Uplus is committed to this AI agent innovation through LLM advancement and welcomes GSMA Open-Telco LLM Benchmarks as our guiding light towards the assured intelligence services that we pursue.”
The Linux Foundation
"The launch of GSMA’s Open-Telco LLM Benchmarks marks a significant milestone in advancing AI adoption across the telecom industry," said Arpit Josphipra, General Manager, Networking, Edge and IoT, The Linux Foundation. "By establishing open, standardised benchmarks, this initiative brings much-needed transparency and performance insights, enabling operators and ecosystem partners to deploy domain-specific AI with confidence. The Linux Foundation supports this effort, as it aligns with our vision of open collaboration to drive innovation and efficiency in telecom networks worldwide."
SK Telecom
Eric Davis, Head of AI Tech Collaboration Office, SK Telecom said: “The introduction of GSMA Open-Telco LLM Benchmarks marks a pivotal milestone for the telecommunications industry in its pursuit of tangible AI benefits. By establishing a standardised evaluation framework, we're simultaneously driving innovation and ensuring AI solutions deliver the robustness, reliability and precision that our rapidly evolving sector demands.”
The launch follows last year’s industry-wide commitment to exploring telco AI use cases ethically and sustainably, central to which was the GSMA’s Responsible AI Maturity Roadmap, which helps MNOs ensure best-practice principles are applied from inception through evolution.
AI at MWC25 Barcelona
The 'Gen AI Summit: Experimentation to Transformation' at MWC25 Barcelona will feature a range of sessions aimed at exploring the practical applications and transformative potential of generative AI within the telecommunications sector. Key sessions will include discussions on AI-driven network optimisation, personalised customer experiences and the integration of generative AI in 5G and beyond. Renowned speakers such as Harry Singh, Chief Digital Officer at BT; Harrison Lung, Group Chief Strategy Officer at e&; Kaniz Mahdi, Director Technology AWS Industries at Amazon Web Services; and Laurent Leboucher, CTO at Orange; will share their insights and experiences on leveraging AI for industry advancement.
Other AI speaker highlights at MWC25 Barcelona include keynote 7, 'Tech Game Changers', where Arthur Mensch, a leading AI researcher and CEO of Mistral AI, will take the stage to discuss the latest developments and the real-world applications that are set to revolutionise the telecoms industry. In addition, keynote 10, ‘Why AI Agents will Change Everything’, will see Bret Taylor, CEO and co-founder of Sierra and Board member of OpenAI, discuss how AI agents are poised to transform businesses and enterprises.
1 https://arxiv.org/abs/2303.08774
2 https://arxiv.org/abs/2310.15051
3 https://huggingface.co/datasets/netop/TeleQnA
4 https://arxiv.org/pdf/2407.09424
5 https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
6 https://huggingface.co/datasets/di-zhang-fdu/MATH500
7 https://arxiv.org/pdf/2103.03874
Additional paper of interest, ‘TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain’, Cornell University, December 2024
About GSMA
The GSMA is a global organisation unifying the mobile ecosystem to discover, develop and deliver innovation foundational to positive business environments and societal change. Our vision is to unlock the full power of connectivity so that people, industry, and society thrive. Representing mobile operators and organisations across the mobile ecosystem and adjacent industries, the GSMA delivers for its members across three broad pillars: Connectivity for Good, Industry Services and Solutions, and Outreach. This activity includes advancing policy, tackling today’s biggest societal challenges, underpinning the technology and interoperability that make mobile work, and providing the world’s largest platform to convene the mobile ecosystem at the MWC and M360 series of events.
We invite you to find out more at gsma.com