MIT’s New Algorithm Could Transform Urban Planning, Supply Chains and the Future of ‘Small Data’
MIT researchers have developed a breakthrough algorithm that identifies the smallest dataset needed to guarantee optimal decisions in complex real-world systems.
When cities build new subway lines, they face an impossible dilemma long before construction begins: how do you identify the cheapest, safest path through hundreds of city blocks without spending years on costly field surveys? Urban planners typically assume that the only way to make an optimal choice is to collect as much data as possible — often far more than budgets, timelines, or logistics allow.
A new study from researchers at the Massachusetts Institute of Technology (MIT) challenges that assumption with a radically different idea: for many complex decisions, you don’t need more data — you just need the right data.
The team has developed a mathematical and algorithmic framework that can identify the smallest possible dataset required to guarantee an optimal decision, even in problems involving thousands of uncertainties. Their findings suggest that a city planning a subway line under Manhattan, or a utility operator optimizing an electricity grid, may be able to cut their data collection needs dramatically.
And the breakthrough doesn’t just reduce the burden of field surveys — it rewrites long-held beliefs about AI and the data economy.
“Data are one of the most important aspects of the AI economy. Models are trained on more and more data, consuming enormous computational resources. But most real-world problems have structure that can be exploited. We’ve shown that with careful selection, you can guarantee optimal solutions with a small dataset,”Asu Ozdaglar, head of MIT’s EECS department, in a statement issued as part of this research.
Rethinking the ‘Big Data’ Era
For over a decade, modern AI has pushed the narrative that “more data is always better.” But many real-world optimization problems — from supply chains and transit networks to energy markets — have predictable structural patterns. The MIT researchers argue that this structure can be used to determine exactly which data points matter.
This is crucial in systems like:subway route selection, supply chain diversification, electricity network optimization, construction planning, logistics and resource allocation.
In all these cases, practitioners often drown in unnecessary data collection, hoping volume will compensate for uncertainty.
The new algorithm takes the opposite approach: start with no data, and add only what is provably essential.
The Key Breakthrough
The team’s method begins by mathematically defining what it means for a dataset to be “sufficient.” They break the decision space into “optimality regions” — scenarios in which a particular route, price, or configuration becomes the best choice.
A dataset is sufficient if it can accurately determine which region the real world belongs to. “When we say a dataset is sufficient, we mean that it contains exactly the information needed to solve the problem. You don’t need to estimate all parameters accurately; you just need data that can discriminate between competing optimal solutions,” said Amine Bennouna, co-lead author.
Once the structure is defined, the algorithm repeatedly asks a critical question: “Is there any scenario in which the optimal decision could change, and my current data would fail to detect it?”
If the answer is yes, the algorithm identifies precisely which new measurement would fill that gap. If no, the dataset is complete — and provably sufficient.
This approach can shrink data collection from thousands of measurements to a handful.
“The algorithm guarantees that, for whatever scenario could occur within your uncertainty, you’ll identify the best decision,” Omar Bennouna, co-lead author, added.
From Manhattan Tunnels to Global Supply Chains
Consider the example of a subway planned beneath New York City. Every city block might hide different soil conditions, underground utilities, or hazard profiles. The traditional assumption: investigate everything.
The MIT model: investigate only the blocks that can change the optimal route, and ignore the rest.
The same applies to: selecting shipping routes in a congested supply chain, configuring electrical grid nodes during volatile energy prices, deciding where to place sensors in large infrastructure projects. This could save millions in surveys, simulations, and engineering assessments.
The researchers were able to show not only that minimal datasets exist — but that their algorithm can find them systematically, preserving optimality with mathematical certainty.
“We challenge this misconception that small data means approximate solutions. These are exact sufficiency results with mathematical proofs,” Saurabh Amin, co-senior author, said.
Why This Matters: Cost, Carbon, and Computational Savings
If AI models can be trained with fewer, smarter data points: computation becomes cheaper, energy consumption drops, model training becomes faster and policymakers can rely on faster decision cycles.
For large infrastructure projects, this could mean shaving months — even years — off planning timelines.
The team plans to expand the framework to more complex scenarios, including situations where data are noisy or partially observable — a common challenge in the real world.
Their work will be presented at the Conference on Neural Information Processing Systems (NeurIPS) — a major stage for breakthroughs that redefine how AI systems learn and make decisions.
If successful, this research could push the global AI ecosystem to rethink its data obsession. Instead of hoarding massive datasets, the future may lie in asking sharper questions — and collecting only the data that truly matters.
EP Staff is the editorial team at EdPublica, an independent media organisation focused on science, education, environment and public policy. The team produces evidence-based news, features, explainers and analysis on issues that shape society and everyday life.
OpenAI Promises Greater Transparency as Data Centre Backlash Grows in the US
OpenAI’s new National Community Compact promises greater transparency in data centre development, including public disclosure of energy and water use and safeguards against higher electricity costs.
OpenAI has announced new transparency commitments for its AI infrastructure projects amid growing scrutiny of data centre expansion. Representational image. Image credit: Solen Feyissa/ Pexels
The rapid expansion of artificial intelligence is increasing demand for the data centres that power it. Across the United States, however, plans to build these energy-intensive facilities are encountering resistance from communities concerned about electricity bills, water consumption, noise and environmental impacts. Against this backdrop, OpenAI announced a National Community Compact outlining commitments to improve transparency and strengthen community participation in the development of its computing infrastructure.
The company has also released a Compute Campus Policy Blueprint, proposing measures for governments and regulators to consider when overseeing data centre development. The announcement comes amid growing public and political scrutiny of the technology industry’s infrastructure expansion.
Ending Secrecy Around Data Centre Negotiations
One of the most significant commitments concerns non-disclosure agreements (NDAs). OpenAI says it will stop requiring public officials to sign NDAs during negotiations over future facilities, except where legitimate security concerns, trade secrets or commercially sensitive information require protection.
Such agreements have attracted criticism because they can restrict the information elected representatives share with residents about proposed developments. Greater disclosure could allow communities to understand the scale and potential consequences of projects before construction begins.
OpenAI has also committed to engaging residents early in the planning process, appointing a local point of contact for each facility and publishing annual reports detailing its progress against the compact’s commitments. The agreement applies to future computing facilities owned and operated by OpenAI, rather than every data centre used by the company.
Who Pays for the Electricity?
Electricity demand is among the most contentious aspects of data centre expansion. These facilities require continuous power to operate servers and cooling systems, potentially placing additional pressure on electricity grids and increasing costs for other consumers.
Electrical control panels inside an industrial power facility, illustrating the electrical infrastructure required to support energy-intensive AI data centres and their growing demand on power grids. Representational image. Image credit: Brett Sayles/ Pexels
OpenAI has pledged to pay the full cost of the electricity its facilities consume, along with the infrastructure upgrades required to supply them. It also proposes financial safeguards to prevent households and businesses from bearing costs if a planned project is delayed, scaled down or cancelled.
The company has proposed reducing electricity consumption during periods of peak demand and making power available to the grid when necessary. Its planned facility in Effingham County, Georgia, illustrates this approach. Under an agreement with Georgia Power, the utility can reduce electricity supplied to the facility by up to 1,000 megawatts during periods of peak demand.
Water Consumption and Environmental Disclosure
Cooling systems are another environmental concern. Data centres generate considerable heat, and some cooling technologies require substantial quantities of water to maintain operating temperatures. OpenAI says it will prioritise closed-loop and low-water cooling systems and develop water-management plans for individual facilities. It has also committed to publishing site-level information on electricity and water consumption.
The company plans to disclose construction details and commission independent noise studies before facilities begin operations. Annual reporting on relevant environmental impacts is also part of the compact. These disclosures could help communities assess the pressures associated with hosting large computing facilities. Their usefulness, however, will depend on the completeness of the information and the scope for independent scrutiny.
Growing resistance to data centre expansion
The announcement comes as opposition to data centre development gains momentum in the United States. Residents have questioned whether the economic benefits of hosting these facilities justify their demands on local electricity, water and infrastructure. According to Data Center Watch, at least 45 data centre projects worth 68 billion dollars faced disruption from local opposition during the second quarter of 2026.
OpenAI has also promised to publish local hiring goals and provide information about employment opportunities. Its commitments include helping local suppliers compete for contracts and encouraging partnerships with community colleges and trade organisations to prepare workers for construction and technical jobs.
However, voluntary corporate commitments do not carry the same force as legally enforceable regulations. OpenAI’s policy blueprint therefore calls on governments to establish stronger oversight of electricity costs, environmental disclosures and community benefits.
As AI computing demand continues to grow, the pressure on electricity networks, water resources and local infrastructure is likely to intensify. Whether OpenAI’s compact delivers meaningful accountability will depend on how consistently its commitments are implemented and independently scrutinised.
How an Indian-made Silicon Chip Could Help Terahertz Signals Overcome Obstacles in Future 6G Networks
An Indian research team at JNCASR has demonstrated a silicon photonic chip that allows terahertz signals to cross engineered barriers with minimal loss, a development that could advance high-speed circuits for future 6G networks.
6G technology is expected to use higher-frequency terahertz waves to support faster data transmission and greater network capacity. Representational image. Image credit: geralt/ Pexels
A sharp bend, a junction or a sudden change in a circuit’s structure can weaken an electromagnetic signal. When communication systems operate at increasingly high frequencies, even these seemingly small disruptions become significant engineering obstacles. For researchers working towards sixth-generation (6G) wireless technology, finding ways to guide signals through compact circuits without losing their strength is a growing challenge.
A research team in India has demonstrated an approach that could help address this problem. Using a specially engineered silicon chip, scientists at the Jawaharlal Nehru Centre for Advanced Scientific Research (JNCASR), Bengaluru, have shown that terahertz waves can cross artificial barriers with very little signal loss, even while carrying high-speed data.
The work draws on a phenomenon called photonic Klein tunnelling, which describes how waves can pass through certain barriers without undergoing the substantial reflection or attenuation commonly associated with such obstacles.
Artistic illustration of photonic Klein tunnelling in a silicon valley photonic crystal chip. A terahertz signal carrying high-speed data crosses the engineered barrier (blue) without significant loss, supporting 100 Gbit/s data transmission for 6G communication. Source: PIB
In one of the three devices developed by the researchers, data transmission reached 100 gigabits per second (Gbps). The findings, published in Advanced Materials, offer a potential design approach for the compact terahertz circuits that future communication technologies may require.
Borrowing a Principle From Quantum Physics
The idea behind the research has its origins in a phenomenon associated with quantum mechanics. In quantum physics, Klein tunnelling describes the unusual ability of certain particles to pass through a potential barrier even when classical physics would predict that they should be reflected. Researchers have explored analogous behaviour in photonic systems, where electromagnetic waves can exhibit similar transmission characteristics.
The JNCASR team applied this principle through topological photonics, a field concerned with controlling the movement of electromagnetic waves using specially designed structures. They created a silicon photonic crystal resembling the honeycomb lattice of graphene, with triangular holes arranged in a precise pattern across a silicon slab. This arrangement produces boundary channels along which terahertz waves can propagate. These channels are resistant to certain forms of backscattering, helping signals retain their direction of travel.
The researchers then introduced artificial obstacles into these pathways by changing the dimensions of the triangular holes over a short section of the crystal. The experiment tested whether the waves could retain their transmission characteristics even when their pathways were deliberately disrupted.
When a Barrier Barely Interrupts the Signal
The results showed that the engineered structures could transmit terahertz waves across the barriers with remarkably little loss, provided the strength of the barrier remained below a characteristic coupling threshold of the crystal. A single barrier produced only about 1–2% signal loss. Even with two barriers placed consecutively, the loss remained around 2–3%. For comparison, a similar barrier in a conventional, non-topological waveguide caused a considerably greater reduction in transmission.
This is significant for circuit design because communication signals must often navigate multiple pathways and changes in geometry within a limited physical space. Reducing the losses associated with these obstacles could give researchers greater flexibility when designing compact terahertz components.
The team demonstrated the principle in three silicon devices. One incorporated two barriers in sequence. Another used an omega-shaped pathway containing four sharp bends and a central barrier. The third was a power splitter, which divided an incoming signal into two branches, with one branch containing an engineered barrier.
The devices were fabricated using high-resistivity silicon, a material compatible with established semiconductor manufacturing processes. Measurements across frequencies of 310–360 GHz closely matched the researchers’ computer simulations. In the power splitter, the two output signals remained nearly identical, indicating that the barrier had little effect on signal transmission.
What does 100 Gbps Mean in Practice?
The researchers also examined whether the structures could carry high-speed data, rather than simply transmit terahertz waves. The double-barrier device achieved a data rate of 48 Gbps, while the omega-shaped configuration supported 60 Gbps. The power splitter recorded the highest rate, reaching 100 Gbps using QAM-32 modulation. This technique encodes information through variations in the amplitude and phase of a signal.
At 100 Gbps, a five-gigabyte high-definition film could theoretically be transferred in less than half a second. The different transmission rates are important because they show how the approach performed across three distinct circuit configurations. The 100 Gbps result was achieved by the power splitter, rather than all three devices. The demonstration suggests that engineered barriers need not substantially compromise data transmission, even in circuits incorporating multiple pathways and sharp bends.
From a Laboratory Demonstration to Future 6G Hardware
Terahertz frequencies, which lie between microwaves and infrared light in the electromagnetic spectrum, are attracting interest for their capacity to support high-bandwidth communication. Their potential lies in accommodating large volumes of data at speeds that conventional wireless technologies may struggle to sustain. However, generating and transmitting terahertz signals is only part of the challenge. Researchers must also develop components capable of directing, dividing and processing these signals within compact devices.
The JNCASR study addresses this less visible aspect of future communication infrastructure. Its findings could help in developing integrated terahertz circuits that route and split signals on a single chip without substantial transmission losses. The next step for the research team is to explore whether the engineered barriers can be adjusted electrically, thermally or mechanically. This could eventually allow a device to redirect signals between different pathways on demand, potentially enabling more adaptable terahertz circuits.
The work remains an experimental demonstration of signal transmission at the chip level; it does not establish a commercially deployable 6G system. Further research will be needed to integrate these components into larger communication architectures.
Led by Abhishek Kumar, Assistant Professor at JNCASR and head of its Meta Optics Laboratory (MetoLab), the study involved joint first authors Samina Mulla and Sangram Kumar Sadhu. Researchers from JNCASR, IISER Kolkata, IIT Bombay, the Institute of Microelectronics at A*STAR, Singapore, and the University of Lille and CNRS, France, also contributed. The research offers a new way of thinking about how high-frequency signals can be guided through increasingly intricate circuits. As work on 6G progresses, such advances in the underlying hardware will help determine how much of the promised speed and bandwidth can eventually be translated into practical communication systems.
India Unveils Three Homegrown AI Models, Covering 22 Indian Languages
India has released three indigenous AI models under the IndiaAI Mission. EKA-1 supports all 22 scheduled Indian languages, while Dhīr 2B and Dhīr 5B offer multilingual AI assistance with offline deployment capabilities.
India's indigenous AI development gains momentum with the release of three foundation models under the IndiaAI Mission. Representational image. Image credit: Markus Spiske/ Pexels
India has released three indigenous artificial intelligence models under the IndiaAI Mission, with applications ranging from multilingual reasoning and coding to offline AI assistance in sensitive sectors such as defence, government and education. The models — EKA-1, developed by Soket Labs Technology and Research Private Limited, and Dhīr 2B and Dhīr 5B, developed by ZenteiQ AiTech Innovations, were unveiled in New Delhi under the IndiaAI Innovation Centre (Foundation Models) pillar.
The three models have different technical capabilities and deployment priorities. EKA-1 is designed as a large multilingual foundation model supporting all 22 scheduled Indian languages and English. The Dhīr models, meanwhile, focus on lightweight, locally deployable AI assistance, allowing organisations to process information without depending on continuous cloud connectivity.
Officials from the Ministry of Electronics and Information Technology and representatives of Soket Labs and ZenteiQ at the launch of indigenous AI models EKA-1, Dhīr 2B and Dhīr 5B under the IndiaAI Mission in New Delhi.
EKA-1 Trained on 7.5 Trillion Tokens
Developed by Soket Labs, EKA-1 is a 26.3-billion-parameter foundation model trained on approximately 7.5 trillion tokens. It supports all 22 scheduled Indian languages alongside English and is designed for mathematical tasks, coding, reasoning and agentic applications.
The model uses an open, sparse hybrid architecture. Its intended applications include banking, financial services and cybersecurity, where understanding India’s digital ecosystem and sector-specific requirements is important. According to Abhishek Upperwal, CEO of Soket Labs, the model was developed with support from the IndiaAI Mission to address requirements within India’s banking, financial services and cybersecurity sectors.
Dhīr Models Target Offline and Secure AI Applications
ZenteiQ has introduced two open-source models, Dhīr 2B and Dhīr 5B, supporting English and 10 Indian languages. Their primary focus is secure deployment within organisations. The models can operate on-premise, enabling institutions to use AI systems within their own computing infrastructure rather than relying entirely on external cloud services.
They support document-grounded automation, knowledge retrieval and citizen-facing or enterprise services. The press release identifies defence, government, industrial and educational institutions as potential application areas, particularly in environments with limited internet connectivity or sensitive information.
Their lightweight architecture also allows deployment on mobile devices, according to ZenteiQ. The company says the models can run across different computing platforms, including graphics processing units (GPUs) and tensor processing units (TPUs). Prof. Sashikumar, CEO of ZenteiQ AiTech Innovations, said local deployment would allow users to retain data on their devices, improving privacy and security while reducing dependence on cloud connectivity.
Public Computing Infrastructure Supports Model Development
Both development teams were selected under the IndiaAI Innovation Centre pillar, which aims to build indigenous foundation models and large language models. The IndiaAI Mission provided access to subsidised national AI computing infrastructure and support for associated research expenses. These resources helped the selected teams develop their models.
Speaking at the launch, S. Krishnan, Secretary of the Ministry of Electronics and Information Technology, said indigenous AI models could strengthen India’s technological and strategic autonomy while creating opportunities to improve productivity across government, industry, research and education. He also pointed to AIKosh, the IndiaAI datasets platform, as a channel through which researchers, students, innovators and the wider public could access these resources and experiment with AI applications.
India’s Wider AI Development Programme
Approved by the Union Cabinet on March 7, 2024, the IndiaAI Mission operates through seven pillars: computing capacity, the Innovation Centre, the AIKosh datasets platform, application development, future skills, startup financing, and safe and trusted AI. The development of EKA-1 and the Dhīr models falls under the Innovation Centre pillar, which supports the creation of foundation models tailored to India’s requirements.
The models represent two different approaches to expanding domestic AI capabilities: large-scale multilingual processing and lightweight AI systems designed for secure local deployment. Their practical impact will depend on how effectively they perform across real-world applications and how widely organisations adopt them.