How Big Data Can Boost Weather Forecasting

Guest author Steve Hamm is a strategist, writer and videographer in IBM's corporate communications department.

Last September, when Typhoon Sanba smashed into the Korean peninsula, it packed winds so strong that they sent rocks flying through the air like missiles and caused massive power outages. “Hwangsa” storms, carrying dense clouds of yellow dust from China’s Gobi Desert that are sometimes loaded with heavy metals and carcinogens, sweep across the peninsula from West to East.

9.3 Petabytes Of Storage For The KMA

Menaced by such destructive weather phenomena, South Korea is upgrading its national weather information system with the goal of understanding weather patterns better and predicting better the location and ferocity of weather events. The upgrade being installed by the Korean Meteorological Administration increases the agency’s data storage capacity by nearly 1,000% to 9.3 petabytes, making it Korea’s most capable storage system.

The KMA project dramatically illustrates today’s big data phenomenon and its impact on weather forecasting. Thanks to the rapid spread of sensors and satellites, and to the increase in computer number-crunching speeds, it’s possible to forecast weather changes more accurately and with improved detail – potentially saving thousands of lives and safeguarding property. 

Increasing evidence of climate change worldwide is prompting governments and scientists to take action to protect people and property from its effects. But to take effective action, they need to know understand a lot more about the weather – everything from what’s going to happen tomorrow to what’s coming next year. For instance, leaders of the city of Hoboken, N.J., in the United States, which flooded badly last fall during Hurricane Sandy, are considering building a wall around Hoboken to keep the tidal Hudson River at bay. The problem is, if they don’t build high enough the wall could end up turning the city into a giant bathtub rather than keeping rising waters out.

Listen To Deep Thunder

IBM Research scientists are working to bring the most sophisticated data analytics to bear on weather forecasting. Their long-term weather analysis project, called Deep Thunder, combines data with sophisticated mathematical algorithms and computing power.

The scientists established at test bed in the New York City metropolitan area, where they set up a three-dimensional grid of thousands of blocks. That makes it possible for them to run calculations that produce very precise weather forecasts for a particular locale. Using this capability, the team was able to predict with remarkable accuracy the snowfall totals in New York City during the mammoth snow storm that blanked the northeastern United States in February – and also to predict accurately when the snowfall would start and stop.

Blame It On Rio

The IBM Research team is putting their algorithms to work on behalf of cities around the world. For instance, Rio de Janeiro, because of its climate and terrain, has recurring flooding and landslide problems in many hilly neighborhoods. The researchers used data describing the physics of the atmosphere to create a mathematical model of how storms are likely to unfold in Rio. With it, they can predict up to 40 hours ahead of time how much rain will fall in a particular location — with 90% accuracy.

In recent months, the Deep Thunder team, lead by Lloyd Treinish, has developed new techniques for ingesting many more measurements from weather sensors. The team is also extending its technology to new applications, including agriculture and wind farming.

For detailed and super-accurate weather information to have maximum impact, it has to be accessible by a large number of people. That’s why IBM has created iPad and cloud applications that deliver the power of Deep Thunder to people’s hands wherever they may be. Hopefully, by the time Rio hosts the summer Olympics in 2016, practically everybody who attends will be able to get their hands on Deep Thunder data so they know exactly what to expect when they venture out to the various game venues.

Note: This post originally appeared on IBM's Smarter Planet blog. IBM provided the storage hardware and software for the KMA project.