Sunday, October 23, 2011

Big Data in the Dirt (and the Cloud)

Big data, the term for scanning loads of information for possibly profitable patterns, is a growing sector of corporate technology. Mostly people think in terms of online behavior, like mouse clicks, LinkedIn affiliations and Amazon shopping choices. But other big databases in the real world, lying around for years, are there to exploit.

A company called the Climate Corporation was formed in 2006 by two former Google employees who wanted to make use of the vast amount of free data published by the U.S. Weather Service on heat and precipitation patterns around the country. At first they called the company Weatherbill, and used the data to sell insurance to businesses that depended heavily on the weather, from ski resorts and miniature golf courses to house painters and farmers.

It did pretty well, raising more than $50 million from the likes of Google Ventures, Khosla Ventures, and Allen & Co. The problem was, it was hard to sell insurance policies to so many little businesses, even using an online shopping model. People like having their insurance explained. The answer was to get even more data, and focus the agriculture market through the same sales force that sells federal crop insurance.

We took 60 years of crop yield data, and 14 terabytes of information on soil types, every two square miles for the United States, from the Department of Agriculture, says David Friedberg, chief executive of Climate Corp., a name Weatherbill started using Tuesday. We match that with the weather information for one million points the government scans with Doppler radar â€" this huge national infrastructure for storm warnings — and make predictions for the effect on corn, soybeans, and winter wheat.

The product, insurance against things like drought, too much rain at the planting or the harvest, or an early freeze, is sold through 10,000 agents nationwide. Climate Corp, which also added former North Dakota Senator Byron Dorgan to its board Tuesday, will likely get into insurance for specialty crops like tomatoes and grapes, which do not have federal insurance.

Like the weather information, the data on soils was free for the taking. The hard and expensive part is turning the data into a product. Mr. Friedberg was an early member of the corporate development team at Google. His co-founder, Siraj Khaliq, worked in distributed computing, which involves apportioning big data computing problems across multiple machines. He works as Climate Corp.s chief technical officer. Out of the staff of 60 in the companys San Francisco office (another 30 work out in the field) about 12 have doctorates, in areas like environmental science and applied mathematics.

They like that this is a real-world problem, not just clicks on a Web site, says Mr. Friedberg. He figures that Climate Corp. is one of the worlds largest users of MapReduce, an increasingly popular software technique for making sense of very large data systems. The number crunching is performed on Amazon.coms Amazon Web Services computers.

Climate Corp is working with data designed to judge how different crops will react with certain soils, water and heat. It might be valuable to commodities traders as well, Mr. Friedberg figures the better business is to expand in farming. Besides the other crops, he is looking at offering the service in Canada and Brazil, or anywhere else where he can get decent long-term data. Its unlikely hell get the quality he got from the federal government, for a price anywhere near free.

Related Articles

0 komentar :

Post a Comment