Big data – big hype or the next big thing?

‘Big data’ is the latest buzzword in the industry but what does it mean, how can it offer a competitive advantage and what’s happening locally?

The upsurge of networked devices and applications means that more data is being collected than ever before. This has lead to an explosion of large-scale data sets which is changing business and science around the world.

Organisations may not know exactly what to use the information for - but they are keen to hold on to it, and this is creating a huge demand for storage.

Christchurch-based ARC Innovations provides metering and field services for electricity retailers and data services to electricity distribution companies. In addition, it is looking at providing services to end-consumers.

General manager of technology Michael Peterson says the company, which has been in business since 2005, is now experiencing “exploding volumes of data”.

The industry has changed radically since the days when someone would physically go out and read meters, perhaps once every two months, collect one data value and return that to the energy retailer.

“Today, our meters are recording a read every 30 minutes. And not just kilowatt-hours – they are import/export capable as well, meaning that they can read the electricity you are pumping back into the grid [as well as the electricity] you are using.”

The meters also return information about power quality, harmonics on the electricity lines, and voltage sags and swells, Peterson says. And they are logging event data, such as power outages, and sending notifications to the back-office system.

“So it’s gone from this one-value read every two months to dozens and dozens of different data elements and information that is collected during the day and returned to you.”

In addition, the smart meters have two-way communications, he says. ARC has deployed radio frequency mesh network infrastructure, running its own communications network. “We are effectively like a telco,” he explains.

However, the smart meters bring back so much more information than a mobile phone would to a telco, he says.

“The more meters there are out there and the longer they have been there, the more data you are collecting,” he says.

“Previously, we didn’t have the ability to collect this data. Smart metering and smart grid technologies are revolutionising the industry.”

“Even if you are doing just simple things, like queries on the data that we are getting back from the meters [every half an hour], just using our standard Oracle relational database won’t cut the mustard any more. Queries will just time out,” he says.

The company has started looking at other ways to deal with large amounts of data, and Peterson says ‘big data’ has the potential to “transform the way that the industry operates.”

To be able to manage this explosion of data, and, even more importantly, extract valuable insights from it, ARC is working with Oracle on improving its existing data warehousing technology. But the company is also looking at IBM and SAP technology, he says.

Peterson and his team have also investigated Hadoop. “But I don’t think we are at that stage yet. That sort of technology works where the data is too big for a database and we are not at that level yet. We may never get there as the relational database vendors are looking at ways to make their solutions able to cope with the data influx.”

About a year ago the company completed its first in-house proof-of-concept project.

“We hand-built everything based on the Oracle toolset we had,” says Peterson. “So a lot of the solution is built in Java. We actually developed all the data extraction and translation capabilities and created our own analytics engine.”

In hindsight, he says it would have been better to partner with a vendor and leverage the capabilities out of the box.

“Our solution proved to be quite heavy on maintenance because everything was hand-done. It was an interesting proof-of-concept but we’d rather initially align ourselves with a vendor in this space.”

“Companies like Oracle, SAP and IBM are spending billions on developing [big data] capabilities and it just makes more sense to me to align with their efforts and look at customising if required.”

So what benefits is Peterson hoping the big data journey will give?

He says one of the company’s biggest challenges is being able to correlate information.

Another one is geospatial analysis. A big data solution would make it easier to understand complex data that is coming in, and the relationships between that data. ARC is looking at developing new services that could be sold back to the industry.

“If we invest in data analytics capabilities, we could profile [an electricity] retailer’s consumer base; look at the actual load usage, and we would be able to potentially sell this information directly to consumers or via the retailer,” he says.

ARC would also be able to match consumption profiles to all the available tariffs in the market and inform consumers which retailer they should go to and what tariff they should be on, based on their actual consumption pattern, he says.

“That is easy enough to do when you are looking at one household at a time, but what we are talking about here is how do you do that over a period of time, taking in seasonal variations and looking at hundreds of thousands of households at once. How do you model that? How do you support those types of queries?”

He sums up the biggest challenge as: “Coming to grips with all the external pieces of data”.

In addition to ARC’s own data, energy distributors have data coming in from probes. There is also hourly weather data and rainfall data. Understanding rainfall is very important, he explains, as there are huge amounts of power consumed in New Zealand by irrigators in rural areas. As ARC is based in Christchurch, it is affected by the earthquake recovery process. The company shares information with Canterbury Earthquake Recovery Authority (CERA) around electricity disconnections and redirections they are doing.

Collecting, summarising and analysing all that information in an efficient way requires a good underlying toolset, but also new skills are required within the company, Peterson says.

ARC is looking at creating completely new roles, such as a data architect, to be able to extract value out of all the unstructured data coming in. Database understanding also has to be improved.

Storing all this data is also a challenge. “We need to annually allocate budget for our storage solutions. We’ve also gone into virtualisation,” he says.