IBM develops ideas for astronomically Big Data

Prototype created in preparation for Square Kilometre Array project

IBM has completed the prototype version of a software architecture to help astronomers work with the huge data flows that will come from the Square Kilometre Array (SKA) telescope.

The SKA project will perform unprecedentedly accurate observation of radio sources in the sky, through a widespread set of dishes and aerials spread either over Australia and New Zealand or through Southern Africa, depending on who wins a competitive bid. The decision is expected early next year.

IBM’s Information Intensive Framework (IIF) enables a certain amount of classification of astronomical objects to be done automatically, with reference to a standard ontology. For routine observations, astronomers can then deal with objects as named entities, for example “galaxy” or “supermassive black hole” rather than a raw mass of data.

The project has “taken knowledge that’s inside the head of an astronomer and put it into this system; so you can do automatically some of the things that an astronomer can do manually,” says Dougal Watt, chief technology officer of IBM New Zealand, and chair of the NZ SKA Industry Consortium.

The system was developed in collaboration with radio-astronomer Melanie Johnston-Hollitt at Victoria University in Wellington. It is based on an astronomical ontology developed by the International Virtual Observatory Alliance, and uses the Ontology Web Language (OWL), maintained by the W3C consortium.

“The point is to make astronomers more efficient and productive, so they can spend time being creative,” says Watt. “It’s pretty disheartening for an astronomer to have to learn the syntax and structure of a catalogue and then have to do these very time-consuming tasks. It’d be much better to give them more time for original research.”

Watt says that because astronomers will be relieved of mundane cataloguing work, they will have more time to spot the unusual.

“We thought we could couple this up to machine-learning and social networking. We would pop up an alert on a web-portal, saying ‘I can’t classify this object and here’s a picture of it,’ so the next person that looks on the portal has a crack at it and then a whole lot of people do and eventually they agree on what it is and they can tell the system; this is a new object of whatever type. So every time the system comes across that object in future it will ‘know’ what it is.”

The framework will also be able to spot changes in objects through time. If it has previously noted an object of a certain type at a certain position in the sky and another kind of object appears there a week or a year later, it will detect that something is changing and mobilise telescopes throughout the world to turn to that location, Watt says.

Similar principles could be adapted to earthbound domains of expertise such as medicine or manufacture, raising everyone’s expertise to the level of the top experts and ensuring faster accurate diagnosis or a better quality product, he suggests.

Now the prototype has proved the concept “the next step is to sit down with some astronomers and get this going in a real live experiment; look at some of the problems astronomers are trying to address and look at what we can automate there.”

Work is also going on to improve the performance of the framework through parallel-processing techniques, Watt says.