Massive 'old and clunky' government database gets $60m upgrade

A massive database that pulls together information from across government agencies, Stats NZ surveys and non-government organisations, is set to receive a nearly $60 million tune-up.

Over the next three years, the Integrated Data Infrastructure (IDI) will see current systems replaced with faster and more reliable versions, with data updated more often - with tools supporting faster and more complex analysis.

Stats NZ general manager data operations and infrastructure Simon Ross told Nine to Noon the system brings in data including health, justice and income data and benefits receipts.

"A lot of the services that New Zealanders will access that you know are funded by the government, that data comes to government and it comes in from there, there are also NGOs (non-governmental organisations) who are providing services and there are Stats NZ surveys that we also bring in there.

"So we connect that together and it becomes a real rich resource for researchers to use."

But Ross said that although the IDI has been a useful resource for researchers the system is now about 15 years old.

"It is old and clunky and that's why we need to upgrade it.

"So we're going to improve the technology that supports it, we're going to make the system work faster and more reliably, we're going to allow more research to be carried out, we're going to allow more data to be brought in, refresh that data more often and we're going to make it easier for researchers to produce timely evidence."

The system can provide evidence that then supports decisions, he said.

The upgrade, which was funded in Budget 2026, is in its very early stages, he said.

Protecting confidentiality

The data does not identify individuals, names and addresses are removed, and there are only partial dates of birth, he said.

Other data protection measures include approving researchers to use the system and "their projects are scrutinised to make sure they're in the public interest", he said.

"When they access that data they do it from a safe data lab and then they're using that de-identified data for their research."

The data outputs must be made fully confidential, he said.

Even though the data is collected at the individual level, researchers have to analyse that data at the group level, he said.

"The expectation is and the policies and all the training and the kind of reference checks we do on researchers make sure that they are looking at groups in the population, not at individuals."

Asked who checked data provided by NGOs for accuracy, Ross said there was a lot of work to be done by Stats NZ and funding agencies with NGOs to ensure that "what does come in is up to scratch and able to be linked to the data that's already in there".

NGOs were able to add data about the people that they were working with and the services that they were providing to those people, he said.

"But the real value is then being able to link that data which they hold which is a small dataset, through to all of the other data about people that's in there, that's de-identified, but then you can get a much richer picture of the circumstances in people's lives than the NGO has themselves."

Who can use the database?

Ross said the types of people using the database include academic researchers working in universities, people who work in various government departments, people who work in NGOs and people who work in analytical organisation.

"Everyone who's a researcher has to meet the same standards, they have to be a bona fide researcher ... we do reference checks and we do training and we do a number of things to make sure that they are trusted and safe."

Their planned projects are also scrutinised to make sure that they are in the public interest, he said.