Police use of AI restarted after officers used unapproved models

OpenAI - which created both ChatGPT and Whisper - told RNZ: "Addressing hallucinations is an ongoing area of research."

Photo: Jonathan Raa / NurPhoto / NurPhoto via AFP

Police officers trialling generative AI significantly misused it and the project was dumped, only to be restarted six months later after officers used unapproved AI models.

The police high-tech crime group has now been using the approved Whisper AI in investigations for nine months after the project was restarted with extra controls.

The trial and use of the AI systems have been revealed in Official Information Act documents released to RNZ.

Officers tested Whisper on translating and transcribing non-English language audio for six months, up to March last year.

The documents say Whisper is 45 percent inaccurate on Māori and Pacific languages and can not be used on those.

But officers broke that ban on using it to transcribe English during the trial, so police dumped Whisper in March last year.

"Trial misuse demonstrated limitations of relying on behavioural controls alone," said a document.

Police restarted it in September, and told RNZ that only trained approved staff can use it and its output is not used in evidence.

Hallucinations

The use of AI for law enforcement investigations in this manner is controversial around the world, with critics saying it is too unreliable with such high stakes. But proponents - including companies selling AI - say it saves officers huge numbers of hours in paperwork.

NZ Police had a ban on generative AI (GAI) until mid-2024 and said it would be very careful about introducing any systems.

A Cornell University study in May 2024 found Whisper AI hallucinated one percent of the time, including making up racial commentary, violent rhetoric and imagined medical treatments.

OpenAI - which created both ChatGPT and Whisper - told RNZ: "Addressing hallucinations is an ongoing area of research."

Whisper was an open-source research model released in 2022.

"As we've consistently stated publicly, speech recognition systems are not perfect and should be evaluated carefully for their intended use case," said a company spokesperson on Tuesday.

It kept improving models and had newer ones, like GPT-Realtime-Whisper, that it said reduced hallucinations and were more accurate across various languages.

The trial began six months before police introduced their first policy on generative AI in March 2025. It began three months after police said they had no plans to lift a ban on using GAI.

A spokesperson for the Muslim community said they trusted police and supported use of technology, but Whisper obviously was used with ethnic groups.

"We should have been consulted," said Abdur Razzaq of the Federation of Islamic Associations.

'Trial terminated'

The trial concluded that the AI "was efficient and valuable for intel".

But the OIA documents included a minute from the police headquarter's tech watchdog group which also said: "Significant misuse noted (English audio processed despite restrictions).

"Continued use not endorsed; trial terminated."

That same month, a March 2025 public presentation by police about GAI said the consequences of misuse or errors were high for the public.

They have also had internal warnings that the chance of irreversible damage from using AI - including to the reputation of police - required high due diligence from the start.

"This is even more crucial given there is no specific legal framework for the use of AI in New Zealand," said a warning.

'Unapproved alternatives'

Whisper came up again at the tech watchdog group in September 2025.

"Memo presented seeking reconsideration of Whisper AI following earlier termination," said a minute.

"Strong operational demand acknowledged; turning off the tool increased risk of staff using unapproved alternatives."

NZ Police director of the national criminal investigations group detective superintendent Keith Borrell did not mention the misuse in his statement to RNZ.

Having noted Whisper's value for investigative work, Borrell said: "This tool was briefly turned off while the trial results were analysed, additional safeguards were put in place and guidance was updated."

Police did not agree to an interview.

'Not aware of any material issues'

Police considered Whisper was the most suitable model to test, said Borrell.

As for US reports about it fabricating content that never existed, he said: "We are not aware of any material issues to date arising from the inaccuracies inherent in automated translation and transcription."

Users of it now had to have completed mandatory AI awareness training and been made aware of the "limitations of these types of tools". The use for recordings primarily in Te Reo Māori or Pacific languages was not authorised.

"Text is added to the outputs to make it clear that transcripts must not be relied upon for critical decision making or evidential purposes without verification by an appropriately qualified person," said Borrell.

The police policy introduced in March 2025 said if GAI was used on a draft, that must be disclosed to the court if the final human-reviewed transcript was used evidentially.

Police policy requires six-monthly use audits of GAI.

RNZ has asked to see these audits for Whisper AI.

Late on Tuesday, Borrell said police had not done an audit of Whisper's use.

"Although no formal audit has been undertaken since the trial and evaluation last year, TranScriptor [as police call the model] is managed under stringent usage controls.

"The service is subject to ongoing oversight to ensure reliable operation, with system performance continuously monitored and every use is logged against the user for any future auditing purposes."

Police documents emphasise having "a continuous evaluation cycle" and open communication with the public about the use of AI.

The Justice Ministry said it was looking to use transcription technology of what went on in court.

"As the New Zealand Police is [sic] also progressing the use of Speech to Text for their transcription requirements, both agencies are in regular contact to share and learn from their respective approaches," a spokesperson said.

It has a tender out.

Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.