Missing ASIO files

Back in October I noticed that my harvest of ASIO files from RecordSearch hadn’t captured the access status properly – in particular ‘Open with exception’ had been truncated to ‘Open’ – oops!

I fixed my harvester and tried to update the metadata for the files I’d harvested previously, but there was a problem. About 400 of the files I’d harvested last year were missing – a search for their barcode came up empty. I created a list of the missing files and shared it to see if anyone had any thoughts on what was going on.

I also submitted a reference inquiry to the NAA and they replied on 16 November. The full explanation is below. It’s a bit complicated, but basically there was a reorganisation and the files were wrongly tagged in RecordSearch – they’re still in the system, they’re just not visible to the public.

I’ve just checked a couple and they still seem to be missing – it’s not clear when the problem will be fixed.

Yet another example of why you shouldn’t trust search interfaces!

Response from NAA, 16 November 2016:

The cause of this problem originates in the practices used when ASIO files first began to be transferred into NAA custody. At the time, when an application for access was received for an ASIO file, ASIO would transfer the original records (or, an unredacted photocopy of the original) into NAA custody along with a redacted reference copy. The unredacted records were kept secure by NAA, while the reference copy was provided to the applicant in response to their application for access.

When these records were received, a RecordSearch registration was created for the original, unredacted records and these were assigned a barcode. In most cases, the reference copies were not separately registered on RecordSearch and were not assigned their own barcodes (although this did start to occur in later cases). When a member of the public requested access to the original item which was registered on RecordSearch, the reference copy would be retrieved. The barcodes which you have identified are all associated with the unredacted records that were transferred to NAA under this system.

Over time, this system proved to be quite cumbersome, and led to some duplication and confusion. The decision was then made that instead of transferring both unredacted and redacted records into custody, ASIO would retain the original records and just send the redacted reference copies to NAA. As these were the only records required to provide public access it was deemed unnecessary to send the originals as well. Although the practice changed, all of the unredacted records transferred to NAA previously were not returned to ASIO, and were held by NAA until recently.

As you may know, we are currently undertaking a major relocation project where our Canberra-based collection is being moved into the new National Archives Preservation Facility. As part of our preparation for the relocation, our collection management staff conducted a survey of records in custody which could be returned to the controlling agencies in order to simplify the relocation process. The unredacted ASIO records that had come into custody under the old system were identified during this survey. It was agreed with ASIO that as NAA already had reference copies of these records, that the unredacted records could be returned to ASIO. This withdrawal of records took place in June-July this year.

However, there was an unintended consequence of this withdrawal of records. When they left NAA custody, all of the barcodes that had been created for these items were marked on our system as being withdrawn. Any items which are marked as withdrawn can no longer be viewed on the public version of RecordSearch, which is why you have not been able to locate them.

In order to correct this problem, we will remove these markers from the item registrations, and associate them with the reference copies that are still in custody, rather than with the unredacted copies which have been withdrawn. The details of each item (including the barcode) will not change, but they will reappear on the public version of RecordSearch. Given that many of our staff are currently busy with arrangements for the relocation, this may take some time, but we are endeavouring to correct it as soon as possible.

This said, as part of this process, we have discovered some instances where the unredacted copy has been withdrawn, but where the reference copy is missing or was not received by NAA. We are following this up with ASIO to have them provide reference copies of the affected items. These items will not ‘reappear’ until a reference copy is received from ASIO.