UC10154 -- Exploring digital heritage -- Week 9

Draft, 05 October 2016
This page should be in a useful state, but still needs work before it's finished.

Redaction art

A few weeks ago I mentioned my recent attempts to try and find redactions in ASIO files. A redaction is where text in an official document is blacked out, or removed, before it’s released to the public. Sometimes there’s not much of the document left by the time the redactions have been applied. (Or has it all been a dreadful mistake? Read my all time favourite Onion article if you’d like a laugh.)

I’ve harvested about 300,000 page images from ASIO files via the National Archives of Australia’s website and I’m interested in extracting information about the amount of information redacted. But how?

I’ve created a little computer script that looks for black blobs on the page, crops them out of the original image, and save them to a new file. I’ve now got a collection of about 250,000 black blobs. In checking the results of my script and weeding out a significant number of false positives I came across a fascinating little gallery of redaction art. There’s more of the story here:

Obviously the person doing the redactions got a bit bored, or was seeking an outlet for their creative impulses.

Seeing like a computer

I thought I’d share my redaction art discoveries today because they’re an example of how digital tools and techniques can lead you in all sorts of unexpected direction; but more importantly they’re an example of the use of computer vision in cultural heritage.

There’s no pre-built tools for finding redactions (at least not that I know of), so I had to code my own. However, there is a very widely-used library of code called OpenCV that meets most computer vision needs.

I’ll save you all the coding details, but I did want to walk you through the process. If you’ve used image editing tools like Photoshop, some of these steps might be familiar.

  • I removed the colour from the image (converted to grayscale).
  • I blurred the image slightly to remove some of the background noise, but I used a technique that is supposed to maintain sharp edges.
  • I applied thresholding to reduce the image to black and white.
  • I found all the black bits in the image.
  • I traced the contours of all the black bits to find their shape.
  • I looped through all the contours rejecting any that seemed too large, too small, or too close to the edge of the image.
  • I found the centres of the contours that were left and checked that a 10px x 10px sample from the centre was all black.
  • I then grabbed a rectangular box around each of the remaining contours and saved it as a new image.

This is probably not the best or most efficient way of finding redactions. I developed my script by trial and error against a set of test images. Even so, there’s so much variation amongst the source images that I’ve ended up with 15-20% of false positives. I’d like to find a way to reduce that.

You might be thinking that it seems like a lot of work to find a little (or not so little) black blob. And it is. One of the things your constantly reminded of when working with computer vision applications is how good humans are at recognising patterns. It’s a lot harder for computers!

Finding faces

People are particularly good at recognising faces. In fact, we’re so good at it that we tend to ‘see’ faces all over the place – this is called pareidolia. Have a look at the photos shared by @FacesPics on Twitter – do you see the faces?

Faces don’t matter much to computers, so they’re not hardwired to recognise them in the way we are. However, facial detection is a very common computer vision task. So how do you do it?

First of all we need to distinguish between ‘facial detection’ and ‘facial recognition’. Facial detection is just finding a face in an image – it’s what happens when your camera draws a box around faces as you’re taking a picture.

Facial recognition means taking an image of a face and then matching it against a database of previously identified faces – it aims to identify the owner of a face. We see facial recognition all the time on TV crime shows, though in the real world it’s rather more complicated. Facial recognition is much harder than facial detection – though the computers are learning fast!

Facial detection is one example of a broader category of computer vision tasks called object detection – the ability to identify particular things in a image. The objects you’re looking for might be faces, dogs, or bananas, the approach is basically the same. First you use a selected set of images to create a model of the thing you’re looking for. Then you can use this model to detect instances of that thing in a new set of images. It’s sometimes referred to as machine learning because you’re training the computer to ‘see’ the thing you’re interested in.

So to find bananas you’d train your machine with two sets of images – one with bananas, and one without. It would gradually learn to spot the features that define a banana. But what makes a face? The models included with OpenCV are pretty basic and really just look for bands of contrast where the eyes nose and mouth should be. To demonstrate this I tested a facial detection script on a set of simplified faces. This, for example, was detected as a face:

Sample face

This video shows how a basic face detection model or classifier analyses a new image. It moves a window across the image looking for matching features. If enough features are detected it saves the window as a possible match. If a number of these marked windows overlap, it’s likely to have found a face.

OpenCV Face Detection: Visualized from Adam Harvey on Vimeo.

It’s simple, but it works pretty well, and it creates some interesting possibilities.

Creepy or compelling?

OpenCV includes a pre-trained face detection model based on many thousands of images, so you don’t have to start from scratch. As a result it’s surprising easily to create your own script and start finding faces.

Let me introduce you to the Vintage Face Depot. @facedepot is a Twitter bot I created that offers specialised face replacement services. Tweet a photo of yourself (or someone else) to @facedepot, and the bot will respond with a new version of you, blended with a historical face drawn from Trove’s digitised newspapers.

Try it! Just tweet a photo of a face (you or your favourite celebrity) to @facedepot. Some hints:

  • The name of the bot @facedepot needs to be at the start of the tweet.
  • Faces are most likely to be detected if the whole of the face is in frame, and there’s some space around it. (So no extreme close ups.)
  • Good even lighting also helps.

Face swap applications are all over the place now. Indeed, there’s a new service in development called Dreambit that is described as a ‘personalised image search engine’:

Given one or more photos and a text query, it outputs versions of the input person in the query appearance. We let people explore a large variety of looks.

Dreambit demo

The Vintage Face Depot is much more basic. When it receives your photo, it:

  • runs a facial detection script over your photo and saves the coordinates of up to four faces;
  • searches a database of faces I’ve already extracted from Trove, looking for ones that are at least as big as you face;
  • randomly selects one of the size-matches faces;
  • resizes the Trove face to match yours, masks the shape to create an oval, blurs the edges, and reduces the opacity (to make it slightly transparent);
  • pastes the new face on top of yours;
  • tweets back the new image.

As you will have noticed, the Vintage Face Depot doesn’t attempt a seamless transformation. You may also see little ghost faces popping up all over the place where the script has wrong detected a face (machine pareidolia?). It’s very obvious the image has been tampered with!

However, by making the new face slightly transparent, you do get interesting and sometimes unsettling results. You may notice the colour of your own eyes or lips showing through the historical image. You may have sprouted a new moustache. The bot also includes a link to the original newspaper article in Trove where you can find out more about the person behind your new face. My hope is that this simple bot can enable people to make new connections to the past – whether they’re funny, disturbing, or tragic.

If you want to build something using my collection of faces from Trove you can use my special Face API, or download the complete set.

I explored this same sort of space in an earlier work, called Eyes on the Past. For this work I found faces in Trove, and then found eyes in those faces – yes OpenCV also includes an eye classifier! The result has been variously described as beautiful and creepy – so I figure I must be doing something right. As with @facedepot, I’m using faces as a way of exploring the types of connections we make with the past.

My work with faces goes back several years. I was looking at a collection of documents held by the National Archives of Australia that were used in the administration of the White Australia Policy. The documents were used for identification by people living in Australia, but deemed non-white – if they wanted to travel overseas they needed these documents, or they would be refused entry on their return. They’re visually compelling documents with portrait photographs on one side, and inky-black handprints on the other.

I’ve now harvested a large collection of documents relating to the White Australia Policy and created a simple interface to browse them – go to http://iabrowse.herokuapp.com/ to explore.

Looking at the documents I wondered whether it might be possible to extract the portrait photographs from the larger images. So I googled ‘facial detection’. Within a couple of days I’d built The Real Face of White Australia – a scrolling wall of thousands of faces. I still find it very moving. You can read more about it in this draft chapter Kate Bagnall and I wrote for a book on the use of computer vision in history.

Face detection as a service

My fiddling about with facial detection is all pretty basic. Nowadays there’s money to be made in offering these sorts of image recognition services, so the tools are becoming much more sophisticated. If your business needs to process lots of photos, it can make use of custom APIs created by companies like IBM and Face++

They do, however, provide some neat little demos that we can play around with.

  • Go to the Face++ demo site.

  • There are a number of potted examples you can try, just click on a face.

  • You can also supply a url, or upload an image. Find an image on the web and try it out.

  • You’ll notice that this service does more than just detect a face – what other information does it provide?

How do you think businesses might make use of this sort of information?

Recognising faces

I said that recognising faces was much harder than just detecting them. However, there’s been significant advances over the last few years. The development of neural networks – computer applications modelled on the brain – have greatly improved the range and accuracy of machine learning.

Google and Facebook are investing a lot of effort to bring facial recognition up to human-like levels of accuracy. In 2014, Facebook announced that its ‘DeepFace’ system had reached an accuracy of 97.35% – close to humans. What did they train their neural network on – perhaps it was you! One advantage the FaceBook research team had was access to millions of identified photos of people – Facebook users. Yep, that’s one of the potential uses of your data you agreed to when you signed up.

But not to be outdone, in 2015 Google announced it’s ‘FaceNet’ system had reached a new record accuracy of 99.63%! It should be noted that these accuracy figures are against one particular set of celebrity images harvested from the web – I’m not really sure how we’re meant to interpret them.

There’s a nice summary of facial recognition developments in this article from Science‘Facebook will soon be able to id you in any photo’.

How might these sorts of technologies be used in cultural heritage? Last year it was announced that a research team had used facial recognition to ‘identify’ a new portrait of Anne Boleyn. Given that the recognition system was trained using this medal, the only undisputed image of Anne Boleyn, I can’t help but be a bit sceptical.

Boleyn medal

You can certainly see how large photographic collections could use this technology to find previously unidentified images of particular people. Can you think of other possibilities?

Issues and dangers

Are you creeped out yet? There’s certainly something disturbing about these technologies, particularly when it comes to rapid advancements in facial recognition. If Google and Facebook are doing it, I think we can safely assume that governments are investing in similar sorts of systems. Indeed, about a year ago the Australian government announced that its ‘newest national security weapon’ was a national facial recognition system linking up existing face-identified databases, such as car licences – it’s called ‘the Capability’.

Never fear, if you want avoid identification on the street, you could adopt one of these new fashion styles, specifically designed to confuse facial detection systems.

CV Dazzle

But it’s not just identification. You may have noticed that the Face++ demo returned information on your sex, whether you’re smiling, and your race. There have been studies examining it’s possible to detect a person’s sexuality from their face. If we, in the cultural heritage sector, are going to make use of these sorts of technologies I think we need to be aware of the political and social implications of using faces as a means of assigning people to categories.

Melissa Terras also raises questions about how easy access to these powerful photo manipulation tools might change the historical record. There have always been fakes of course, but is it just becoming too easy, too seamless. What can we trust?

For the ultimate in creepiness watch this video demonstrating a live face-swapping system.

Face Substitution from Kyle McDonald on Vimeo.

Image tagging and classification

Another area in which neural networks having been making rapid advances is in adding subject tags or descriptions to images. Just the other week Google released details of its latest image captioning system. Against a set of test images it achieved an accuracy of 93.9%. But not only does it detect the subjects of an image, it writes a caption in natural language. Once again, this model has been trained on a set of images manually captioned by human beings, but Google claims it can go beyong merely applying the same captions to new images, it can identify wholly new contexts.

It’s easy to see how such technologies could be of immense value to cultural heritage organisations that never have enough money to describe all of their holdings. But once again we have to be wary. Last year Google’s photo app caused considerable controversy when it’s automatic tagging system labelled African-American people as ‘gorillas’. It’s not that the image recognition systems are racist, but there are always likely to be biases in the image sets we use to train them. We have to always be thinking about the judgements implicit in the development of these sorts of systems.

Once again, image tagging and recognition are being marketed to business as a way of extracting useful information. Let’s try using these forces for good, and see if IBM’s demo system can help me sort my redactions from the false positives.

I’ve prepared three zip files:

Download all the files. You can unzip the ‘Assorted’ examples, but you don’t need to worry about the others as we’ll upload the zip files directly.

  • Go to the IBM demo page.

  • Have a play with their potted examples to see the sorts of classifications that are possible.

  • Now click on the ‘Train’ tab.

  • Once again they have some potted examples that you can play around with – feel free to try them out.

  • Now let’s make our own classifier! Click on the square under ‘Use your own’.

  • Under ‘Positive classes’ select and upload the positive.zip file.

  • In the text box give your positive examples the name ‘Redactions’.

  • Under ‘Optional negative classes’ select and upload negative.zip.

  • In the text box give your negative examples the name ‘Non redactions’.

  • Click on the ‘Train your classifier’ and wait. It could take a minute or two.

  • Once your classifier is ready you can feed it one of the images in the assorted file. Just select and upload it.

  • The classifier will tell you whether it thinks the image you uploaded is a redaction or not. Was it right?

Other cool stuff

Keep your eye out for other interesting projects in the cultural heritage domain making use of computer vision.

  • Here’s a project which is trying to find poetry in digitised newspapers by looking at the characteristic shapes of poems – jagged lines, white space etc.

  • Here’s a cool experiment that uses image morphing to bring old stereoscopic images to life.