Project 6:

An Image Labeler

Due:  Tuesday, October 31st  *in class*

In this project, you will build a Chrome extension that will label the images on a web page that do not already have “alt text” using computer vision.

Yet Another Chrome Extension

The basis for this project will be a Chrome extension. To get started, copy and paste the extension that you built for Project 3 into a new directory. Enter its manifest to have a different name. I suggest something like, “Image Labeler,” and then load it into the browser. I would disable the extensions you created for prior weeks. As you add in more and more, the chances increase that something will conflict with something else, and it might be confusing to tell what is going wrong.

To make sure all the communication works, especially for images that aren’t hosted on the same domain as the main page, you’ll need to give your extension some more permissions, which you can do by editing your extension’s manifest. Mine looks like this:

{
 "manifest_version": 2,
 "name": "Automatic Alt Text",
 "minimum_chrome_version": "24.0.1307.0",
 "version": "0.0.1",
 "homepage_url": "http://www.accessibilitycourse.com",
 "icons": {
 },
 "content_scripts": [
   {
     "matches": [
       "*://*/*"
     ],
     "js": [
         "scripts/jquery.min.js",
         "scripts/ocr.js",
         "scripts/tesseract.js"
     ],
     "css": ["css/reader.css"],
     "run_at": "document_idle"
   }
 ],
 "permissions": [
   "*://*/*"
 ]
}

Checking which Images Might Need Alternative Text

The first step is to iterate through the images on the page that you load with your extension to see which images might need better alternative text. You can iterate through the images like this:

$('img').each(function() {

  if($(this).is(‘img:not[alt]’) {  // image does not have alt text

  } else if($(this).attr(“alt”)==””) { // image has empty alt text

  } else {  // image has some alt text defined

  }

})

This allows you to access the different conditions an image might be in, but doesn’t do much to help you figure out how to fix a particular image.

What’s in the Image?

What makes good alternative text is highly dependent on image contents and how the image is used. Thus, we first need a way to start understanding what is in our images.


To help facilitate that, I’ve made available the Amazon Rekognition service via a web service.

To access it, you’ll use something called AJAX, which will allow you to query the service from your extension, and then use the results to help determine the alternative text.
http://api.jquery.com/jquery.getjson/

You can call this service using this URL:
http://www.accessibilitycourse.com/support/rekognition.php?url=<URL>

Recognizing Text

Another really handy thing you can do to help write alternative text is to recognize text. These days, basic OCR engines can be written entirely in Javascript(!), which is pretty crazy. For instance, this is a port of the popular Tesseract OCR:

http://tesseract.projectnaptha.com

Use this to generate alternative text for images that contain text.

You’ll want to download the tesseract.js file, and include it from your extension.


The Tesseract Javascript library can be called in a number of different ways, but you’ll be most successful if you call the
recognize function on the URL of the image (src attribute). For instance, this code from the Tesseract.js web page would call recognize on the first (0th) image in the page.

Tesseract.recognize($($("img")[0]).attr("src"))

    .progress(message => console.log(message))

    .catch(err => console.error(err))

    .then(result => console.log(result))

    .finally(function(resultOrError) {

      console.log(resultOrError);

    })

})

Note that instead of calling this directly on the jQuery result, recognize function is call on the [0]th element, which gives you the base DOM element that represents the image. You will want to replace the code in the finally function to have it instead attempt to add alternative text.

The easiest way to apply this to multiple images is to do so sequentially. That is, call this function once for one image, wait for it to complete, and then call it again for the next image.

Further instructions for using Tesseract are on the web page above. Please follow them in order to add alternative text for your web page.

Assigning Alternative Text

The idea is to then use what might be in the image, and possibly the recognized text to assign appropriate alternative text to each image.

You can assign new alternative text to an image using:
img.attr(“alt”, “the new alt text”)

The trick here is how to use the information available to assign “good” alternative text. I encourage you to think about how you might reason about this to assign the highest-quality alternative text that you can, while expressing appropriate confidence in the results that you get back.

Grading

Main components of our grading will be:

Using the appropriate keyboard commands, does the following happen:


This page and contents are copyright Jeffrey P. Bigham except where noted.