Project 6:

An Image Labeler

Due:  Tuesday, October 31st  *in class*

In this project, you will build a Chrome extension that will label the images on a web page that do not already have “alt text” using computer vision.

Yet Another Chrome Extension

The basis for this project will be a Chrome extension. To get started, copy and paste the extension that you built for Project 3 into a new directory. Enter its manifest to have a different name. I suggest something like, “Image Labeler,” and then load it into the browser. I would disable the extensions you created for prior weeks. As you add in more and more, the chances increase that something will conflict with something else, and it might be confusing to tell what is going wrong.

Checking which Images Might Need Alternative Text

The first step is to iterate through the images on the page that you load with your extension to see which images might need better alternative text. You can iterate through the images like this:

$('img').each(function() {

  if($(this).is(‘img:not[alt]’) {  // image does not have alt text

  } else if($(this).attr(“alt”)==””) { // image has empty alt text

  } else {  // image has some alt text defined



This allows you to access the different conditions an image might be in, but doesn’t do much to help you figure out how to fix a particular image.

What’s in the Image?

What makes good alternative text is highly dependent on image contents and how the image is used. Thus, we first need a way to start understanding what is in our images.

To help facilitate that, I’ve made available the Amazon Rekognition service via a web service.

To access it, you’ll use something called JSONP, which will allow you to query the service from your extension, and then use the results to help determine the alternative text.

You can call this service using this URL:<URL>

Recognizing Text

Another really handy thing you can do to help write alternative text is to recognize text. These days, basic OCR engines can be written entirely in Javascript(!), which is pretty crazy. For instance, this is a port of the popular Tesseract OCR:

Use this to generate alternative text for images that contain text.

Instructions for using Tesseract are on the web page above. Please follow them in order to add alternative text for your web page.

Assigning Alternative Text

The idea is to then use what might be in the image, and possibly the recognized text to assign appropriate alternative text to each image.

You can assign new alternative text to an image using:
img.attr(“alt”, “the new alt text”)

The trick here is how to use the information available to assign “good” alternative text. I encourage you to think about how you might reason about this to assign the highest-quality alternative text that you can, while expressing appropriate confidence in the results that you get back.


Main components of our grading will be:

Using the appropriate keyboard commands, does the following happen:

This page and contents are copyright Jeffrey P. Bigham except where noted.