Visual extraction in Refiner

This feature of Refiner will help you extract images from portions of similar documents and detect their contents, such as if a checkbox is checked or a signature line is signed, even if these regions aren’t always in the same locations across multiple documents.

By the end of this guide, you will be able to:

  • Create a Refiner workspace, and upload files for visual extraction.

  • Navigate the Refiner UI.

  • Define and create anchors based on consistent file structure.

  • Extract an image.

  • Use an extracted image to return a boolean for checkbox and signature completion.

1. Setting up your workspace

Refiner has a few built in project templates to help you get started quickly.

The Refiner project template

Activity:

  1. Create a new workspace, and remove any starter folders.

  2. Select New > New Project > Refiner Lesson 2 - Extracting Image Features, and name your project.

2. The Refiner UI

The .ibrefiner file stores information about what you’ll be extracting from your uploaded files. This section breaks down how to navigate and edit the Refiner interface as a whole, beyond individual files.

Activity:

  1. From the viewme.ibrecipebook file, click on Edit in “Edit Refiner Program”.

  2. From the Document tab, click through each invoice file in turn. There might be a view delay after you select different files, but notice that each file has varying degrees of completion for their signature and checkbox fields.

  3. Above each document, you can select the magnifying icons to change zoom levels, horizontal arrows for zoom to fit width, and vertical arrows for zoom to fit height.

  4. The Fields column on the right of the page allows you to set extraction fields, which we’ll do in the activities below. Notice that the Output Table tab includes an example field, because we have not created any fields of our own.

  5. In the Settings tab, ensure that “Enable Provenance Tracking” is on, as well as “Best Guess Provenance for UDFs”.

3. Defining anchors

Defining an anchor is the first step you need to take in any visual extraction workflow.

Anchors are simply Refiner functions that you believe will have a consistent location over all of your documents. Visual extraction uses them to figure out where in a record to locate the image, checkbox, or signature you want to extract. Anchor functions don’t move page position in relation to the items that you’re extracting.

Discovering repeated text

In this task, we’ll try to decode the checkbox next to the phrase, “This is a domestic invoice”. Because this phrase is constant in our document templates, we can use it to discover the state of its nearby checkbox.

Activity:

  1. From the Document tab in the .ibrefiner file, select the document-1.pdf.ibdoc file.

  2. Hover over the existing example_function, and select the Red bin icon.

  3. In the “Fields” pane, select + New Field to add a new text anchor.

  4. From the bottom pane, you can edit, rename, and test fields. Next to the “Text field” button, double click on field_1 and replace the field name with __anchor_domestic

    • The double underscore is a naming convention used to hide helper fields from being generated in your output and downstream applications like “Review OCR”.

  5. In the text field below this title, where it reads “Enter Refiner Function..”, type scan_line(). Notice that the right side of the bottom pane now updates with relevant function documentation.

  6. Add two arguments to your function, so that it reads: scan_line(INPUT_COL, 'This is a domestic invoice').

  7. Select Run, and see if your expected outcome is highlighted in red in the visual field, and follow it up with a Save to preserve your work.

    Note

    You also could highlight the same field by using a combination of match() and list_get()! Try it out if you want a more advanced exercise.

4. Image extraction

After you’ve defined an anchor, you’ll want to use it to discover positionally relevant items.

Checkbox extraction

Because we’re working on visual extraction, we’ll create another function to extract an image of the nearby checkbox.

Activity:

  1. From the Document tab in the .ibrefiner file, make sure you’re still located in the document-1.pdf.ibdoc file.

  2. Create a new field using the same instructions as above, and rename it to img_domestic_checkbox.

    • Image extraction must begin with img. If you name it anything else, it will be encoded as a string, and you will not be able to preview your result effectively.

  3. To the left of the renamed field, click the Text Field button and change it to Visual Field.

  4. Leave “Image Crop” as your default, but add __anchor_domestic as your “Anchor Field”.

  5. Return to your visual pane above. Use the Zoom in button to magnify your field, and scroll down until you find the checkbox located to the left of “This is a domestic invoice”.

  6. Use your cursor to click and drag a square around the selected checkbox. If successful, you have an orange-red box that surrounds the marked checkbox in your image.

  7. Click Run, and then navigate to the Output Table tab. Select the first cell under _img_domestic_checkbox.

  8. If successful, you can view the checkbox as an image from the Output tab in your bottom pane. Click Save to preserve your work.

5. Decoding an image

After you’ve extracted an image, you might want to create functionality to discover things like page completion, checkbox selection, or signatures.

Discovering a checkbox output via image functions

Because we’ve successfully extracted an image of a checkbox, we can use that knowledge to perform a similar step with built-in Instabase functionality. You’ll notice that the “Checkbox extraction” activity above is not required if you want to use this type of functionality in the future.

Activity:

  1. Create a new field and name it domestic_is_checked.

  2. Convert this field to a Visual Field, and change the “Visual Function Type” to Checkbox. Again, use __anchor_domestic as the anchor.

  3. As before, return to your visual pane and zoom in to the same checkbox field. Use your cursor to box the selected checkbox field, then click Run.

  4. Select the Output tab, where True is the expected response.

  5. Navigate to Output Table, and notice the results. Which one is false? Click Save to preserve your work.

Discovering a checkbox output via text functions

This activity uses Instabase’s built-in functions on extracted images.

Activity:

  1. Create a new text field and name it domestic_is_checked_alternative.

  2. In the text field below this title, where it reads “Enter Refiner Function..”, type image_decode_checkbox()

  3. Use img_domestic_checkbox as the function’s argument, then click Run.

  4. Select the Output Table tab, where True is the expected response.

  5. Click Save to conclude the activity.

6. On your own—extracting a signature

Now that you’ve learned how to define anchors and extract images, use that knowledge to extract the signature field from each of the images. Feel free to use steps from the checkbox activity to help jog your memory if you get stuck.

Hints:

  • Across each of the three invoice files, it looks like “Sincerely yours” is a common field.
  • When you think you’ve completed the activity, click on the Output Table tab to view the fields you’ve returned across each image. Does each field return its expected result?

Conclusion

That’s it! You can now navigate and extract visual fields in the Refiner app.

If not, reach out to us at training@instabase.com. We’d love to chat about any questions, comments, or concerns that you might’ve had in completing this guide.