Accuracy metrics

Verify flow accuracy with a ground truth set, which specifies accurate data for a set of documents. By comparing the ground truth set to actual data generated from a flow, you can see how accurate your solution is.

Ground truth sets are associated with a specific flow. For a 1:1 comparison, you must use the same set of input documents to generate the ground truth set and test the flow. You can create multiple ground truth sets for a flow to test different batches of input documents; just be sure you select the appropriate ground truth set when you test a given set of documents.

Ground truth sets are exported from Flow Review (Export > Ground Truth), preferably using corrected flow results, although you can manually correct ground truth values later. You then re-run the flow using the same set of documents, and select Compare against ground truth set to generate an accuracy report. During flow runs that you’re testing for accuracy, bypass any validation checkpoints, because correcting results in Flow Review doesn’t give you accurate metrics about flow performance.

Within Solution Builder, in the Evaluate section, you can view, configure, and edit ground truth sets, and view and compare accuracy reports.

Configuring and editing ground truth sets

You can save a variety of configurations to ground truth sets which affect how accuracy metrics are calculated, such as ignoring whitespace, ignoring casing, or rounding numbers.

To configure a ground truth set, from Solution Builder, locate the ground truth set and click Configure. You can modify configuration globally for all fields on the Global Config tab, or for individual fields on the Fields Config tab.

To manually edit ground truth values, click Open, then correct values as needed. Edits are persistent, so the next time a comparison is run against the ground truth set, your modified values are used.

Viewing and comparing accuracy reports

Accuracy reports summarize performance metrics when you compare flow output to a ground truth set.

After running a flow with ground truth comparison enabled, click View Accuracy to see your accuracy report. Alternatively, you can view all accuracy reports generated by a flow: in Solution Builder, select the Accuracy Reports tab.

The default view of the accuracy report shows the overall accuracy for the flow job. Additionally, a Class Level Accuracy table details accuracy for each class in the flow. If your flow doesn’t include classification, only one class is listed with the title flow.

To view field-level accuracy metrics, select a class in the left sidebar or in the Class Level Accuracy table. To compare extracted data to the ground truth set, drill in to specific fields.

You can compare accuracy reports to any other report generated using the same ground truth set. Comparison can be helpful in fine-tuning a flow. For example, you might test flow performance with different OCR settings or modules, but using the same ground truth set and input documents. To compare an accuracy report, in the report controls at top right, click Compare and select a report to compare against. Each metric displays the difference between the current report and the past report. Increases are green and decreases are red.

Accuracy metrics

These key terms are helpful in understanding overall accuracy metrics.

• Automation Rate measures how many extracted fields generated any value.

• Validated Accuracy measures how many extracted fields generated the correct value.

• Raw Accuracy measures how many extracted fields generated the correct value, including any that triggered validation.

• Classification Accuracy measures how many records were classified correctly. If your flow doesn’t include classification, all records are effectively assigned to a default class and this metric reports 100 percent.

These key terms are helpful in understanding field-level accuracy metrics.

• True Positive indicates that the correct value was extracted.

• False Positive indicates that an incorrect value was extracted.

• True Negative indicates that an incorrect value was extracted but triggered validation.

• False Negative indicates that a correct value was extracted but triggered validation.