Test Runner

The Test Runner app uses test specifications to run regression tests for flows and apps on the Instabase platform. Test Runner leverages the previous output of flows that were successfully run and compares the representative ground truth (also known as the golden output) with the current test output. Test Runner runs the comparison operation using the test specification and creates a comparison report.

Usage guide

Getting started

  1. Open the Test Runner app (All apps > Test Runner).

  2. Click Open tests folder.

  3. Select the folder that contains your test specification files (.ibtest). You can also right-click the folder in the file explorer and select Open with > Test Runner.

    Note: If you haven’t created any test specifications files, select any empty directory.

The test library page opens and displays a list of all .ibtest files in the selected directory.

Note: .ibtest files can be nested inside other folders in the opened directory.

Creating and managing test suites

To create a new test suite and add test cases:

  1. On the test library tab, click + New suite.

  2. Name your test suite.

  3. Click Create test.

  4. Select the test suite in the list.

  5. Click Add new test.

  6. Configure the new test case.

For each test, add the test name, description, and test method. Based on the test method, configure the flow or solution path, the input folder, and the output folder. For post-test comparison, you can configure one of the comparators as shown in the test specification section.

Note

Click + Add section to add pre-run hooks, post-run hooks, or comparators to your test case.

  1. Click Save.

  2. Add additional test cases as needed.

To view and edit a test suite:

  1. On the test library tab, select the test suite to view.

  2. From the test suite view, you can add, edit, delete, and view test cases in the test suite. See the test specification section for more details.

To permanently delete a test suite from the file system:

  1. On the test library tab, locate the test suite to delete.

  2. Click the delete (trash can) icon.

  3. Click Confirm.

Running test suites and viewing results

  1. On the test library tab, select the checkbox next to the test suite to run. You can select multiple test suites to run at the same time.

    Note: To select all test suites, select the checkbox at the top of the test suite list. You can also run an individual test suite by selecting the test suite on the test library tab and clicking Run suite from the test suite view.

  2. Click Run test. The selected test suites run and Test Runner navigates to the execution history tab. From the execution history tab, you can view the results of your test. Test results save in a new folder named results in the opened directory.

You can also view past run results, test statistics, and errors in the execution history tab.

Using the legacy app

While the updated Test Runner UI improves the process of creating and maintaining regression test suites, the legacy version is still available. Click View in legacy app to access the legacy version of Test Runner.

Supported APIs

Test type API Description
Flow run_flow_async single Flow async
Flow run_flows_async multi Flows async
Flow Binary run_binary_async single Flow Binary async
Metaflow run_metaflow_async non-binary Metaflow async
Metaflow Binary run_binary_async binary Metaflow async
IBSolution run_solution run solution async

Test specification

The JSON-formatted .ibtest file includes a version and one or more test specifications. Enclose each test specification in square brackets.

For the test to succeed, test specification settings that alter the output of the Flow must match the settings of the Flow that produced the golden outputs.

Settings that do not alter the output of the Flow can be different, like enable tracing.

run_flow_async example test specification

{
  "version" : "1",
  "tests" : [
    {
        "name":"run-flow-test",
        "owner": "user@instabase.com",
        "description": "Sample flow test",
        "api":"run_flow_async",
        "payload": {
          "ibflow_path": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows/sample-3.ibflow",
          "input_folder": "input",
          "compile_and_run_as_binary": true,
          "delete_out_dir":true,
          "output_has_run_id":false,
          "skip_steps":[],
          "log_to_timeline": false
        },
        "on_done_config":
        {
          "failure_expected": false
        },
        "comparison_list": [
          {
            "comparator": "ibocr_comparator",
            "params": {
              "test_output" : [
              "s4_merge_files/out.ibocr"
            ],
              "golden_output": [
              "prai/my-repo/fs/Instabase Drive/flow-experiment/samples/sample-3/out-10-02-2020/sample-3/out/s4_merge_files/out.ibocr"
            ],
            "compare_config": {
                    "check_refined_phrases": true,
                    "compare_fields": ["annotation_data.separator"]
                }
            }
          }
        ]
    }
  ]
}

run_binary_async example test specification

{
 "version" : "1",
 "tests" : [
   {
       "name":"run-bin-test",
       "owner": "user@instabase.com",
       "description": "Sample flow binary test",
       "api":"run_binary_async",
       "payload": {
         "binary_path": "/prai/my-repo/fs/Instabase Drive/flow-experiment/build/bin/exercise1_Amazon/exercise1_Amazon-2020-06-08-16:20:26.ibflowbin",
         "input_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/input amazon",
         "settings": {
           "delete_out_dir": true,
           "disable_step_timeout": true,
           "flow_batch_size": "10",
           "log_to_timeline": true,
           "notification_emails": [],
           "notification_emails_string": "",
           "on_done_webhook_url": "",
           "output_has_run_id": true,
           "runtime_config": {},
           "use_flow_batch_mode": true
         }
       },
       "comparison_list": [
         {
           "comparator": "ibocr_comparator",
           "params": {
             "test_output" : [
             "s4_merge_files/out.ibocr"
             ],
             "golden_output": [
               "prai/my-repo/fs/Instabase Drive/flow-experiment/build/bin/golden-binary/s4_merge_files/out.ibocr"
             ]
           }
         }
       ]
   }
 ]
}

run_flows_async example test specification

{
 "version" : "1",
 "tests" : [
   {
       "name":"run-flows-custom-posthook",
       "owner": "user@instabase.com",
       "description": "Run multiple flows test",
       "api":"run_flows_async",
       "payload": {
         "flow_root_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows/",
         "input_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/samples/sample-3/input",
         "compile_and_run_as_binary": true,
         "delete_out_dir":true,
         "log_to_timeline":false,
         "output_has_run_id":false,
         "post_flow_fn": "none"
       },
       "pre_run_hook": {
           "type": "custom",
           "params": {
               "abs_script_path": "prai/my-repo/fs/Instabase Drive/golden-tests/Custom/flow_features/test_delete_output.py",
               "method_name": "log_time_test_runner"
           }
       },
       "comparison_list": [
         {
           "comparator": "ibocr_comparator_easy",
           "params": {
             "compare_files" : [
                 "sample-3/out/s4_merge_files/out.ibocr",
                 "sample-4/out/s4_merge_files/out.ibocr"
             ],
             "golden_output_root": "/prai/my-repo/fs/Instabase Drive/flow-experiment/samples/sample-3/out-10-02-2020"
           }
         }
       ]
   }
 ]
}

run_metaflow_async example test specification

 {
  "version" : "1",
  "tests" : [
    {
        "name":"metaflow-test",
        "owner": "user@instabase.com",
        "api":"run_metaflow_async",
        "payload": {
            "metaflow_file_path": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows2/workflows63d181f1-1d1e-475d-ab29-8d9a4c3471f8.ibmetaflow",
            "classifier_file_path": "prai/my-repo/fs/Instabase Drive/flow-experiment/classifier/classifier_custom.ibclassifier",
            "compile_and_run_as_binary": true,
            "delete_out_dir": false,
            "enable_ibdoc": false,
            "flow_root_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows2/",
            "input_dir": "prai/my-repo/fs/Instabase Drive/flow-experiment/metaflow-input",
            "log_to_timeline": false,
            "output_has_run_id": false,
            "post_flow_fn": "none",
            "skip_process_files": false
        },
        "comparison_list": [
          {
            "comparator": "ibocr_comparator",
            "params": {
              "test_output" : [
              "exercise1_Amazon/out/s4_merge_files/out.ibocr",
              "exercise2_Uber/out/s4_merge_files/out.ibocr",
              "exercise3_Food/out/s4_merge_files/out.ibocr"
            ],
              "golden_output": [
              "prai/my-repo/fs/Instabase Drive/golden-output/metaflow-out-8June/exercise1_Amazon/out/s4_merge_files/out.ibocr",
              "prai/my-repo/fs/Instabase Drive/golden-output/metaflow-out-8June/exercise2_Uber/out/s4_merge_files/out.ibocr",
              "prai/my-repo/fs/Instabase Drive/golden-output/metaflow-out-8June/exercise3_Food/out/s4_merge_files/out.ibocr"
            ],
            "compare_config": {
                    "check_refined_phrases": true,
                    "compare_fields": ["annotation_data.separator"]
                }
            }
          }
        ]
    }
  ]
}

run_solution example test specification

{
    "version": "1",
    "tests": [
        {
            "name": "simple_app",
            "owner": "dummy@instabase.com",
            "description": "Test an ibsolution",
            "api": "run_solution",
            "payload": {
                "solution_path": "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/simple_app.ibsolution",
                "input_folder": "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/inputs",
                "settings": {
                    "delete_out_dir": false,
                    "disable_step_timeout": false,
                    "flow_batch_size": "2",
                    "log_to_timeline": true,
                    "notification_emails": [],
                    "notification_emails_string": "",
                    "on_done_webhook_url": "",
                    "output_has_run_id": true,
                    "runtime_config": {
                        "sample_runtime_key": "sample_runtime_value"
                    },
                    "use_flow_batch_mode": true
                }
            },
            "comparison_list": [
                {
                    "comparator": "file_comparator",
                    "params": {
                        "test_output": [
                            "s1_apply_udf/input_1.txt.text",
                            "s1_apply_udf/input_2.txt.text"
                        ],
                        "golden_output": [
                            "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/out/s1_apply_udf/input_1.txt.text",
                            "ib_solutions/UAT/fs/Instabase Drive/graveyard-for-cd-jobs/simple_app/chetanmishra-2021-03-03 08:45:16.354873/unzipped/out/s1_apply_udf/input_2.txt.text"
                        ]
                    }
                }
            ]
        }
    ]
}

Test specification field reference

The test specifications fields depend on the API called and the Flow you are testing. Most fields are described below.

  • version: the version of Test Runner. Default: 1.

  • tests: the test category for one or more named tests

  • name: self-describing name of a test. For example, "run-flow-test".

  • owner: owner for the test - can be an email or username.

  • description: (optional) describes the test.

  • api: a supported API name: run_flow_async, run_flows_async, run_binary_async, run_metaflow_async, run_solution

  • payload: category that describes the Flow specifications.

  • ibflow_path: path to the Flow or root directory for flows.

  • input_folder: by convention, this folder is "input"

  • compile_and_run_as_binary:

    • true - to compile and run the Flow as Binary.

    • false - to run Binary version of Flow.

  • output_dir: Output directory for the test results. If output_dir isn’t specified, results are generated in the flow root folder. This configuration overrides output_has_run_id parameter.

  • delete_out_dir: (optional) Delete existing content in the output folder before running the Flow. If not set, default is false.

  • output_has_run_id: (optional) Write output to a directory with a unique, timestamped run_id. If not set, default is false.

  • skip_steps: (optional) A comma-separated list of integers that represent consecutive steps to skip while running a Flow. The steps must be the same as the ground truth (golden output) Flow. For example, to skip 2 consecutive steps at the beginning of the Flow with five steps, "skip_steps": [1,2]. To skip three consecutive steps at the end of the same Flow, "skip_steps": [3,4,5].

  • on_done_config: category that describes the on done configurations

  • failure_expected: true or false, true if flow failure is expected and has to be ignored.

  • post_flow_fn: none or a function name

  • skip_process_files: true or false

  • pre_run_hook: a function that just runs before the flow. Create a Python file with a fixed format as shown in Example: pre-hook function. Call this function by specifying a JSON object with the following fields:

    • type: should be the string "custom"

    • params: A JSON object specifying settings for the customer pre-hook:

      • abs_script_path: absolute path to the scripts directory

      • method_name: registered function name

  • comparison_list: A list of JSON objects specifying different comparisons to run for the test. Each object must have the following format:

    • comparator: type of comparator

      • "file_comparator_easy": compares two JSON, .csv, .txt, and .xls files when the output directory is the same for all ground truth outputs.

      • "ibocr_comparator_easy": compares two .ibocr, ibdoc, or .ibmsg files when the output directory is the same for all ground truths (golden outputs).

      • "custom_comparator": use Custom comparison logic

      Older versions of comparators for file comparison and ibocr or ibdoc comparison are still supported for backward compatibility:

      • "file_comparator": compares two JSON, .csv, .txt, or .xls files.

      • "ibocr_comparator": compares two .ibocr, .ibdoc, or .ibmsg files based on the fields specified under compare_fields.

    • params: A JSON object of specifications for the given comparator, which can have the following fields:

      • test_output: a list of all the comparison (new) output files relative to the root folder.

      • golden_output: a list of all ground truth (golden output) files with absolute paths.

      • compare_config: section of optional key-value pairs to configure comparisons.

      • check_refined_phrases (bool): if True, compare refined phrases for the output and ground truth set. By default this is set to True.

      • skip_hidden_phrases (bool): if True, skip comparing refined phrases for hidden fields (fields that start with __). By default this is set to False.

      • check_extracted_tables (bool): if True, the output file will include a section “Table Accuracies By Field” that displays the accuracy for each table in a field’s extracted tables list. By default this is set to False.

      • compare_fields (list[str]): a list of fields in the .ibocr, .ibdoc, or .ibmsg to compare against. Fields must be from the following list:

        • 'text' - the text of an .ibocr, .ibdoc, .ibmsg record.

        • 'annotation_data.separators' - the record’s annotation data separators.

        • 'classification_payload' - the record’s classification payload, which includes information like the class label, score, and page ranges.

        • 'barcodes' - compares all extracted barcodes for the record.

Custom comparison logic

Custom comparators use a registered custom comparator that lets you write your own logic for test. For example, write a custom comparator that checks if a file exists or counts the number of .ibocr files in a folder.

To register a custom comparator:

  1. Create a file with Python script that holds the custom comparator logic.

  2. Register the function with these three arguments: {file_client, logged_in_user, params}

  3. Define results.

    On success return: [True, None] Otherwise, return: [False, error]

Custom comparator field reference

  • params: specifications for custom comparator

  • script_path: absolute path of the file that holds the Python code for custom comparator

  • test_method: the function to execute in your script

  • additional_params: (optional) define other parameters of any name to pass some more information to your comparator logic, these parameters can be consumed by the test_method

from typing import Tuple, Text
from instabase.storage.fileservice import FileService
from instabase.test_runner_utils.common import ComparisonParams
from instabase.content.filehandle import ibfile


def is_rbi_pdf_present(file_client, logged_in_user, params):
  # type: (FileService.Iface, ComparisonParams) -> Tuple[bool, Text]
  file_path = params.comparison_params['additional_params']['output_file_path']
  if ibfile.exists(file_path)
    return True, None
  return False, 'RBI.pdf File not found'

Define pre-hook function

A pre-hook function is a Python script that runs before the Flow.

For Test Runner, register the pre-hook function using a specific structure.

  • Create a class that inherits PreHookBase class

  • Provide an implementation for the ‘execute’ method of this class

  • Register a function that calls this execute method internally

  • Give the name of this method as a parameter of pre-hook in the test specification configuration as a part of a test.

  • The execute method must have the return type as Tuple[bool, Text].

    • If the core logic inside execute succeeds, then return [True, None]

    • If the logic is unsuccessful, return [False, error_message]

Example pre-hook function

This example shows the required format of the Python script for a pre-hook function.

from instabase.test_runner_utils.common import PreHookParams, PreHookBase, PreHookContext

class ClassName(PreHookBase):
  def __init__(self, context):
    super(ClassName, self).__init__(context)


  def execute(self):
    # type: () -> Tuple[bool, Text]

    # utilities to write pre hook function
    file_client = self.context.file_client
    logged_in_user = self.context.logged_in_user

    # your business logic for pre_run_hook here
    # return [True, None] if succeeded otherwise return [False, error_message]

def function_name(file_client, logged_in_user):
  context = PreHookContext(file_client, logged_in_user)
  prehook_base = ClassName(context)
  return prehook_base.execute()

To register a prehook, add the type and parameters to the test specification:

"pre_run_hook": {
            "type": "custom",
            "params": {
                "abs_script_path": "absolute path to the python script",
                "method_name": "function_name"
            }
        }

View test results

When you’re running test suites, the flow or solution defined in the test suite is executed. After the execution, the list of defined validations is run against the results of the flow. The flow executes and generates output to the specified output folder.

Note: If output_has_run_id is set to True, the flow or solution output is written to a folder named with a run ID within the flow or solution root folder. If output_has_run_id is set to False, output is written to the output folder specified in the test configuration.

If the test runs and succeeds, a success message along with the statistics of the test is presented in the UI. If the test executes and fails during validations, failure details are presented in the UI.

In both cases, a link to the flow output folder and flow job ID is also added in the test summary view.

In general, the result of a test is written to: <current_test_dir>/results/<timestamp>/.

The output directory contains a test results file with extension .ibtest.json. This JSON-formatted results file contains the overall result of the test with details of the individual validations.

A single Flow successful test result looks like:

{
  "Summary": {
    "Total Tests": 1,
    "Successful Tests": 1,
    "Failed Tests": 0
  },
  "Detailed_Information": {
    "Passed Tests": [
      "test_invoice"
    ],
    "Failed Tests": [],
    "Field Level Details for all Tests": [
      {
        "Test Name": "test_invoice",
        "Fields Accuracy": {
          "Individual Field Accuracy": {
            "Country": 100.0,
            "DOC_TYPE": 100.0,
            "PersonNames": 100.0,
            "__FILENAME": 100.0,
            "__entities": 100.0,
            "__nlp_persons": 100.0,
            "__person_names": 100.0,
            "locations": 100.0
          },
          "Aggregate Field Accuracy": 100.0
        }
      }
    ]
  }
}

A test is marked as passed if all the validations passed.

An unsuccessful single Flow Binary test result looks like:

{
  "Summary": {
    "Total Tests": 1,
    "Successful Tests": 0,
    "Failed Tests": 1
  },
  "Detailed_Information": {
    "Passed Tests": [],
    "Failed Tests": [
      "post-metaflow-with-udf"
    ],
    "Field Level Details for all Tests": [
      {
        "Test Name": "post-metaflow-with-udf",
        "Statistics": [
          {
            "Individual Field Accuracy": {
              "__FILENAME": 100.0,
              "__helper_for_items": 100.0,
              "items": 100.0,
              "payment_type": 100.0,
              "total_paid": 100.0,
              "paystub_type": 100.0,
              "total": 100.0,
              "__helper_for_payment_type": 100.0
            },
            "Aggregate Field Accuracy": 100.0,
            "Field Mismatch Information": []
          },
          {
            "Successful File Comparisons": [
              {
                "Base File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/golden_out-with_post_metaflow_udf/class_output_folders_new.json",
                "Comparison File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out/class_output_folders_new.json"
              }
            ],
            "Failed File Comparisons": [
              {
                "Base File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/golden_out-with_post_metaflow_udf/root_output_folder.txt",
                "Comparison File": "ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out/root_output_folder.txt"
              }
            ]
          }
        ],
        "Errors": [
          {
            "Comparison Errors": [
              "Files not equal:\nGolden path: ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/golden_out-with_post_metaflow_udf/root_output_folder.txt\nOutput path: ib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out/root_output_folder.txt\nOutput content:\nib_eng/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out\nGolden content:\nib_eng_dne/tests/fs/Test Drive/tests/golden-dataset-tests/MetaflowTests/data/grocery_receipts/out"
            ]
          }
        ]
      }
    ]
  }
}