Performing upgrades

From the Deployment Manager Upgrades tab you can perform version upgrades for your Instabase deployment.

How version upgrades work with Deployment Manager

When a new platform release is available, you can use the Deployment Manager Upgrades tab to upgrade your deployment to the new release version. You’ll upload the new set of base configuration files for the release version along with any required default patches. After uploading and validating your configuration files, you can review all configuration changes, including seeing a list of which objects will be deleted.

During the upgrade process, Deployment Manager handles reapplying all patches applied to your previous base configurations as well as publishing your new materialized configurations to your Kubernetes clusters. The upgrade process also involves turning down your deployment to allow for any database migration, and then bringing it back online with the updated configurations. This means you can expect some downtime during an upgrade; at any time, you can monitor the state of your deployment from the infra dashboard and the progress of the upgrade from the Upgrades tab. During this time, we advise not making any changes on the Configs tab unless otherwise directed.

After your deployment is brought back online, it then undergoes a series of post-install actions, including actions designed to test the performance and functionality of your deployment post-upgrade.

Before you complete the upgrade process, you have the opportunity to test the performance of your deployment while running the new release version. If you observe any issues, you can roll back the upgrade before completing the upgrade process. If your testing is successful, you’re ready to confirm and complete the upgrade.

Before upgrading

Before starting the upgrade process:

• Review the release notes for the release. The deployment guide section of the release notes lists steps you must complete before and after upgrading.

• Obtain the release bundle for the release, typically called installation.zip. The bundle includes the base configurations and default patches for the release, and, optionally, any custom network policies.

  Info

  The release version of your new base configurations file must be greater than the release version of your current base configurations. You cannot downgrade to a previous release through Deployment Manager.

• Create a test plan for how to assess the effect of the upgrade on your deployment. Examples of activities to perform as part of testing include running a flow, clicking through the UI, training a model, or running any tests set up in the Test Runner app.

• (Optional) If you have any custom patches that must be applied during the upgrade, add them to the default patches file (default_patches.zip) found in the installation.zip file. This step of adding patches to default_patches.zip is only required if you have additional custom patches that must be applied during the upgrade. Any previously applied patches are reapplied by Deployment Manager as part of the upgrade.

  Note

  During the upgrade, you must upload the release’s default_patches.zip file, regardless of whether you’ve added additional custom patches or not. default_patches.zip contains required patches for a given release.

  1. Unzip the installation.zip file.

  2. Unzip the default_patches.zip file in the installation folder.

  3. Add your custom patch files to the default_patches folder.

  4. Select all files in the default_patches folder and compress them, creating a new .zip file of patches. You can rename the .zip file, but there are no file name requirements. This new .zip file is what you pass in the Upload configs step.

• (Optional) If network policies are enabled on your cluster, apply the latest Deployment Manager network policies file (control-plane-np.yml) included in your release bundle:

  Note

  While optional, this step should be done before applying the latest Deployment Manager configuration file.

  1. Unzip the installation.zip file for the new release.

  2. On the command line, navigate to the unzipped installation folder.

  3. Apply the new Deployment Manager network policies file contained within by running the following command: kubectl -n $IB_NS apply -f control-plane/control-plane-np.yml, where $IB_NS is your Instabase namespace.

• Ensure you’re running an up-to-date and compatible version of Deployment Manager. Your Deployment Manager version, which can be found on the Version tab of Deployment Manager, must match the release version you are upgrading to. Update Deployment Manager if needed.

• If upgrading from release 22.08 or earlier to release 23.04 or later, before starting the upgrade process, you must change the default value of the ENABLE_CONTROL_PLANE_UPGRADES_ROLLBACK variable to false. If you don’t, the upgrade will fail. See the deployment guide section of the target release’s release notes for instructions.

Upgrading your deployment

When you’re ready to begin the upgrade process:

1. Open Deployment Manager by navigating to {{YOUR-INSTABASE-URL}}/deployment-manager/ and log in with your admin credentials.

   Note

   For upgrades, we recommend navigating directly to the Deployment Manager URL and logging in with your Deployment Manager admin credentials. Accessing Deployment Manager through the Instabase desktop instead might result in you being logged out of Deployment Manager during the upgrade process.

2. Select the Upgrades tab.

3. At the Welcome step, click Start Upgrade.

Preflight Checks

Preflight checks validate aspects of your infrastructure to assess if the upgrade can proceed. The following checks are available on the Preflight Checks step:

• Service accounts check: This check confirms the service account used by the Instabase observability monitoring services exists in the Kubernetes cluster.

• Role-based access control (RBAC) check: The RBAC check reports back any inconsistencies between permissions the Deployment Manager service account requires and permissions it has been granted. A success message indicates the Kubernetes RBAC resource associated with the Deployment Manager service account matches the permissions Deployment Manager requires. A failure message shows the expected RBAC configuration alongside the configuration found in the cluster.

To complete the Preflight Checks step:

1. Click Run Preflight Checks.

2. Review the results of each check.

   • If each check reports back a successful result, you can proceed with the upgrade.
   • If any check reports a failure, review the error message for a description of the failure. Make any required changes, then click Retry.
   Note

   If one or more preflight checks fail and you want to continue the upgrade without resolving, click Force Override, then click Confirm.

3. Click Continue.

Upload Configs

After completing preflight checks, you proceed to the Upload Configs step. Here, you upload the base configurations and default patches for the new release version.

This step also includes a Select Resourcing Size field, where you can optionally update the Instabase-suggested replica count by selecting a preset cluster size. If you select a new cluster size, ensure the option selected meets but does not exceed your Kubernetes cluster size. After upgrading, you can manually adjust the Instabase cluster size configuration using patches to more closely match your Kubernetes cluster size. If you are unsure of your cluster sizing, connect with your customer success manager for guidance or select the No Change option to maintain your current settings.

To complete the Upload Configs step:

1. Upload the base configurations file:

   1. Under Upload Base_Configs.zip, click Upload.

   2. Select your base configurations file from the file explorer.

2. Upload the default patches file:

   1. Under Upload Default Patches, click Upload.

   2. Select your default patches file from the file explorer.

3. From the Select Resourcing Size field, select an option:

   • No change: Do not modify the current cluster size setting. (Recommended for customers with custom cluster size or custom resourcing.)

   • Development: Apply the same cluster sizing as the Standard option, and all non-zero replica counts are set to 1. This setting is used for temporary testing only.

   • Standard: 64 CPU cores/256 GB RAM

   • Large: 128 CPU cores/512 GB RAM

   • xLarge: 256 CPU cores/1024 GB RAM

4. Click Validate. Deployment Manager validates that all files are correctly formatted and can be applied. Validation can take upwards of ten minutes; do not navigate away from the Upgrades tab before it completes.

   Note

   If any validation errors occur, an error message displays indicating the problem, such as a required object not being found in the base configurations file. Connect with your customer success manager for guidance, if needed.

   If your customer success manager advises that you can continue the upgrade without resolving, click Force Continue, then click Confirm.

5. Click Continue.

Confirm Changes

After uploading configurations, Deployment Manager generates a detailed list of all configuration changes to be applied and displays them at the Confirm Changes step. The tab’s left pane contains the Config Changes list, which is separated into the following:

• Included in upgrade: Lists all objects that will be included in the upgrade, along with a summary of modifications to the object. Select an object to view a detailed file diff.

• Configs to delete: Lists all objects that will be deleted, unless manually added to the Included in upgrade list. Deprecated objects are automatically added to the Configs to delete list.

To complete the Confirm Changes step:

1. Review all items in the Config Changes list to understand what changes will occur with the upgrade. To view a detailed file diff for an object, select it.

2. Optionally add objects in the Configs to delete list to the Included in upgrade list.

   1. Locate the object in the Configs to delete list.

   2. Click the + (plus) icon next to its name. The object moves to the Included in upgrade list.

   3. To return an object to the Configs to delete list, click the - (minus) icon next to its name.

3. Click Continue.

Upgrade

After confirming the expected changes, you’re ready to apply those changes at the Upgrade step. As the upgrade proceeds, the Upgrade Steps list provides a visual indicator of what step of the upgrade process your deployment is in. At the same time, the right pane provides live logs of the upgrade process.

Throughout the upgrade, Pause, Rollback, and Abandon Upgrade buttons are visible.

• Pausing an upgrade: You can pause and resume an upgrade at any point without canceling the upgrade process. After clicking Pause, the in-progress step completes then the upgrade process is paused.

• Rolling back an upgrade: You can roll back an upgrade up until the Confirm Completion step, at which point you must either confirm your readiness to complete the upgrade or initiate a rollback. After confirming that you wish to complete the upgrade, you can no longer roll it back, so be sure to test the effect of the upgrade on your deployment before confirming.

• Abandoning an upgrade: Abandoning an upgrade that has neither been completed nor rolled back puts your deployment in an unusable state. Do not click Abandon Upgrade unless directed to by Instabase support as part of troubleshooting steps.

To complete the Upgrade step:

1. Click Start. Logs populate as Deployment Manager moves through the Upgrade Steps list. After logs have begun to display, you can navigate away from the Upgrades tab if needed.

   Info

   If you encounter a failed or stalled step during the upgrade process, review the troubleshooting section for guidance.

   Tip

   You can filter the upgrade logs using the Filters button. Logs can be filtered by stage (corresponding to the steps of the Upgrade Steps list) or by log level, such as error, warning, or success.

2. At the Confirm Completion step, you have the opportunity to test the upgrade before completing the upgrade process. Open your deployment in a separate tab or window and test the effect of the upgrade on your deployment according to your test plan.

   Warning

   If you observe any issues with your deployment during testing, you can roll back the upgrade and not complete the upgrade process. If you choose to complete the upgrade, you cannot later roll it back. See the troubleshooting section for instructions on initiating a rollback.

3. If testing is successful, return to the Deployment Manager Upgrades tab and click Complete Upgrade.

4. After clicking Complete Upgrade, Deployment Manager completes the upgrade process and displays a success message at the Complete step.

5. Click Finish to exit the upgrade process.

Troubleshooting

Errors and failed steps during the Upgrade step

You might encounter errors during the Upgrade step, indicated by a Failed log in the logs pane and an accompanying red x icon next to the failed step in the Upgrade Steps list. Review the error logs for any actionable insights and connect with your customer success manager for guidance on making any required changes. To retry a failed step, Retry Step.

Rolling back an upgrade

If a step continues to fail after clicking Retry Step, click Rollback to exit out of the upgrade process and roll back any configuration changes. Rolling back the upgrade can take some time; you can follow the progress of the rollback from the Upgrades tab.

• After the rollback completes: Click Retry to retry the upgrade, or click Cancel until you exit out of the upgrade flow.

• If the rollback fails: Contact Instabase support for further assistance and do not click Abandon Upgrade unless advised to.

Stalled steps or rollbacks

If a step or rollback is no longer progressing or producing logs, and no activity is seen for more than ten minutes, you can restart Deployment Manager.

1. If you haven’t already, click Rollback to initiate the rollback.

2. Restart Deployment Manager by running the following kubectl command: $ kubectl rollout restart deployment deployment-control-plane -n {{YOUR-IB-NAMESPACE}}

When Deployment Manager restarts, the rollback of the previously stalled step is initiated and you can retry the upgrade from the Upgrades tab.