HA functionality for Datastage Cartridge

Background

•In general, the best practice to ensure high availability for OpenShift cluster is to spread the masters and workers across three AZ’s in a cloud. But for DataStage, it only makes sense if the platform allows multiple conductors spread across AZ’s.

•Even if the platform allows spreading conductors, the job would not perform well if the PXRuntime assigns pods ($pod placeholder in APT_CONFIG_FILE) for a job in different AZ’s. AZ’s are physically isolated and the network throughput between AZ’s is in general significantly lower than within an AZ. Cloud vendors provide options to improve the throughput but it will not achieve same throughput as within AZ. If this happens, the job performance would be bottlenecked by the pod in different AZ and would not scale.

•This is critical as the expected SLA for system availability is getting to 99.99 or even to 99.999 recently for many customers. Waiting for pods coming back up doesn’t meet this requirement. This multi-zone HA architecture achieves active-active HA and guarantees the expected availability.

Requests

•Add a new function to make PXRuntime aware of AZ to place PXRuntime and Compute pods evenly in all AZ’s.

•Allow users choose pods that are running in the AZ where the connected conductor is running. Users wouldn’t know which conductor they will be connected to so that users cannot specify the AZ. Instead, need an option to assign pods in the same AZ in which users connected to the conductor.

•It should also allow ‘preference’ and ‘requirement’ options. If pods are not available in the desired AZ, there may be cases you do or don’t want to run the job.