About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Hello,
Wells Fargo needs this feature to restrict CPU utilization by pyspark drivers within a certain limits. We have observed that when a data scientist use pure python code on pyspark kernel, the workload runs only on the driver node. This can potentially cause resource issue on the node and impact other drivers running on the same node. By enabling Cgroup feature on driver node, we can restrict resource utilized by a driver within the node and hence avoid the issue I mentioned above.