Kubernetes Scheduling in Python

The topic of scheduling can be quite daunting. It is a full area of research in Computer Science.

At the end of the day scheduling requests is about looking at who is waiting and finding a plan to serve those requests. Generally you want to reduce the time to serve, and use the most appropriate resources to do so.

Kubernetes is no different. Kubernetes schedules Pods (groups of co-located containers) onto nodes of your cluster. That means that one of the core components of the Kubernetes head node is a scheduler.

To demystify further the scheduling process, we can approximate it as Kubernetes needing to look at all the nodes that are in the cluster, filter out the ones that are not a good match to run Pods that are in Pending state, and then pick one.

In more complex terms, the scheduler will run a set of predicates and then run some some priority functions. The end result is a list of ranked nodes that can run the workload (i.e the Pod), the Pod ends up on the node with the highest rank.

This filtering and ranking is nothing new, and can be found in 30 years old batch processing systems.

A Pod with a non-default scheduler

Let’s dig deeper and start a Pod that specifies a non-default scheduler. This can be done in a Pod manifest using schedulerName in the Pod Spec. In the gist below we specify a scheduler named foobar. Once you create this Pod, since Kubernetes does not know this scheduler, the Pod will remain in Pending state and never get scheduled until, we somehow assign a node to this Pod.

A Kubernetes scheduler in Python

While we can run some very advanced functions to determine the list of available nodes who can serve a Pod request, and then run some equally advanced functions to rank nodes that can serve said Pod request, we can also pick a node at random. Random selection is not as bad as it sounds and in some cases can be a very good compromise since it has very little computational cost.

To implement a scheduler, we run a watch on the Pods endpoint. Every time there is a change in the set of Pods in the system, we get a notification. When a Pod is in Pending state and has specified our scheduler name, we call our random scheduler.

To do random scheduling, we just build a list of available nodes in our system and just pick one at random.

To schedule the Pod on our random node, we create a Binding object. This is basically a POST that attaches an object which describes the target node.

Thanks to Ian Lewis for the tip and a more advanced blog.

The gist below use the Kubernetes Python client to implement this simple scheduling.

Now, stick this Python script in a container, run it in a Pod and you have a custom scheduler for Kubernetes.