Comments on: Autoscaling on Kubernetes with KEDA and Kafka

By: piotr.minkowski

piotr.minkowski — Thu, 07 Mar 2024 14:53:55 +0000

In reply to Alex. Well, defining multiple autoscalers may result in conflicts, e.g. for one topic it will scale up the app, while for the second topic it will do the scale down.So, it is not the best idea.

By: Alex

Alex — Thu, 07 Mar 2024 10:59:20 +0000

Good example, thanks.
My question is about the situation when consumer application is listening to 10 different topics. What is the correct approach to configure deployment? Should I create ScaledObjecs for each topic or I can combine the logic for all topics somehow, let say default config for all existing topics.

By: piotr.minkowski

piotr.minkowski — Wed, 03 May 2023 23:19:42 +0000

In reply to Sanyam. rebalance. yes

By: Sanyam

Sanyam — Wed, 03 May 2023 18:13:40 +0000

Let’s say I have 6 partitions and 2 consumers (each consumer listening to 3 partitions). If lag increases to 6 on any of the partition and let’s assume using above KEDA, it increases consumers from 2 to 3. In that case will Kafka automatically re-map partitions (2 partitions per consumer) ?

By: piotr.minkowski

piotr.minkowski — Wed, 01 Feb 2023 22:18:16 +0000

In reply to Tim. In my opinion, two different things are described there: the nature of how KEDA and HPA work and the nature of Kafka's rebalancing. Yes, with Kafka scaling costs, due to the rebalancing mechanism. Currently, with KEDA we can just analyze the current consumer lag, and based on that value perform autoscaling.

By: Tim

Tim — Wed, 01 Feb 2023 00:57:13 +0000

I think rebalancing cost matters. Do you have any thoughts on that? ref https://github.com/kedacore/keda/issues/2401

By: piotr.minkowski

piotr.minkowski — Thu, 08 Dec 2022 09:21:37 +0000

In reply to Marek.

Thanks! As you see I used env for that `export VERSION=$(curl -s https://api.github.com/repos/vectorizedio/redpanda/releases/latest | jq -r .tag_name)` and set it on the `helm install` command

By: piotr.minkowski

piotr.minkowski — Thu, 08 Dec 2022 09:19:54 +0000

In reply to adry. Which commands do not work?

By: Marek

Marek — Fri, 04 Mar 2022 12:28:41 +0000

Hello Piotr! Great work. Thank you for sharing! I’ve been doing similar KEDA + Kafka experiments just based on [kedacore samples](https://github.com/kedacore/sample-python-kafka-azure-function) with custom consumer and producer written in Python, now thanks to your article I’ve also tried with Redpanda and it too works great.
BTW I needed to add “?ref=$VERSION” in `kubectl apply \
-k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION` [found it in the redpanda docs](https://docs.redpanda.com/docs/quickstart/kubernetes-qs-cloud/). Cheers

By: adry

adry — Wed, 02 Mar 2022 09:58:32 +0000

nice post, but too bad that the example and command lines don’t work anymore