Posts on Ankit Paudel

Cloud-native monitoring with Lightweight Prometheus + Thanos on GKE

Sun, 10 Aug 2025 08:33:09 +0000

Preface
#

You require a cheap monitoring solution that is cloud agnostic.

Problem
#

Google’s manged prometheus costs a lot; so self hosting prometheus+thanos to get around it.
- As the required metrics to properly observe our application increases, the requirement to selfhost the monitoring platform to curb the cloud costs makes more sense, rather than using managed monitoring service.
- This opens up self hosting and maintaining Prometheus+Thanos+GCS stack for all the internal observability needs.
- low cost of GCS + infrequent access to older metrics data makes Thanos’s requirement more sensible.

The Solution
#

Store less data in prometheus’s tsdb while maintaining historical data in gcs for low cost query.
- Setup proper retention period as business requirement to reduce the latency of metrics query.

Prometheus
#

A stable and mature TSDB, designed to record and host application metrics.
Pros:
- Easy to get started with grafana for basic monitoring needs.
- Low barrier to entry if the requirement is not high availability.
Cons:
- Hard to scale if HA is explicity required.
- Multiple Prometheus instances work independently, it does not have built in feature for master-slave replication and HA deployment like postgres and other stateful applications.

Thanos
#

Solution built on top of Prometheus to solve HA and scalibility needs.
Requires active connection to Prometheus instance to function.
Manages multiple Prometheus instances and handles deduplication, downsampling and other higher order features that Prometheus does not support natively.

High level proposed architecture
#

Store as less of data as required in memory in the prometheus instances.
Configure thanos side car to upload metric data as frequent as possible to gcs or other colder storage as compared to tsdb for low memory and cluster resource requirement.

Possible issues with this approach
#

Chatty communication between monitoring components might incur higher networking bills for multi-region k8s clusters.
Needs manual security considerations to secure grafana and other public endpoints.
Might take up engineering bandwidth: requiring more engineers to maintain it in the long term.

Installation
#

This uses helm-charts by bitnami for thanos and the official Prometheus community maintained helm chart.
Optimized values for helm install of our specific use case:
- Chart used: kube_prometheus_stack and bitnami_thanos_chart.
- Image tags are latest as per the period of this article.
Notes
- Some names like the service endpoints might be changed as per the updation of helm chart and your release names.
- This is before bitnami announced its paid offerings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78


# Prometheus
alertmanager: # not used, in our case.
 enabled: false
coreDns:
 enabled: false
grafana:
 datasources:
 datasources.yaml:
 apiVersion: 1
 datasources:
 - access: proxy
 isDefault: false
 jsonData:
 prometheusType: thanos
 name: prom-default
 type: prometheus
 url: http://thanos-query-frontend.monitoring.svc.cluster.local:9090
 deploymentStrategy:
 type: Recreate # we are using RWO persistence
 persistence:
 enabled: true
 size: 5Gi
kube-state-metrics:
 enabled: true
kubeApiServer:
 enabled: false
kubeControllerManager:
 enabled: false
kubeProxy:
 enabled: false
kubeScheduler:
 enabled: true
kubeStateMetrics:
 enabled: true
kubelet:
 enabled: true
nodeExporter:
 enabled: false
prometheus:
 prometheusSpec:
 containers:
 - env:
 - name: GOGC
 value: "75"
 name: prometheus
 externalLabels:
 cluster: <label-identifying-your-cluster-for-multicluster-setups>
 replica: $(POD_NAME)
 podMonitorSelectorNilUsesHelmValues: false
 replicas: 2
 retention: 3h
 retentionSize: 8GiB
 serviceMonitorSelectorNilUsesHelmValues: false
 storageSpec:
 volumeClaimTemplate:
 spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 10Gi
 storageClassName: premium-rwo # this is specific to GCP, change as you require
 thanos:
 objectStorageConfig:
 existingSecret: # this is to access gcs for metrics storage; for aws, use per documentation of thanos.
 key: objstore.yml
 name: thanos-config
 thanosService:
 enabled: true
prometheus-node-exporter:
 enabled: false # This is covered by the provided metrics of gke.
prometheusOperator:
 admissionWebhooks:
 patch:
 image:
 tag: v1.5.2 
 thanosImage:
 tag: v0.39.2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64


# Thanos
bucket:
 enabled: true
 replicaCount: 1
compactor:
 cronJob:
 activeDeadlineSeconds: 10800
 enabled: true
 failedJobsHistoryLimit: 3
 schedule: 0 */6 * * *
 successfulJobsHistoryLimit: 2
 enabled: true
 extraFlags:
 - --delete-delay=12h
 - --deduplication.replica-label=replica
 - --deduplication.replica-label=prometheus_replica
 persistence:
 enabled: false
 retentionResolution1h: 0d
 retentionResolution5m: 180d
 retentionResolutionRaw: 30d
existingObjstoreSecret: thanos-config
existingServiceMonitor: false
metrics:
 enabled: true
 serviceMonitor:
 enabled: true
objectStorageConfig:
 existingSecret:
 key: objstore.yml
 name: thanos-config
query:
 dnsDiscovery:
 sidecarsNamespace: monitoring
 sidecarsService: prometheus-stack-kube-prom-thanos-discovery
 enabled: true
 extraFlags:
 - --query.auto-downsampling
 - --query.partial-response
 - --query.timeout=2m
 replicaCount: 2
queryFrontend:
 enabled: true
 extraFlags:
 - --query-range.split-interval=24h
 - --query-range.max-retries-per-request=3
 - --query-frontend.compress-responses
 - --query-range.partial-response
 replicaCount: 2
ruler:
 enabled: false
storegateway:
 enabled: true
 extraFlags:
 - --index-cache-size=512MB
 - --chunk-pool-size=512MB
 - --store.grpc.series-max-concurrency=10
 - --block-sync-concurrency=3
 - --store.enable-index-header-lazy-reader
 persistence:
 enabled: true
 size: 5Gi
 storageClass: premium-rwo
 replicaCount: 2

Final notes and recommendations
#

Setup proper alerting for failed compactors and other important metrics like consistent storage-gateway cache misses.
- Use proper memory requirements for storage gateway and querier for faster queries as business requirements.
This setup does not factor in the cardinality constraints of your specific usecase.
- You might be a startup just starting with low enough unique users or a B2B product with low enough user-base that can get away with taking in user-id as a label in prom-metrics, but this might blow up your prom instance if your userbase is large enough.
Keep monitoring the resource usage and latency of thanos and prom components for few days/weeks for resource exhaustion and your needs. If your prom starts getting OOM failures, you might need to reconfigure this setup to use remote-write instead of sidecar based pattern.
- Remote-Write is a little complicated but is a good trade-off if you have a quite a few components that queries for latest metrics and have enough engineering bandwidth to maintain it in the long term.
Use a separate Node pool for monitoring components like prometheus and stateful components like databases and distributed caches for resource isolation and to reduce failure domains.

Mongodb triggers and change streams

Sun, 16 Feb 2025 03:32:36 +0545

Intro on how to use triggers and change-streams in mongo

Rwx Volume in k8s: Adventure with Longhorn

Thu, 06 Feb 2025 00:44:36 +0545

Intro to solutions for RWX volume in k8s cluster. The experience with using longhorn for the job.

Guide to worker types in gunicorn

Sun, 22 Dec 2024 11:41:14 +0000

Python is single threaded
#

Python by design is single threaded. It can only run one thread at once due to GIL.
This makes making web server which serves a lot requests at once a bit complicated.
Thread based concurrency is good enough for most of the applications but its control block is heavier than traditional async coroutines.
This led to necessity of eventloop based concurrency which can share almost eveything of main python thread except the request data.
This would make coroutines very light with fast context switches.

Async in python before asyncio
#

To handle thousands of request at once, there were a few implementations that made python work somewhat like async runtime.
Gevent and eventlets being the major one.
There was also a framework called tornado which had its own implementation of coroutine based server which is not that popular now.
First eventlet was released to tackle this problem with its pure python implementation but later gevent came to picture with its blazing fast speed as it was implemented in cython itself.
They work by monkeypatching specific functions in python that does I/O operations. They replace those functions with their own version so that they emit some information to the eventloop manager before going to I/O wait state. This allowed for another function to run till it entered same kind of I/O wait state.
They were built upon the concept that a python process should never be sitting idle for I/O wait.

Role of Gunicorn <-> workers
#

Gunicorn is just a process manager, that makes sure that “worker” process never stop processing the requests.
It has two distinct section of architecture; one being “arbiter” and the other being “worker”.
Arbiter does the job of manging worker processes while the worker process does the job of communicating with arbiter along with starting/monitoring the application execution context that processes request as per worker type defined.
The worker type defines how the server application gets run. The default sync worker process one request at a single time per process.
The workers that comes packaged in gunicorn are process-worker, threads, gevent, eventlet and tornado. But you will be able to find other third party workers like uvicorn workers that supports fastapi like async frameworks too.

Sync vs Async workers in gunicorn
#

Sync workers are those workers which can handle requests sychronously i.e. one request at a time.

1
2
3


# both the gunicorn instances below can process 2 requests at once. The first one uses threads to do so while the second one uses python processes.
gunicorn test:app --threads 2 --timeout 5
gunicorn test:app --workers 2 --timeout 5

Async workers, as the name suggests, can handle multiple requests at once. So there will theoritically be good resource utilization for i/o bound task.
Gunicorn provides us with gevent, eventlet and tornado workers out of the box.
Gevent is the most widely used due to its compatibility and speed as compared to tornado and eventlet.

When to use which workers
#

When to use sync default workers

Introduction to WSGI and gunicorn

Sun, 22 Dec 2024 11:41:14 +0000

Webserver in Python
#

Originally, python was not made to develop webservers, but due to its simplicity and very easy learning curve, along time a few web frameworks started popping.
Different frameworks/tools had their own serving method that tried to alleviate the fact that python was a single threaded application and io calls would block the application from receiving any further requests.
Solution: use threads, multiple python processes and their combination.
This issue led to development of some standard interface that all frameworks can adhere to so that frameworks don’t have to worry about how to manage concurrency and be best on what they do, acting as an easy and fast tool to create a httpserver.

What is WSGI
#

WSGI is the interface that most of the python web frameworks provide for their application to be served without worrying about the specifics of concurrency and load management in high rps environments. At the core, its just set of documents that describe how a compatible web application should be structured, process the request and respond accordingly.
A simple hello world WSGI application looks like this:

1
2
3
4
5
6
7
8
9


def application(environ, start_response):
 method = environ['REQUEST_METHOD']
 print(method)
 path = environ['PATH_INFO']
 print(path)
 status = '200 OK'
 response_headers = [('Content-Type', 'text/plain; charset=utf-8')]
 start_response(status, response_headers)
 return [b'Hello World!']

This application can be ran using gunicorn which is a wsgi server designed to serve these kind of applications. The simplest command to do so would be gunicorn app:application.
So all the flask or Django applications can be accessed as a simple function above. All the ifs are abstracted away in different ways in different frameworks, decorators in flask and functions/methods in django.

Gunicorn
#

As per gunicorn’s documentation, gunicorn is inspired from ruby’s unicorn project which used to manage multiple ruby web-server processes.
It essentially is a process manager that kills/restarts hung processes that is supposed to listen to requests.
When gunicorn starts, it starts its main process called “arbiter” which listens to some port. Then it gets forked into multiple “worker” processes that inherit the socket that was opened during the startup of arbiter thus, multiple processes would be listening to same os port.
The “worker” that I mentioned previously is a python process that runs the WSGI compatible web server.
As multiple workers are listening to a single port, the role of load-balancing is handed over to kernel. The kernel manages the resource contention between multiple “worker” processes for the opportunity to handle the request.
This disproves the fact that would be plaguing people’s mind assuming that arbiter acts as some sort of reverse proxy like nginx.
The “management” of “worker” processes by arbiter includes things like restarting workers when workers gets hung up.
- “hang up” in this context means that the worker is not able to message arbiter saying its okay for some period of time. This is somewhat like linux’s “watchdog” concept.
- Async workers can send periodic heartbeats but sync workers can’t do that if they are stuck in some I/O operations. In this case, arbiter relies in messages in specific occations like acknowleding the request, returning response and likes.
Sends SIGKILL to timed out workers which gets logged as log somewhat like this: [CRITICAL] WORKER TIMEOUT. This might be due to anything, If your request is too slow that it took more than the timeout specified to response.
There can be different worker types in gunicorn. Gunicorn is very simple, it just does not let worker live without it processing a request.
- The workers can be put into two distinct buckets that explain how they process python requests. Sync and Async workers. Sync workers can handle one request at a time while async ones can handle multiple requests at the same time.

Timeout in gunicorn
#

As gunicorn is a process manager(not request handler), the timeout you give during the gunicorn’s startup is timeout of a worker process.
This timeout is equal to the timeout of request when you are using sync workers but can be totally different if you are using async workers.
- Value in --timeout does not mean the max time a client can wait for response.
Following simple flask program demonstrates this behaviour of gunicorn:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


import time
from flask import Flask

app=Flask("__name__")

@app.get("/slow")
def test():
 time.sleep(10)
 return "slept for 10s"

@app.get("/fast")
def test1():
 time.sleep(1)
 return "return super fast, in 1 second"

When you run the program using this gunicorn command: gunicorn test:app --threads 2 --timeout 5 which spawns multipel execution context per worker process. Then you send a request to the /slow endpoint, you get your response after 10 secs. The other thread was alive thus the worker process was able to prove that its alive during the whole time.
Contrary to above, if you set the thread count to 1 or use sync worker, you will get the worker timeout log and 5xx would be returned to the client.
The responsibility of timeout is soly given to the application side where the developer must set proper timeout for each request.
- This can be done in various ways I am giving only specific outlines to a few ways because this topic in itself is worthy of a whole article.
  - Setting every socket connection’s alive time globally in each process. This can be very easy fix but might not work in scenarios like http/2 because they use single connection for multiple http requests.
  - Making a timeout decorator and using them in each of the route handler. This is very common in flask like application where you define the route handler using decorators.
  - Use Middleware. This can be applicable in express-like frameworks or django where middlewares can ensure that the request can timeout after certain seconds.
  - Use of reverse proxy like nginx. As they create a new http connection between themself and the server application, nginx can track time taken by each request and close connection to server after some time. Nginx will normally return 504 to the clients if it can’t reach the application server within the time defined.

Request body aware dynamic routing in ingress nginx using custom plugins.

Wed, 10 Jul 2024 00:44:36 +0545

How do we make ingress nginx intelligent enough to route to different backend not just based on url path match but on the information that request body itself? What if request routing itself requires database access to find the right backend?

Managing on demand ephemeral microservice development environment in a single k8s cluster.

Sun, 23 Jun 2024 09:30:05 +0545

How we made debugging on cloud easier without costing a fortune.
#

Preface
#
Any organization will require good and flexible environment for every developer to test their code before merging it to the main branch.
This is normally not required for small-medium monolith application but if the application is too big or consists of a lot microservices and we want to test integration between mutliple microservices. This does not occur often but occurs when you have a feature to implement that will span changes across a lot of microservices. There are other method to test these like writing unit tests for each api of microservice so that it adhers to certain schema and changing the schema as per the changes done and the expected value.

Debugging a python program inside of a container

Sat, 22 Jun 2024 10:45:58 +0545

Debugging python inside of a docker container
#

Premises
#

Lets say you are working on some legacy python project which has very obscure and obsolete dependencies and you now want to debug it. First lets pray to god that, this day does not come to anybody but if you are working on some project without standard practices on package management, this kind of situation is not uncommon.

Posts on Ankit Paudel

Cloud-native monitoring with Lightweight Prometheus + Thanos on GKE

Preface #

Problem #

The Solution #

Prometheus #

Thanos #

High level proposed architecture #

Possible issues with this approach #

Installation #

Final notes and recommendations #

Mongodb triggers and change streams

Rwx Volume in k8s: Adventure with Longhorn

Guide to worker types in gunicorn

Python is single threaded #

Async in python before asyncio #

Role of Gunicorn <-> workers #

Sync vs Async workers in gunicorn #

When to use which workers #

Introduction to WSGI and gunicorn

Webserver in Python #

What is WSGI #

Gunicorn #

Timeout in gunicorn #

Request body aware dynamic routing in ingress nginx using custom plugins.

Managing on demand ephemeral microservice development environment in a single k8s cluster.

How we made debugging on cloud easier without costing a fortune. #

Preface #

Debugging a python program inside of a container

Debugging python inside of a docker container #

Premises #

Preface
#

Problem
#

The Solution
#

Prometheus
#

Thanos
#

High level proposed architecture
#

Possible issues with this approach
#

Installation
#

Final notes and recommendations
#

Python is single threaded
#

Async in python before asyncio
#

Role of Gunicorn <-> workers
#

Sync vs Async workers in gunicorn
#

When to use which workers
#

Webserver in Python
#

What is WSGI
#

Gunicorn
#

Timeout in gunicorn
#

How we made debugging on cloud easier without costing a fortune.
#

Preface
#

Debugging python inside of a docker container
#

Premises
#