Preface # You require a cheap monitoring solution that is cloud agnostic. Problem # Google’s manged prometheus costs a lot; so self hosting prometheus+thanos to get around it. As the required metrics to properly observe our application increases, the requirement to selfhost the monitoring platform to curb the cloud costs makes more sense, rather than using managed monitoring service. This opens up self hosting and maintaining Prometheus+Thanos+GCS stack for all the internal observability needs. low cost of GCS + infrequent access to older metrics data makes Thanos’s requirement more sensible. The Solution # Store less data in prometheus’s tsdb while maintaining historical data in gcs for low cost query. Setup proper retention period as business requirement to reduce the latency of metrics query. Prometheus # A stable and mature TSDB, designed to record and host application metrics. Pros: Easy to get started with grafana for basic monitoring needs. Low barrier to entry if the requirement is not high availability. Cons: Hard to scale if HA is explicity required. Multiple Prometheus instances work independently, it does not have built in feature for master-slave replication and HA deployment like postgres and other stateful applications. Thanos # Solution built on top of Prometheus to solve HA and scalibility needs. Requires active connection to Prometheus instance to function. Manages multiple Prometheus instances and handles deduplication, downsampling and other higher order features that Prometheus does not support natively. High level proposed architecture # Store as less of data as required in memory in the prometheus instances. Configure thanos side car to upload metric data as frequent as possible to gcs or other colder storage as compared to tsdb for low memory and cluster resource requirement. Possible issues with this approach # Chatty communication between monitoring components might incur higher networking bills for multi-region k8s clusters. Needs manual security considerations to secure grafana and other public endpoints. Might take up engineering bandwidth: requiring more engineers to maintain it in the long term. Installation # This uses helm-charts by bitnami for thanos and the official Prometheus community maintained helm chart. Optimized values for helm install of our specific use case: Chart used: kube_prometheus_stack and bitnami_thanos_chart. Image tags are latest as per the period of this article. Notes Some names like the service endpoints might be changed as per the updation of helm chart and your release names. This is before bitnami announced its paid offerings. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 # Prometheus alertmanager: # not used, in our case. enabled: false coreDns: enabled: false grafana: datasources: datasources.yaml: apiVersion: 1 datasources: - access: proxy isDefault: false jsonData: prometheusType: thanos name: prom-default type: prometheus url: http://thanos-query-frontend.monitoring.svc.cluster.local:9090 deploymentStrategy: type: Recreate # we are using RWO persistence persistence: enabled: true size: 5Gi kube-state-metrics: enabled: true kubeApiServer: enabled: false kubeControllerManager: enabled: false kubeProxy: enabled: false kubeScheduler: enabled: true kubeStateMetrics: enabled: true kubelet: enabled: true nodeExporter: enabled: false prometheus: prometheusSpec: containers: - env: - name: GOGC value: "75" name: prometheus externalLabels: cluster: <label-identifying-your-cluster-for-multicluster-setups> replica: $(POD_NAME) podMonitorSelectorNilUsesHelmValues: false replicas: 2 retention: 3h retentionSize: 8GiB serviceMonitorSelectorNilUsesHelmValues: false storageSpec: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: premium-rwo # this is specific to GCP, change as you require thanos: objectStorageConfig: existingSecret: # this is to access gcs for metrics storage; for aws, use per documentation of thanos. key: objstore.yml name: thanos-config thanosService: enabled: true prometheus-node-exporter: enabled: false # This is covered by the provided metrics of gke. prometheusOperator: admissionWebhooks: patch: image: tag: v1.5.2 thanosImage: tag: v0.39.2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 # Thanos bucket: enabled: true replicaCount: 1 compactor: cronJob: activeDeadlineSeconds: 10800 enabled: true failedJobsHistoryLimit: 3 schedule: 0 */6 * * * successfulJobsHistoryLimit: 2 enabled: true extraFlags: - --delete-delay=12h - --deduplication.replica-label=replica - --deduplication.replica-label=prometheus_replica persistence: enabled: false retentionResolution1h: 0d retentionResolution5m: 180d retentionResolutionRaw: 30d existingObjstoreSecret: thanos-config existingServiceMonitor: false metrics: enabled: true serviceMonitor: enabled: true objectStorageConfig: existingSecret: key: objstore.yml name: thanos-config query: dnsDiscovery: sidecarsNamespace: monitoring sidecarsService: prometheus-stack-kube-prom-thanos-discovery enabled: true extraFlags: - --query.auto-downsampling - --query.partial-response - --query.timeout=2m replicaCount: 2 queryFrontend: enabled: true extraFlags: - --query-range.split-interval=24h - --query-range.max-retries-per-request=3 - --query-frontend.compress-responses - --query-range.partial-response replicaCount: 2 ruler: enabled: false storegateway: enabled: true extraFlags: - --index-cache-size=512MB - --chunk-pool-size=512MB - --store.grpc.series-max-concurrency=10 - --block-sync-concurrency=3 - --store.enable-index-header-lazy-reader persistence: enabled: true size: 5Gi storageClass: premium-rwo replicaCount: 2 Final notes and recommendations # Setup proper alerting for failed compactors and other important metrics like consistent storage-gateway cache misses. Use proper memory requirements for storage gateway and querier for faster queries as business requirements. This setup does not factor in the cardinality constraints of your specific usecase. You might be a startup just starting with low enough unique users or a B2B product with low enough user-base that can get away with taking in user-id as a label in prom-metrics, but this might blow up your prom instance if your userbase is large enough. Keep monitoring the resource usage and latency of thanos and prom components for few days/weeks for resource exhaustion and your needs. If your prom starts getting OOM failures, you might need to reconfigure this setup to use remote-write instead of sidecar based pattern. Remote-Write is a little complicated but is a good trade-off if you have a quite a few components that queries for latest metrics and have enough engineering bandwidth to maintain it in the long term. Use a separate Node pool for monitoring components like prometheus and stateful components like databases and distributed caches for resource isolation and to reduce failure domains.
Webserver in Python # Originally, python was not made to develop webservers, but due to its simplicity and very easy learning curve, along time a few web frameworks started popping. Different frameworks/tools had their own serving method that tried to alleviate the fact that python was a single threaded application and io calls would block the application from receiving any further requests. Solution: use threads, multiple python processes and their combination. This issue led to development of some standard interface that all frameworks can adhere to so that frameworks don’t have to worry about how to manage concurrency and be best on what they do, acting as an easy and fast tool to create a httpserver. What is WSGI # WSGI is the interface that most of the python web frameworks provide for their application to be served without worrying about the specifics of concurrency and load management in high rps environments. At the core, its just set of documents that describe how a compatible web application should be structured, process the request and respond accordingly. A simple hello world WSGI application looks like this: 1 2 3 4 5 6 7 8 9 def application(environ, start_response): method = environ['REQUEST_METHOD'] print(method) path = environ['PATH_INFO'] print(path) status = '200 OK' response_headers = [('Content-Type', 'text/plain; charset=utf-8')] start_response(status, response_headers) return [b'Hello World!'] This application can be ran using gunicorn which is a wsgi server designed to serve these kind of applications. The simplest command to do so would be gunicorn app:application. So all the flask or Django applications can be accessed as a simple function above. All the ifs are abstracted away in different ways in different frameworks, decorators in flask and functions/methods in django. Gunicorn # As per gunicorn’s documentation, gunicorn is inspired from ruby’s unicorn project which used to manage multiple ruby web-server processes. It essentially is a process manager that kills/restarts hung processes that is supposed to listen to requests. When gunicorn starts, it starts its main process called “arbiter” which listens to some port. Then it gets forked into multiple “worker” processes that inherit the socket that was opened during the startup of arbiter thus, multiple processes would be listening to same os port. The “worker” that I mentioned previously is a python process that runs the WSGI compatible web server. As multiple workers are listening to a single port, the role of load-balancing is handed over to kernel. The kernel manages the resource contention between multiple “worker” processes for the opportunity to handle the request. This disproves the fact that would be plaguing people’s mind assuming that arbiter acts as some sort of reverse proxy like nginx. The “management” of “worker” processes by arbiter includes things like restarting workers when workers gets hung up. “hang up” in this context means that the worker is not able to message arbiter saying its okay for some period of time. This is somewhat like linux’s “watchdog” concept. Async workers can send periodic heartbeats but sync workers can’t do that if they are stuck in some I/O operations. In this case, arbiter relies in messages in specific occations like acknowleding the request, returning response and likes. Sends SIGKILL to timed out workers which gets logged as log somewhat like this: [CRITICAL] WORKER TIMEOUT. This might be due to anything, If your request is too slow that it took more than the timeout specified to response. There can be different worker types in gunicorn. Gunicorn is very simple, it just does not let worker live without it processing a request. The workers can be put into two distinct buckets that explain how they process python requests. Sync and Async workers. Sync workers can handle one request at a time while async ones can handle multiple requests at the same time. Timeout in gunicorn # As gunicorn is a process manager(not request handler), the timeout you give during the gunicorn’s startup is timeout of a worker process. This timeout is equal to the timeout of request when you are using sync workers but can be totally different if you are using async workers. Value in --timeout does not mean the max time a client can wait for response. Following simple flask program demonstrates this behaviour of gunicorn: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 import time from flask import Flask app=Flask("__name__") @app.get("/slow") def test(): time.sleep(10) return "slept for 10s" @app.get("/fast") def test1(): time.sleep(1) return "return super fast, in 1 second" When you run the program using this gunicorn command: gunicorn test:app --threads 2 --timeout 5 which spawns multipel execution context per worker process. Then you send a request to the /slow endpoint, you get your response after 10 secs. The other thread was alive thus the worker process was able to prove that its alive during the whole time. Contrary to above, if you set the thread count to 1 or use sync worker, you will get the worker timeout log and 5xx would be returned to the client. The responsibility of timeout is soly given to the application side where the developer must set proper timeout for each request. This can be done in various ways I am giving only specific outlines to a few ways because this topic in itself is worthy of a whole article. Setting every socket connection’s alive time globally in each process. This can be very easy fix but might not work in scenarios like http/2 because they use single connection for multiple http requests. Making a timeout decorator and using them in each of the route handler. This is very common in flask like application where you define the route handler using decorators. Use Middleware. This can be applicable in express-like frameworks or django where middlewares can ensure that the request can timeout after certain seconds. Use of reverse proxy like nginx. As they create a new http connection between themself and the server application, nginx can track time taken by each request and close connection to server after some time. Nginx will normally return 504 to the clients if it can’t reach the application server within the time defined.
Python is single threaded # Python by design is single threaded. It can only run one thread at once due to GIL. This makes making web server which serves a lot requests at once a bit complicated. Thread based concurrency is good enough for most of the applications but its control block is heavier than traditional async coroutines. This led to necessity of eventloop based concurrency which can share almost eveything of main python thread except the request data. This would make coroutines very light with fast context switches. Async in python before asyncio # To handle thousands of request at once, there were a few implementations that made python work somewhat like async runtime. Gevent and eventlets being the major one. There was also a framework called tornado which had its own implementation of coroutine based server which is not that popular now. First eventlet was released to tackle this problem with its pure python implementation but later gevent came to picture with its blazing fast speed as it was implemented in cython itself. They work by monkeypatching specific functions in python that does I/O operations. They replace those functions with their own version so that they emit some information to the eventloop manager before going to I/O wait state. This allowed for another function to run till it entered same kind of I/O wait state. They were built upon the concept that a python process should never be sitting idle for I/O wait. Role of Gunicorn <-> workers # Gunicorn is just a process manager, that makes sure that “worker” process never stop processing the requests. It has two distinct section of architecture; one being “arbiter” and the other being “worker”. Arbiter does the job of manging worker processes while the worker process does the job of communicating with arbiter along with starting/monitoring the application execution context that processes request as per worker type defined. The worker type defines how the server application gets run. The default sync worker process one request at a single time per process. The workers that comes packaged in gunicorn are process-worker, threads, gevent, eventlet and tornado. But you will be able to find other third party workers like uvicorn workers that supports fastapi like async frameworks too. Sync vs Async workers in gunicorn # Sync workers are those workers which can handle requests sychronously i.e. one request at a time. 1 2 3 # both the gunicorn instances below can process 2 requests at once. The first one uses threads to do so while the second one uses python processes. gunicorn test:app --threads 2 --timeout 5 gunicorn test:app --workers 2 --timeout 5 Async workers, as the name suggests, can handle multiple requests at once. So there will theoritically be good resource utilization for i/o bound task. Gunicorn provides us with gevent, eventlet and tornado workers out of the box. Gevent is the most widely used due to its compatibility and speed as compared to tornado and eventlet. When to use which workers # When to use sync default workers
How do we make ingress nginx intelligent enough to route to different backend not just based on url path match but on the information that request body itself? What if request routing itself requires database access to find the right backend?
How we made debugging on cloud easier without costing a fortune. # Preface # Any organization will require good and flexible environment for every developer to test their code before merging it to the main branch.
This is normally not required for small-medium monolith application but if the application is too big or consists of a lot microservices and we want to test integration between mutliple microservices. This does not occur often but occurs when you have a feature to implement that will span changes across a lot of microservices. There are other method to test these like writing unit tests for each api of microservice so that it adhers to certain schema and changing the schema as per the changes done and the expected value.
Debugging python inside of a docker container # Premises # Lets say you are working on some legacy python project which has very obscure and obsolete dependencies and you now want to debug it. First lets pray to god that, this day does not come to anybody but if you are working on some project without standard practices on package management, this kind of situation is not uncommon.
