[{"content":"","date":"10 August 2025","externalUrl":null,"permalink":"/","section":"Ankit Paudel","summary":"","title":"Ankit Paudel","type":"page"},{"content":"","date":"10 August 2025","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":" Preface # You require a cheap monitoring solution that is cloud agnostic. Problem # Google\u0026rsquo;s manged prometheus costs a lot; so self hosting prometheus+thanos to get around it. As the required metrics to properly observe our application increases, the requirement to selfhost the monitoring platform to curb the cloud costs makes more sense, rather than using managed monitoring service. This opens up self hosting and maintaining Prometheus+Thanos+GCS stack for all the internal observability needs. low cost of GCS + infrequent access to older metrics data makes Thanos\u0026rsquo;s requirement more sensible. The Solution # Store less data in prometheus\u0026rsquo;s tsdb while maintaining historical data in gcs for low cost query. Setup proper retention period as business requirement to reduce the latency of metrics query. Prometheus # A stable and mature TSDB, designed to record and host application metrics. Pros: Easy to get started with grafana for basic monitoring needs. Low barrier to entry if the requirement is not high availability. Cons: Hard to scale if HA is explicity required. Multiple Prometheus instances work independently, it does not have built in feature for master-slave replication and HA deployment like postgres and other stateful applications. Thanos # Solution built on top of Prometheus to solve HA and scalibility needs. Requires active connection to Prometheus instance to function. Manages multiple Prometheus instances and handles deduplication, downsampling and other higher order features that Prometheus does not support natively. High level proposed architecture # Store as less of data as required in memory in the prometheus instances. Configure thanos side car to upload metric data as frequent as possible to gcs or other colder storage as compared to tsdb for low memory and cluster resource requirement. Possible issues with this approach # Chatty communication between monitoring components might incur higher networking bills for multi-region k8s clusters. Needs manual security considerations to secure grafana and other public endpoints. Might take up engineering bandwidth: requiring more engineers to maintain it in the long term. Installation # This uses helm-charts by bitnami for thanos and the official Prometheus community maintained helm chart. Optimized values for helm install of our specific use case: Chart used: kube_prometheus_stack and bitnami_thanos_chart. Image tags are latest as per the period of this article. Notes Some names like the service endpoints might be changed as per the updation of helm chart and your release names. This is before bitnami announced its paid offerings. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 # Prometheus alertmanager: # not used, in our case. enabled: false coreDns: enabled: false grafana: datasources: datasources.yaml: apiVersion: 1 datasources: - access: proxy isDefault: false jsonData: prometheusType: thanos name: prom-default type: prometheus url: http://thanos-query-frontend.monitoring.svc.cluster.local:9090 deploymentStrategy: type: Recreate # we are using RWO persistence persistence: enabled: true size: 5Gi kube-state-metrics: enabled: true kubeApiServer: enabled: false kubeControllerManager: enabled: false kubeProxy: enabled: false kubeScheduler: enabled: true kubeStateMetrics: enabled: true kubelet: enabled: true nodeExporter: enabled: false prometheus: prometheusSpec: containers: - env: - name: GOGC value: \u0026#34;75\u0026#34; name: prometheus externalLabels: cluster: \u0026lt;label-identifying-your-cluster-for-multicluster-setups\u0026gt; replica: $(POD_NAME) podMonitorSelectorNilUsesHelmValues: false replicas: 2 retention: 3h retentionSize: 8GiB serviceMonitorSelectorNilUsesHelmValues: false storageSpec: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: premium-rwo # this is specific to GCP, change as you require thanos: objectStorageConfig: existingSecret: # this is to access gcs for metrics storage; for aws, use per documentation of thanos. key: objstore.yml name: thanos-config thanosService: enabled: true prometheus-node-exporter: enabled: false # This is covered by the provided metrics of gke. prometheusOperator: admissionWebhooks: patch: image: tag: v1.5.2 thanosImage: tag: v0.39.2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 # Thanos bucket: enabled: true replicaCount: 1 compactor: cronJob: activeDeadlineSeconds: 10800 enabled: true failedJobsHistoryLimit: 3 schedule: 0 */6 * * * successfulJobsHistoryLimit: 2 enabled: true extraFlags: - --delete-delay=12h - --deduplication.replica-label=replica - --deduplication.replica-label=prometheus_replica persistence: enabled: false retentionResolution1h: 0d retentionResolution5m: 180d retentionResolutionRaw: 30d existingObjstoreSecret: thanos-config existingServiceMonitor: false metrics: enabled: true serviceMonitor: enabled: true objectStorageConfig: existingSecret: key: objstore.yml name: thanos-config query: dnsDiscovery: sidecarsNamespace: monitoring sidecarsService: prometheus-stack-kube-prom-thanos-discovery enabled: true extraFlags: - --query.auto-downsampling - --query.partial-response - --query.timeout=2m replicaCount: 2 queryFrontend: enabled: true extraFlags: - --query-range.split-interval=24h - --query-range.max-retries-per-request=3 - --query-frontend.compress-responses - --query-range.partial-response replicaCount: 2 ruler: enabled: false storegateway: enabled: true extraFlags: - --index-cache-size=512MB - --chunk-pool-size=512MB - --store.grpc.series-max-concurrency=10 - --block-sync-concurrency=3 - --store.enable-index-header-lazy-reader persistence: enabled: true size: 5Gi storageClass: premium-rwo replicaCount: 2 Final notes and recommendations # Setup proper alerting for failed compactors and other important metrics like consistent storage-gateway cache misses. Use proper memory requirements for storage gateway and querier for faster queries as business requirements. This setup does not factor in the cardinality constraints of your specific usecase. You might be a startup just starting with low enough unique users or a B2B product with low enough user-base that can get away with taking in user-id as a label in prom-metrics, but this might blow up your prom instance if your userbase is large enough. Keep monitoring the resource usage and latency of thanos and prom components for few days/weeks for resource exhaustion and your needs. If your prom starts getting OOM failures, you might need to reconfigure this setup to use remote-write instead of sidecar based pattern. Remote-Write is a little complicated but is a good trade-off if you have a quite a few components that queries for latest metrics and have enough engineering bandwidth to maintain it in the long term. Use a separate Node pool for monitoring components like prometheus and stateful components like databases and distributed caches for resource isolation and to reduce failure domains. ","date":"10 August 2025","externalUrl":null,"permalink":"/posts/prom-thanos/","section":"Posts","summary":"Preface # You require a cheap monitoring solution that is cloud agnostic. Problem # Google’s manged prometheus costs a lot; so self hosting prometheus+thanos to get around it. As the required metrics to properly observe our application increases, the requirement to selfhost the monitoring platform to curb the cloud costs makes more sense, rather than using managed monitoring service. This opens up self hosting and maintaining Prometheus+Thanos+GCS stack for all the internal observability needs. low cost of GCS + infrequent access to older metrics data makes Thanos’s requirement more sensible. The Solution # Store less data in prometheus’s tsdb while maintaining historical data in gcs for low cost query. Setup proper retention period as business requirement to reduce the latency of metrics query. Prometheus # A stable and mature TSDB, designed to record and host application metrics. Pros: Easy to get started with grafana for basic monitoring needs. Low barrier to entry if the requirement is not high availability. Cons: Hard to scale if HA is explicity required. Multiple Prometheus instances work independently, it does not have built in feature for master-slave replication and HA deployment like postgres and other stateful applications. Thanos # Solution built on top of Prometheus to solve HA and scalibility needs. Requires active connection to Prometheus instance to function. Manages multiple Prometheus instances and handles deduplication, downsampling and other higher order features that Prometheus does not support natively. High level proposed architecture # Store as less of data as required in memory in the prometheus instances. Configure thanos side car to upload metric data as frequent as possible to gcs or other colder storage as compared to tsdb for low memory and cluster resource requirement. Possible issues with this approach # Chatty communication between monitoring components might incur higher networking bills for multi-region k8s clusters. Needs manual security considerations to secure grafana and other public endpoints. Might take up engineering bandwidth: requiring more engineers to maintain it in the long term. Installation # This uses helm-charts by bitnami for thanos and the official Prometheus community maintained helm chart. Optimized values for helm install of our specific use case: Chart used: kube_prometheus_stack and bitnami_thanos_chart. Image tags are latest as per the period of this article. Notes Some names like the service endpoints might be changed as per the updation of helm chart and your release names. This is before bitnami announced its paid offerings. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 # Prometheus alertmanager: # not used, in our case. enabled: false coreDns: enabled: false grafana: datasources: datasources.yaml: apiVersion: 1 datasources: - access: proxy isDefault: false jsonData: prometheusType: thanos name: prom-default type: prometheus url: http://thanos-query-frontend.monitoring.svc.cluster.local:9090 deploymentStrategy: type: Recreate # we are using RWO persistence persistence: enabled: true size: 5Gi kube-state-metrics: enabled: true kubeApiServer: enabled: false kubeControllerManager: enabled: false kubeProxy: enabled: false kubeScheduler: enabled: true kubeStateMetrics: enabled: true kubelet: enabled: true nodeExporter: enabled: false prometheus: prometheusSpec: containers: - env: - name: GOGC value: \"75\" name: prometheus externalLabels: cluster: \u003clabel-identifying-your-cluster-for-multicluster-setups\u003e replica: $(POD_NAME) podMonitorSelectorNilUsesHelmValues: false replicas: 2 retention: 3h retentionSize: 8GiB serviceMonitorSelectorNilUsesHelmValues: false storageSpec: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: premium-rwo # this is specific to GCP, change as you require thanos: objectStorageConfig: existingSecret: # this is to access gcs for metrics storage; for aws, use per documentation of thanos. key: objstore.yml name: thanos-config thanosService: enabled: true prometheus-node-exporter: enabled: false # This is covered by the provided metrics of gke. prometheusOperator: admissionWebhooks: patch: image: tag: v1.5.2 thanosImage: tag: v0.39.2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 # Thanos bucket: enabled: true replicaCount: 1 compactor: cronJob: activeDeadlineSeconds: 10800 enabled: true failedJobsHistoryLimit: 3 schedule: 0 */6 * * * successfulJobsHistoryLimit: 2 enabled: true extraFlags: - --delete-delay=12h - --deduplication.replica-label=replica - --deduplication.replica-label=prometheus_replica persistence: enabled: false retentionResolution1h: 0d retentionResolution5m: 180d retentionResolutionRaw: 30d existingObjstoreSecret: thanos-config existingServiceMonitor: false metrics: enabled: true serviceMonitor: enabled: true objectStorageConfig: existingSecret: key: objstore.yml name: thanos-config query: dnsDiscovery: sidecarsNamespace: monitoring sidecarsService: prometheus-stack-kube-prom-thanos-discovery enabled: true extraFlags: - --query.auto-downsampling - --query.partial-response - --query.timeout=2m replicaCount: 2 queryFrontend: enabled: true extraFlags: - --query-range.split-interval=24h - --query-range.max-retries-per-request=3 - --query-frontend.compress-responses - --query-range.partial-response replicaCount: 2 ruler: enabled: false storegateway: enabled: true extraFlags: - --index-cache-size=512MB - --chunk-pool-size=512MB - --store.grpc.series-max-concurrency=10 - --block-sync-concurrency=3 - --store.enable-index-header-lazy-reader persistence: enabled: true size: 5Gi storageClass: premium-rwo replicaCount: 2 Final notes and recommendations # Setup proper alerting for failed compactors and other important metrics like consistent storage-gateway cache misses. Use proper memory requirements for storage gateway and querier for faster queries as business requirements. This setup does not factor in the cardinality constraints of your specific usecase. You might be a startup just starting with low enough unique users or a B2B product with low enough user-base that can get away with taking in user-id as a label in prom-metrics, but this might blow up your prom instance if your userbase is large enough. Keep monitoring the resource usage and latency of thanos and prom components for few days/weeks for resource exhaustion and your needs. If your prom starts getting OOM failures, you might need to reconfigure this setup to use remote-write instead of sidecar based pattern. Remote-Write is a little complicated but is a good trade-off if you have a quite a few components that queries for latest metrics and have enough engineering bandwidth to maintain it in the long term. Use a separate Node pool for monitoring components like prometheus and stateful components like databases and distributed caches for resource isolation and to reduce failure domains. ","title":"Cloud-native monitoring with Lightweight Prometheus + Thanos on GKE","type":"posts"},{"content":"","date":"10 August 2025","externalUrl":null,"permalink":"/categories/kubernetes/","section":"Categories","summary":"","title":"Kubernetes","type":"categories"},{"content":"","date":"10 August 2025","externalUrl":null,"permalink":"/tags/monitoring/","section":"Tags","summary":"","title":"Monitoring","type":"tags"},{"content":"","date":"10 August 2025","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"10 August 2025","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"16 February 2025","externalUrl":null,"permalink":"/categories/database/","section":"Categories","summary":"","title":"Database","type":"categories"},{"content":"","date":"16 February 2025","externalUrl":null,"permalink":"/tags/mongodb/","section":"Tags","summary":"","title":"Mongodb","type":"tags"},{"content":" Preface # This post talks about solutions available in mongodb and mongo atlas to keep track of database changes. This was required in our case to sync external elastic search instance in sync with changes going on in multiple collections of mongodb. Wrong choices of config which is setup default will result in terrible performance, might even repeling users from using this architecture. Problem # A feature came up that required fast retrieval from millions of documents in a collection of mongo. We needed to filter a few hundred entries from millions of docs. We decided to go with elasticsearch for full text search across thousands of documents but the hurdle here was keeping the elastic in sync with changes in db done by information extraction. We had data manipulation happening in multiple stages of document processing, thus our backend had database operation all over the place. How the Hunt for a good solution started # Initial Solutions: Hook into database for event based insertion. Change each part of backend that refrenced mongodb operation. A team member among us experimented much with triggers and its connection to private elastic deployment. Could not make this work without letting traffic leak into the wide internet. Somehow atlas trigger did not let us connect using private ip of our elastic instance, it kept saying its a restricted host. It also had other challenges like the ip of serverless instances that run this \u0026ldquo;trigger\u0026rdquo; was elusive to even get if we wanted to allow it in our firewall. After failure of this, it led us experiment with mongodb streams. Mongodb had built in feature called streams to listen for the changes by clients through the same connection that enables its access. It did this by advertising modified oplogs to the clients. The first experiment with streams was reported by the team to have very bad performance, the messages would get received very late, sometimes even after 3-4 mins. My first thought was, this should not be this bad, a database advertising its oplog should not be this slow, thus I started digging in this. The Experiment # I with the help my trustee friend gpt created the right env to test this claim. I was keen on learning golang, thus chose it for the job:\nThe first go would continuously send operation requests like update, create and delete while writing the current timestamp, operation along with doc_id in a local file. The second one would listen for events through streams, writing the same details in another file. Then these two files were tallied cronologically to find the latency of operation. These were the initial results: 1 2 3 4 5 6 7 8 9 10 11 12 13 count 2872423.000000 mean 199.910508 std 601.038466 min 0.000000 50% 34.986000 90% 456.072600 95% 1037.822600 99% 2964.515180 max 23052.747000 All latencies in ms Not bad numbers IMHO. But I had been missing one critical thing in this experiment. The data in the change stream had ony the delta. i.e. just the actual oplog I had already presented this as a good possible solution of the problem, that made my heart melt a little. There was a option that you could pass to watch function of mongo client that is supposed to give us the updated document called fullDocument:updateLookup.\nThis gave me the results that my teammates were getting. The latencies shot up to 300-400 secs till the ops/sec died down in the primary node, after which all the pending oplogs would flood at once. This had me thinking, this is one of the basic requirement of a good database engine, why would mongodb be so bad at this.\nThen I changed the readpreference of the client to prefer secondary, it improved somewhat but was still not satisfied with the result.\nDigging down a little, mongodb had a feature called changeStreamPreAndPostImages as a property of a collection that stored the previous and later state of a document for each and every update till all the secondaries and the ones listening for changes were notified. There is also other setting to set the ttl for documents to be on this collection so that it does not flood the storage. Mongo Docs\nAfter enabling this for the collection in question we got the numbers we were looking for. The latencies were almost equivalent to the first part but with the data we want. The Challenges # High Availability How do you ensure that all the data has been synced? What if the system responsible for sync is down? Change streams are cursor based systems so we can maintain leaderboard or cache like system in redis that keeps track of some cursor id and a newly started sync application can use that to resume its job. What if the changestreams backlog crosses limit in the mongo side? Mongodb must have some limit on how much change oplog can it store before sending it to changestreams? Answers to these question still requires more research on architecture of mongo before we implement this in production. Conclusion # We can use mongo change streams for reliable and near realtime data sync across systems where additional processing needs to be done for each database operation. ","date":"16 February 2025","externalUrl":null,"permalink":"/posts/triggers-change-streams-mongo/","section":"Posts","summary":"Intro on how to use triggers and change-streams in mongo","title":"Mongodb triggers and change streams","type":"posts"},{"content":" Preface: # This Blog talks about my experience with setting up and using Longhorn for our RWX needs. We decided to go against longhorn due to it being too unstable in rapidly changing cluster The final choice we landed on was using a VM to host nfs off cluster with proper hardning. Problem: # Our setup required sharing large data between kubernetes pods. Much like map reduce, lot of tasks would create a large number of files, save it to nfs, and one last task that collated all those files, ran some logic to make sense out of it and return. We used Celery’s canvas for this. This required us to setup a simple nfs server (How to for this is available all over the internet) This involved, a deployment with single pod running nfs-server, a pv using nfs csi native with k8s and finally a rwx pvc for a pod to use. This made nfs our single point of failure, random nfs-restarts would cause badly configured application writing to it, error out with PermissionError . This happened in our python application. This issue would only arise when the existing pod of nfs-server switches nodes. This can happen in any scenario, during cluster scaling events or during node upgrade period. The Solution: # Make highly available RWX Storage that can survive node unavailability. Searched a bit around internet, they were suggesting us to use managed nfs solution provided by cloud provider. GCP’s offering filestore requires minimum size of the storage be 1TB which was a lot. We needed around 20GB of storage at max AWS has its offering with EFS but we did not have luxury of using it. Stumbled upon a few implementations: OpenEBS: Good, but its rwx offering was still not GA. Rook: Best, but was too complex for our usecase. Ceph itself is too complex thus decided to not include its complexity for a small team like us. Suspected it as a future tech debt, requiring ceph and rook specific engineer to maintain it in long run. Longhorn: Suited our case. RWX offering at GA. Not too complex architecture. Hence we chose Longhorn. Longhorn # Very easy to setup, finicky sometimes.\nMajor issue, clusters with spot instances are on high chances of faulting.\nUsed following helm-chart values for fast fail-over and good resiliency:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 defaultSettings: createDefaultDiskLabeledNodes: false persistence.defaultClass: false # set this to false in prod guaranteedInstanceManagerCPU: 5 kubernetesClusterAutoscalerEnabled: true autoSalvage: true # tries to recover from multiple node failures nodeDownPodDeletionPolicy: delete-both-statefulset-and-deployment-pod nodeDrainPolicy: block-for-eviction-if-contains-last-replica replicaAutoBalance: best-effort replicaReplenishmentWaitInterval: 30 rwxVolumeFastFailover: true fastReplicaRebuildEnabled: true snapshotDataIntegrity: \u0026#34;fast-check\u0026#34; # this is required for fast replica rebuild. persistence: defaultClassReplicaCount: 3 defaultDataLocality: disabled longhornManager: log: format: json longhornDriver: log: format: json Uses a open source project called NFS-Ganesha for its rwx volume.\nThis allows nfs to run in userspace without nfs’s kernel module. Creating a new RWX PV ( Its not provided by default during installation) # First create a volume from LonghornUI or from longhorn cli application.\nCreate a storage class that supports RWX and uses longhorn csi driver:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: longhorn-nfs provisioner: driver.longhorn.io allowVolumeExpansion: true reclaimPolicy: Delete volumeBindingMode: Immediate parameters: numberOfReplicas: \u0026#34;3\u0026#34; staleReplicaTimeout: \u0026#34;2880\u0026#34; fromBackup: \u0026#34;\u0026#34; fsType: \u0026#34;ext4\u0026#34; nfsOptions: \u0026#34;vers=4.1,noresvport,softerr,timeo=600,retrans=5\u0026#34; Create a Longhorn Volume from Longhorn UI or from Longhorn CLI, set the replicas to at least 3, Set Access Mode to ReadWriteMany ,Set Data Locality to best-effort .\nGo to the Volume Created in the UI, there should be a dropdown besides it which has option saying something like Create PV/PVC , click that.\nSet Storage Class Name to longhorn-nfs , PVC Name , Namespace to namespace you want to bind the volume to .\nSolving Basic Issues: # Once a volume faults, its going to be hard to recover it Manual salvage. How to Okay when failure rate is not that high, when used with spot instances, I experienced random volume being faulted overnight. How to recover from faulted volume: # First check what is the actual issue. Case 1: When there are a few replicas online but replication sequence is not starting. Navigate to the replica that is faulted and is having hard time to get out of the quorum, set the faultedAt part to empty string, apply it and let the controller handle the retry. Case 2: It says instance manager not running in the node running a pod that is trying to attach the pvc. Longhorn ui shows it being attached to some pod that does not exist anymore. Try to evict the pod from that node. Just deleting the pod might not work, you can use one a kubectl extension available in krew named evict-pod use it. If the pod is controlled by some other resource, you can try to scale down it to 0 and then to rescale back. If that does not work, you might need to force detach the volume. It might not be possible from Longhorn UI, but you can edit the yaml of longhorn volume, set the nodeID or some other label that indicates the volume attachment to node to an empty string. Case 3: The volume itself is faulted, No healthy replica remains. Check if the volume still says attached to non existing node, if that is the case, remove that node from the volume by editing the yaml of the faulted resource. If the volume says its detached and Faulted, you can just reduce the replicas by editing yaml of the volume, make it 1, edit the remaining replica, set the field containing attached node information and failedAt both to an empty string. The Conclusion # Longhorn is not stable enough or is difficult to stabilize it in spot instance cluster. If it does not work efficiently in a spot instance cluster, its not reliable enough to ship to production. ","date":"6 February 2025","externalUrl":null,"permalink":"/posts/rwx-in-k8s-longhorn/","section":"Posts","summary":"Intro to solutions for RWX volume in k8s cluster. The experience with using longhorn for the job.","title":"Rwx Volume in k8s: Adventure with Longhorn","type":"posts"},{"content":"","date":"6 February 2025","externalUrl":null,"permalink":"/tags/storage/","section":"Tags","summary":"","title":"Storage","type":"tags"},{"content":" Python is single threaded # Python by design is single threaded. It can only run one thread at once due to GIL. This makes making web server which serves a lot requests at once a bit complicated. Thread based concurrency is good enough for most of the applications but its control block is heavier than traditional async coroutines. This led to necessity of eventloop based concurrency which can share almost eveything of main python thread except the request data. This would make coroutines very light with fast context switches. Async in python before asyncio # To handle thousands of request at once, there were a few implementations that made python work somewhat like async runtime. Gevent and eventlets being the major one. There was also a framework called tornado which had its own implementation of coroutine based server which is not that popular now. First eventlet was released to tackle this problem with its pure python implementation but later gevent came to picture with its blazing fast speed as it was implemented in cython itself. They work by monkeypatching specific functions in python that does I/O operations. They replace those functions with their own version so that they emit some information to the eventloop manager before going to I/O wait state. This allowed for another function to run till it entered same kind of I/O wait state. They were built upon the concept that a python process should never be sitting idle for I/O wait. Role of Gunicorn \u0026lt;-\u0026gt; workers # Gunicorn is just a process manager, that makes sure that \u0026ldquo;worker\u0026rdquo; process never stop processing the requests. It has two distinct section of architecture; one being \u0026ldquo;arbiter\u0026rdquo; and the other being \u0026ldquo;worker\u0026rdquo;. Arbiter does the job of manging worker processes while the worker process does the job of communicating with arbiter along with starting/monitoring the application execution context that processes request as per worker type defined. The worker type defines how the server application gets run. The default sync worker process one request at a single time per process. The workers that comes packaged in gunicorn are process-worker, threads, gevent, eventlet and tornado. But you will be able to find other third party workers like uvicorn workers that supports fastapi like async frameworks too. Sync vs Async workers in gunicorn # Sync workers are those workers which can handle requests sychronously i.e. one request at a time. 1 2 3 # both the gunicorn instances below can process 2 requests at once. The first one uses threads to do so while the second one uses python processes. gunicorn test:app --threads 2 --timeout 5 gunicorn test:app --workers 2 --timeout 5 Async workers, as the name suggests, can handle multiple requests at once. So there will theoritically be good resource utilization for i/o bound task. Gunicorn provides us with gevent, eventlet and tornado workers out of the box. Gevent is the most widely used due to its compatibility and speed as compared to tornado and eventlet. When to use which workers # When to use sync default workers\nIf the endpoints that your server serves is resource heavy and can fully utilize the cpu of the application in each request then sync workers is the only option for you.\nIf your system is mutltithreaded, then you will benefit from running multiple workers that fully tanks your cpu. If your application is very resource intensive, number_of_workers=number_of_cpu_threads can perform well.\nAssuming you have around half of the application which is cpu intensive and the other half is somewhat io intensive you should be okay with the golden rule floating around, num_workers=(2*num_cpu_cores)+1\nAs increasing the number of workers directly increases the python processes running your application code, it has the highest memory overhead.\nWhen to use thread workers\nIf your endpoints are I/O intensive that can somewhat benefit from python\u0026rsquo;s inbuilt gil, then go for threaded workers.\nThis can be used as default for most of the new flask or WSGI applications.\nThe number of threads will depend on your workload.\nThis is a bit lighter than worker processes, as not the whole python application is loaded in memory as soon as a new thread gets spawned.\nThreaded worker is the only option for a few cases where gevent support is not there for application code.\nAs this shares same python process, use of any global state that any endpoint modifies or uses is prohibited. If done, undefined behaviour might occur.\nWhen to use Gevent workers\nIf your endpoints are almost exclusively I/O bound this should be the best type of worker for you.\nThis will ensure highest throughput for your application with linear resource scaling for I/O bound tasks.\nThis also prohibits any global state being shared across requests.\nThe issue with this worker is not all the applications and library support it.\nTrue async workers vs gevent # The asyncio library of python is superior in portability and operability. But when it comes to sheer number of coroutines that an application can handle if the coroutine is totally network bound, gevents can also sometimes be better. Gevents don\u0026rsquo;t support context switching in all the internal I/O apis like file operations, so file operation in gevent worker will stop the whole program from responding. Fastapi and starlette are built around asyncio framework of python, thus are better at handling real world workloads and would certainly be better performing in medium to high loads as compared to gevent. Pallet Projects, the parent project of flask has another flask like framework that supports true async worker called Quart. But the community for fastapi and starlette is very large as compared to those. Hence my personal recommendation would be to use fastapi or such for web application in python. ","date":"22 December 2024","externalUrl":null,"permalink":"/posts/gunicorn_workers/","section":"Posts","summary":"Python is single threaded # Python by design is single threaded. It can only run one thread at once due to GIL. This makes making web server which serves a lot requests at once a bit complicated. Thread based concurrency is good enough for most of the applications but its control block is heavier than traditional async coroutines. This led to necessity of eventloop based concurrency which can share almost eveything of main python thread except the request data. This would make coroutines very light with fast context switches. Async in python before asyncio # To handle thousands of request at once, there were a few implementations that made python work somewhat like async runtime. Gevent and eventlets being the major one. There was also a framework called tornado which had its own implementation of coroutine based server which is not that popular now. First eventlet was released to tackle this problem with its pure python implementation but later gevent came to picture with its blazing fast speed as it was implemented in cython itself. They work by monkeypatching specific functions in python that does I/O operations. They replace those functions with their own version so that they emit some information to the eventloop manager before going to I/O wait state. This allowed for another function to run till it entered same kind of I/O wait state. They were built upon the concept that a python process should never be sitting idle for I/O wait. Role of Gunicorn \u003c-\u003e workers # Gunicorn is just a process manager, that makes sure that “worker” process never stop processing the requests. It has two distinct section of architecture; one being “arbiter” and the other being “worker”. Arbiter does the job of manging worker processes while the worker process does the job of communicating with arbiter along with starting/monitoring the application execution context that processes request as per worker type defined. The worker type defines how the server application gets run. The default sync worker process one request at a single time per process. The workers that comes packaged in gunicorn are process-worker, threads, gevent, eventlet and tornado. But you will be able to find other third party workers like uvicorn workers that supports fastapi like async frameworks too. Sync vs Async workers in gunicorn # Sync workers are those workers which can handle requests sychronously i.e. one request at a time. 1 2 3 # both the gunicorn instances below can process 2 requests at once. The first one uses threads to do so while the second one uses python processes. gunicorn test:app --threads 2 --timeout 5 gunicorn test:app --workers 2 --timeout 5 Async workers, as the name suggests, can handle multiple requests at once. So there will theoritically be good resource utilization for i/o bound task. Gunicorn provides us with gevent, eventlet and tornado workers out of the box. Gevent is the most widely used due to its compatibility and speed as compared to tornado and eventlet. When to use which workers # When to use sync default workers\n","title":"Guide to worker types in gunicorn","type":"posts"},{"content":" Webserver in Python # Originally, python was not made to develop webservers, but due to its simplicity and very easy learning curve, along time a few web frameworks started popping. Different frameworks/tools had their own serving method that tried to alleviate the fact that python was a single threaded application and io calls would block the application from receiving any further requests. Solution: use threads, multiple python processes and their combination. This issue led to development of some standard interface that all frameworks can adhere to so that frameworks don\u0026rsquo;t have to worry about how to manage concurrency and be best on what they do, acting as an easy and fast tool to create a httpserver. What is WSGI # WSGI is the interface that most of the python web frameworks provide for their application to be served without worrying about the specifics of concurrency and load management in high rps environments. At the core, its just set of documents that describe how a compatible web application should be structured, process the request and respond accordingly. A simple hello world WSGI application looks like this: 1 2 3 4 5 6 7 8 9 def application(environ, start_response): method = environ[\u0026#39;REQUEST_METHOD\u0026#39;] print(method) path = environ[\u0026#39;PATH_INFO\u0026#39;] print(path) status = \u0026#39;200 OK\u0026#39; response_headers = [(\u0026#39;Content-Type\u0026#39;, \u0026#39;text/plain; charset=utf-8\u0026#39;)] start_response(status, response_headers) return [b\u0026#39;Hello World!\u0026#39;] This application can be ran using gunicorn which is a wsgi server designed to serve these kind of applications. The simplest command to do so would be gunicorn app:application. So all the flask or Django applications can be accessed as a simple function above. All the ifs are abstracted away in different ways in different frameworks, decorators in flask and functions/methods in django. Gunicorn # As per gunicorn\u0026rsquo;s documentation, gunicorn is inspired from ruby\u0026rsquo;s unicorn project which used to manage multiple ruby web-server processes. It essentially is a process manager that kills/restarts hung processes that is supposed to listen to requests. When gunicorn starts, it starts its main process called \u0026ldquo;arbiter\u0026rdquo; which listens to some port. Then it gets forked into multiple \u0026ldquo;worker\u0026rdquo; processes that inherit the socket that was opened during the startup of arbiter thus, multiple processes would be listening to same os port. The \u0026ldquo;worker\u0026rdquo; that I mentioned previously is a python process that runs the WSGI compatible web server. As multiple workers are listening to a single port, the role of load-balancing is handed over to kernel. The kernel manages the resource contention between multiple \u0026ldquo;worker\u0026rdquo; processes for the opportunity to handle the request. This disproves the fact that would be plaguing people\u0026rsquo;s mind assuming that arbiter acts as some sort of reverse proxy like nginx. The \u0026ldquo;management\u0026rdquo; of \u0026ldquo;worker\u0026rdquo; processes by arbiter includes things like restarting workers when workers gets hung up. \u0026ldquo;hang up\u0026rdquo; in this context means that the worker is not able to message arbiter saying its okay for some period of time. This is somewhat like linux\u0026rsquo;s \u0026ldquo;watchdog\u0026rdquo; concept. Async workers can send periodic heartbeats but sync workers can\u0026rsquo;t do that if they are stuck in some I/O operations. In this case, arbiter relies in messages in specific occations like acknowleding the request, returning response and likes. Sends SIGKILL to timed out workers which gets logged as log somewhat like this: [CRITICAL] WORKER TIMEOUT. This might be due to anything, If your request is too slow that it took more than the timeout specified to response. There can be different worker types in gunicorn. Gunicorn is very simple, it just does not let worker live without it processing a request. The workers can be put into two distinct buckets that explain how they process python requests. Sync and Async workers. Sync workers can handle one request at a time while async ones can handle multiple requests at the same time. Timeout in gunicorn # As gunicorn is a process manager(not request handler), the timeout you give during the gunicorn\u0026rsquo;s startup is timeout of a worker process. This timeout is equal to the timeout of request when you are using sync workers but can be totally different if you are using async workers. Value in --timeout does not mean the max time a client can wait for response. Following simple flask program demonstrates this behaviour of gunicorn: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 import time from flask import Flask app=Flask(\u0026#34;__name__\u0026#34;) @app.get(\u0026#34;/slow\u0026#34;) def test(): time.sleep(10) return \u0026#34;slept for 10s\u0026#34; @app.get(\u0026#34;/fast\u0026#34;) def test1(): time.sleep(1) return \u0026#34;return super fast, in 1 second\u0026#34; When you run the program using this gunicorn command: gunicorn test:app --threads 2 --timeout 5 which spawns multipel execution context per worker process. Then you send a request to the /slow endpoint, you get your response after 10 secs. The other thread was alive thus the worker process was able to prove that its alive during the whole time. Contrary to above, if you set the thread count to 1 or use sync worker, you will get the worker timeout log and 5xx would be returned to the client. The responsibility of timeout is soly given to the application side where the developer must set proper timeout for each request. This can be done in various ways I am giving only specific outlines to a few ways because this topic in itself is worthy of a whole article. Setting every socket connection\u0026rsquo;s alive time globally in each process. This can be very easy fix but might not work in scenarios like http/2 because they use single connection for multiple http requests. Making a timeout decorator and using them in each of the route handler. This is very common in flask like application where you define the route handler using decorators. Use Middleware. This can be applicable in express-like frameworks or django where middlewares can ensure that the request can timeout after certain seconds. Use of reverse proxy like nginx. As they create a new http connection between themself and the server application, nginx can track time taken by each request and close connection to server after some time. Nginx will normally return 504 to the clients if it can\u0026rsquo;t reach the application server within the time defined. ","date":"22 December 2024","externalUrl":null,"permalink":"/posts/gunicorn_wsgi_intro/","section":"Posts","summary":" Webserver in Python # Originally, python was not made to develop webservers, but due to its simplicity and very easy learning curve, along time a few web frameworks started popping. Different frameworks/tools had their own serving method that tried to alleviate the fact that python was a single threaded application and io calls would block the application from receiving any further requests. Solution: use threads, multiple python processes and their combination. This issue led to development of some standard interface that all frameworks can adhere to so that frameworks don’t have to worry about how to manage concurrency and be best on what they do, acting as an easy and fast tool to create a httpserver. What is WSGI # WSGI is the interface that most of the python web frameworks provide for their application to be served without worrying about the specifics of concurrency and load management in high rps environments. At the core, its just set of documents that describe how a compatible web application should be structured, process the request and respond accordingly. A simple hello world WSGI application looks like this: 1 2 3 4 5 6 7 8 9 def application(environ, start_response): method = environ['REQUEST_METHOD'] print(method) path = environ['PATH_INFO'] print(path) status = '200 OK' response_headers = [('Content-Type', 'text/plain; charset=utf-8')] start_response(status, response_headers) return [b'Hello World!'] This application can be ran using gunicorn which is a wsgi server designed to serve these kind of applications. The simplest command to do so would be gunicorn app:application. So all the flask or Django applications can be accessed as a simple function above. All the ifs are abstracted away in different ways in different frameworks, decorators in flask and functions/methods in django. Gunicorn # As per gunicorn’s documentation, gunicorn is inspired from ruby’s unicorn project which used to manage multiple ruby web-server processes. It essentially is a process manager that kills/restarts hung processes that is supposed to listen to requests. When gunicorn starts, it starts its main process called “arbiter” which listens to some port. Then it gets forked into multiple “worker” processes that inherit the socket that was opened during the startup of arbiter thus, multiple processes would be listening to same os port. The “worker” that I mentioned previously is a python process that runs the WSGI compatible web server. As multiple workers are listening to a single port, the role of load-balancing is handed over to kernel. The kernel manages the resource contention between multiple “worker” processes for the opportunity to handle the request. This disproves the fact that would be plaguing people’s mind assuming that arbiter acts as some sort of reverse proxy like nginx. The “management” of “worker” processes by arbiter includes things like restarting workers when workers gets hung up. “hang up” in this context means that the worker is not able to message arbiter saying its okay for some period of time. This is somewhat like linux’s “watchdog” concept. Async workers can send periodic heartbeats but sync workers can’t do that if they are stuck in some I/O operations. In this case, arbiter relies in messages in specific occations like acknowleding the request, returning response and likes. Sends SIGKILL to timed out workers which gets logged as log somewhat like this: [CRITICAL] WORKER TIMEOUT. This might be due to anything, If your request is too slow that it took more than the timeout specified to response. There can be different worker types in gunicorn. Gunicorn is very simple, it just does not let worker live without it processing a request. The workers can be put into two distinct buckets that explain how they process python requests. Sync and Async workers. Sync workers can handle one request at a time while async ones can handle multiple requests at the same time. Timeout in gunicorn # As gunicorn is a process manager(not request handler), the timeout you give during the gunicorn’s startup is timeout of a worker process. This timeout is equal to the timeout of request when you are using sync workers but can be totally different if you are using async workers. Value in --timeout does not mean the max time a client can wait for response. Following simple flask program demonstrates this behaviour of gunicorn: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 import time from flask import Flask app=Flask(\"__name__\") @app.get(\"/slow\") def test(): time.sleep(10) return \"slept for 10s\" @app.get(\"/fast\") def test1(): time.sleep(1) return \"return super fast, in 1 second\" When you run the program using this gunicorn command: gunicorn test:app --threads 2 --timeout 5 which spawns multipel execution context per worker process. Then you send a request to the /slow endpoint, you get your response after 10 secs. The other thread was alive thus the worker process was able to prove that its alive during the whole time. Contrary to above, if you set the thread count to 1 or use sync worker, you will get the worker timeout log and 5xx would be returned to the client. The responsibility of timeout is soly given to the application side where the developer must set proper timeout for each request. This can be done in various ways I am giving only specific outlines to a few ways because this topic in itself is worthy of a whole article. Setting every socket connection’s alive time globally in each process. This can be very easy fix but might not work in scenarios like http/2 because they use single connection for multiple http requests. Making a timeout decorator and using them in each of the route handler. This is very common in flask like application where you define the route handler using decorators. Use Middleware. This can be applicable in express-like frameworks or django where middlewares can ensure that the request can timeout after certain seconds. Use of reverse proxy like nginx. As they create a new http connection between themself and the server application, nginx can track time taken by each request and close connection to server after some time. Nginx will normally return 504 to the clients if it can’t reach the application server within the time defined. ","title":"Introduction to WSGI and gunicorn","type":"posts"},{"content":"","date":"22 December 2024","externalUrl":null,"permalink":"/categories/python/","section":"Categories","summary":"","title":"Python","type":"categories"},{"content":"","date":"10 July 2024","externalUrl":null,"permalink":"/tags/ingress/","section":"Tags","summary":"","title":"Ingress","type":"tags"},{"content":"","date":"10 July 2024","externalUrl":null,"permalink":"/tags/nginx/","section":"Tags","summary":"","title":"Nginx","type":"tags"},{"content":" Problem # Consider your product is going through a big overhaul and re-architecturing where the existing microservices might be broken into multiple ones or merged to single ones, there will be a big jumble of mess of routes at the ingress level. This will make transitioning a bit hard as using canary based routing will gradually become complicated. So to simplify this canary setup, we take the routing matter into our own hands. If you can use Consul and it works right for you, that might be the better option as Consul is more mature and production ready. You will be less likely to break things. The Solution # Architecture proposed: Implementation # Its Almost same as auth implementation but will require a bit of scripting as we are not just checking status code to return 401 or to allow the traffic. This can be easily achieved with configuration snippet annotation in ingress-nginx. We write a configuration snippet that sends request to the server that is responsible for getting the route information from the request body. Which more or less looks like one below: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: docs-service annotations: nginx.ingress.kubernetes.io/configuration-snippet: | access_by_lua_block { local cjson = require(\u0026#34;cjson\u0026#34;) local http = require \u0026#34;resty.http\u0026#34; local default_backend=\u0026#34;docs1\u0026#34; local httpc = http.new() local res, err = httpc:request_uri(\u0026#34;http://authserver.ingress-nginx-helpers.svc.cluster.local:5000/find-backend\u0026#34;, { method = \u0026#34;POST\u0026#34;, body = cjson.encode({ real_url = ngx.var.request_uri }), headers = { [\u0026#34;Content-Type\u0026#34;] = \u0026#34;application/json\u0026#34;, [\u0026#34;Token\u0026#34;] = ngx.req.get_headers()[\u0026#39;token\u0026#39;], [\u0026#34;cookie\u0026#34;] = ngx.req.get_headers()[\u0026#39;cookie\u0026#39;], [\u0026#34;x-api-key\u0026#34;] = ngx.req.get_headers()[\u0026#39;x-api-key\u0026#39;] } }) local response, decode_err = cjson.decode(res.body) if response then ngx.var.proxy_upstream_name = ngx.var.namespace .. \u0026#34;-\u0026#34; .. response.backend .. ngx.var.service_port end ngx.ctx.balancer=nil } spec: ingressClassName: nginx rules: - host: hello.world.com http: paths: - backend: service: name: docs1 port: number: 5000 path: /api/v1/docs1/docs/custom/ pathType: Exact This will match for path /api/v1/docs1/docs/custom/ and send request to the server to get backend information. Which in turn will be used to populate proxy_upstream_name Don\u0026rsquo;t forget to set ngx.ctx.balancer to nil at last. Read here Enhancement # This for POC looks good but for real world application we try to make it as frictionless as possible. Creating a ingress-nginx plugin # We create a plugin that does all of this and a flag that will determine weather to run this plugin or not for each ingress. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 apiVersion: v1 kind: ConfigMap metadata: name: ngx-custom-script namespace: ingress-nginx data: main.lua: | local ngx = ngx local _M = {} local cjson = require(\u0026#34;cjson\u0026#34;) local http = require \u0026#34;resty.http\u0026#34; function _M.rewrite() if not (ngx.var.dynamic_plugin == \u0026#34;on\u0026#34;) then return end local final_backend=\u0026#34;eevee\u0026#34; local httpc = http.new() local res, err = httpc:request_uri(\u0026#34;http://authserver.ingress-nginx-helpers.svc.cluster.local:5000/find-backend\u0026#34;, { method = \u0026#34;POST\u0026#34;, body = cjson.encode({ real_url = ngx.var.request_uri }), headers = { [\u0026#34;Content-Type\u0026#34;] = \u0026#34;application/json\u0026#34;, [\u0026#34;Token\u0026#34;] = ngx.req.get_headers()[\u0026#39;token\u0026#39;], [\u0026#34;cookie\u0026#34;] = ngx.req.get_headers()[\u0026#39;cookie\u0026#39;], [\u0026#34;x-api-key\u0026#34;] = ngx.req.get_headers()[\u0026#39;x-api-key\u0026#39;] } }) local response, decode_err = cjson.decode(res.body) if response then final_backend=response.backend end ngx.var.proxy_upstream_name = ngx.var.namespace .. \u0026#34;-\u0026#34; .. final_backend .. \u0026#34;-\u0026#34; .. ngx.var.service_port ngx.ctx.balancer=nil end return _M As we can see there is a flag up top dynamic_plugin that will be set in each ingress as a configurations snippet to trigger the execution of this script in each request. The plugin code will be written in config-map and will be mounted to ingress controller kubernetes pod as a lua file in the plugins directory. Mounting the plugin # We add the volume mount in out controller deployment by adding following lines in the right place: 1 2 3 4 5 6 7 8 9 10 volumes: - configMap: defaultMode: 420 name: ngx-custom-script name: lua-scripts-volume volumeMounts: - mountPath: /etc/nginx/lua/plugins/modify_request/ name: lua-scripts-volume readOnly: true More about creating a nginx lua plugin here. Running the plugin # Create a new ingress, omitting the lua part and adding a configuration snippet annotation that sets the flag written in the plugin lua. The annotations will be like: 1 2 nginx.ingress.kubernetes.io/configuration-snippet: | set $dynamic_plugin \u0026#34;on\u0026#34;; Now each request sent to the ingress with this variable set will have its route dynamically created as per your requirement. The BE-finder server # This is the backbone of your ingress, so should be as fast as possible. Thus it should be written in efficient languages like Rust, Golang or C/C++. I used Go. The application will use redis or inmemory cache to speed things if your data does not change frequently. If this does not suite your needs, you can setup cron like system that triggers refresh of keys in the cache periodically. The cache miss will trigger a goroutine that fetches the information from database for the next request to pick up. Conclusion # We do not need complicated setup with service meshes to get intelligent routing. Performance will be better than service mesh based solution. Easy to setup and use. ","date":"10 July 2024","externalUrl":null,"permalink":"/posts/request-body-routing-nginx/","section":"Posts","summary":"How do we make ingress nginx intelligent enough to route to different backend not just based on url path match but on the information that request body itself? What if request routing itself requires database access to find the right backend?","title":"Request body aware dynamic routing in ingress nginx using custom plugins.","type":"posts"},{"content":" How we made debugging on cloud easier without costing a fortune. # Preface # Any organization will require good and flexible environment for every developer to test their code before merging it to the main branch.\nThis is normally not required for small-medium monolith application but if the application is too big or consists of a lot microservices and we want to test integration between mutliple microservices. This does not occur often but occurs when you have a feature to implement that will span changes across a lot of microservices. There are other method to test these like writing unit tests for each api of microservice so that it adhers to certain schema and changing the schema as per the changes done and the expected value.\nBut before any release the end to end test will anyhow be necessary.\nThe Problem # All the pushes makes their code live on single shared environment (Testing Env) clashing feature specific tests. If someone pushes faulty code, making a microservice halt, all the other ongoing test will be affected. This will be a major setback if there is some feature that requires quite a bit of time to test one feature. Resulting in Increased time for almost every feature release. Which sometime meant lost first mover advantage. The Solution # Separate resource for each developer or shared resources among a team # This method will work but will be expensive as there will be constant resource consumption if any team forgets to release their resources, unless we think about premption of unused resources which will be difficult to implement or require cloud platform specific implementation or some terraform script to do so. This will also allow team to use the resource allocated to them for something else then work. Its okay if you have enough resources as developers also need some breathing room to test new stuffs but will be difficult to maintain if you already have thousands of dollars of bill from your cloud provider every week. Automatic Creation of required resources and their premption on inactivity the k8s way. # This method was most suitable in our case as separating the resources for different teams to test means less traffic on existing shared Testing Env. This made such that we can decrease the resource footprint of the existing shared env. We settled in the conclusion of creating 12 static namespaces that will house our entire product\u0026rsquo;s deployment. And making a mapping sheet to map teams to those namespaces. Our company has unfettered mania of naming things after things in pokemon so we decided to go with the same flow to name our namespaces and environment by pokemon teams, 12 of them. Setup Minimal permission Make it such that people have view access to the 12 namespaces defined so they can realtime stream container logs or look at the exact config of the deployment or pod. Mapping people to namespaces We wanted to make this part as effortless as possible for the developers along with decreasing our effort of managing the pipeline configuration. So we settled with Google Sheet. We made a sheet with 3 colums: Slot, Usernames, Purpose. The first had list of 12 namespaces we created, second would house github usernames of guys currently claiming that space separated by comma, and the last would be some sort of remarks to indicate why was the slot occupied by that person. The Workflow Someone pushes code to our backend repos. Github workflow is triggred. We used google\u0026rsquo;s gspread client and inlined python script to access the content of the namespace sheet. Installs the backend helm package to the namespace found from the sheet. Below is the actual step in github workflows that we used to interface gsheet with github workflow: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 shell: bash run: |- pip3 install gspread==5.12 oauth2client SLOT_INFO=$(python3 -c \u0026#34; import gspread; from oauth2client.service_account import ServiceAccountCredentials; scope = [ \u0026#39;https://www.googleapis.com/auth/spreadsheets\u0026#39;, \u0026#39;https://www.googleapis.com/auth/drive.file\u0026#39;, \u0026#39;https://www.googleapis.com/auth/drive\u0026#39; ]; creds = ServiceAccountCredentials.from_json_keyfile_name(\u0026#39;$GOOGLE_APPLICATION_CREDENTIALS\u0026#39;, scope); client = gspread.authorize(creds); datas = client.open(\u0026#39;$SHEET_NAME\u0026#39;).sheet1.get_all_records(); print(f\u0026#39;slot mapping data: {datas}\u0026#39;); slot_of_interest = [val for val in datas if \u0026#39;$TRIGGER_EMAIL\u0026#39; in list(map(lambda e: e.strip(), val[\u0026#39;Emails\u0026#39;].split(\u0026#39;,\u0026#39;))) and \u0026#39;$TRIGGER_EMAIL\u0026#39; ]; [print(f\u0026#39;\u0026#39;\u0026#39;{data[\u0026#39;Slot\u0026#39;]}::{data[\u0026#39;Emails\u0026#39;]}\u0026#39;\u0026#39;\u0026#39;)for data in slot_of_interest ]; \u0026#34; ) echo $SLOT_INFO SLOT_INFO=$(echo \u0026#34;$SLOT_INFO\u0026#34; | grep -m 1 -E \u0026#39;::\u0026#39; ) || : if [[ \u0026#34;$SLOT_INFO\u0026#34; == \u0026#34;\u0026#34; ]]; then echo \u0026#34;User not allocated on any namepace. Please update the sheet at https://tinyurl.com/docsumo-namespace with your username ($TRIGGER_EMAIL).\u0026#34; exit 1 fi echo \u0026#34;found slot details of current user. $SLOT_INFO\u0026#34; KUBE_NAMESPACE=$(echo $SLOT_INFO | awk -F\u0026#34;::\u0026#34; \u0026#39;{print $1}\u0026#39; ) IFS=$\u0026#39;,\u0026#39; read -ra EMAILS \u0026lt;\u0026lt;\u0026lt; $(echo $SLOT_INFO | awk -F\u0026#34;::\u0026#34; \u0026#39;{print $2}\u0026#39; ) HOST=${KUBE_NAMESPACE}.docsumo.com echo \u0026#34;HOST=$HOST\u0026#34; \u0026gt;\u0026gt; $GITHUB_ENV echo \u0026#34;KUBE_NAMESPACE=$KUBE_NAMESPACE\u0026#34; \u0026gt;\u0026gt; $GITHUB_ENV echo \u0026#34;final env values; HOST; $HOST, emails; ${EMAILS[@]}, KUBE_NAMESPACE; $KUBE_NAMESPACE\u0026#34; Each namespace will have a cronjob monitoring controller logs and if no request is coming for the last 1 hour, we delete all the resource consuming resources in that namespace, That will include all the deployments and statefulsets. The Premption\nThe premption of resource was handled by a cronjob installed in each namespace that ingested gcloud ingress logs to see if someone was using the namespace. If there is no request logs in the last 1 hour, we delete all the resources of that namespace by doing helm uninstall on all the installed charts. Below is the exact cronjob that we used for this purpose. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 apiVersion: batch/v1 kind: CronJob metadata: name: delete-on-inactivity namespace: {{ .Release.Namespace }} spec: schedule: \u0026#34;0 * * * *\u0026#34; jobTemplate: spec: template: spec: serviceAccount: pod-manager containers: - name: log-checker image: {{ .Values.cronjobimage.repository }}:{{ .Values.cronjobimage.tag }} args: - /bin/bash - -c - | RELEASES=$(helm ls --short -n {{ .Release.Namespace }} | grep -vE \u0026#39;roles\u0026#39;) if [[ ! \u0026#34;$RELEASES\u0026#34; ]]; then echo \u0026#34;Nothing found that can be deleted, exiting.\u0026#34;; exit 0; fi echo $LOG_READER_CREDS | base64 -d \u0026gt; log_reader.json gcloud auth activate-service-account --key-file=log_reader.json gcloud config set project {{ .Values.project_id }} OFFICE_TIME=(3 14) if [[ $(date -u +%-H) -le ${OFFICE_TIME[1]} \u0026amp;\u0026amp; $(date -u +%-H) -ge ${OFFICE_TIME[0]} ]]; then echo \u0026#34;Its office time..\u0026#34; TIME_PERIOD=`date -u -d\u0026#39;2 hour ago\u0026#39; +%Y-%m-%dT%TZ` else TIME_PERIOD=`date -u -d\u0026#39;1 hour ago\u0026#39; +%Y-%m-%dT%TZ` fi gcloud logging read \u0026#34;timestamp \u0026gt;= \\\u0026#34;${TIME_PERIOD}\\\u0026#34; AND resource.type=\\\u0026#34;k8s_container\\\u0026#34; AND resource.labels.cluster_name=\\\u0026#34;testing-cluster\\\u0026#34; AND resource.labels.container_name=\\\u0026#34;controller\\\u0026#34; AND resource.labels.namespace_name=\\\u0026#34;ingress-nginx\\\u0026#34; AND jsonPayload.proxyUpstreamName~=\\\u0026#34;{{ .Release.Namespace }}\\\u0026#34;\u0026#34; --limit 10 --format=json \u0026gt; output.json count=$(jq \u0026#39;. | length\u0026#39; output.json); if [[ $count -eq 0 ]]; then echo \u0026#34;no Logs found in namespace \u0026#39;{{ .Release.Namespace }}\u0026#39; for last one hour, deleting all the resources.\u0026#34; helm ls --short -n {{ .Release.Namespace }} | grep -vE \u0026#39;roles|config\u0026#39; | xargs -I {} helm uninstall {} -n {{ .Release.Namespace }} --debug else echo \u0026#34;logs found on last one hour, not deleting the releases\u0026#34; fi env: - name: LOG_READER_CREDS valueFrom: secretKeyRef: name: google-app-credential-logging key: GOOGLE_APPLICATION_CREDENTIALS_LOGGING restartPolicy: OnFailure Blockers faced in the way We are using external secret operator to sync our backend secrets with GCP\u0026rsquo;s secret manager which when deleted in bulk was failing saying it failed to call some internal webhook. Probably overloading the internal webhook deployment of the operator. We then decided to not delete configmaps and secrets from namespaces as they anyways would not take much resources. Possible enhancements # The next thing we can do is debug totally in cloud. I have another article written about how we can actually remote debug our python code in vscode. Here\nWe setup each deployment to run debugpy instead of normal gunicorn and with a little bit of firewall setup and ip whitelisting, we can actually connect to remote debugger. This allows for debugging in the actual hardware that the production server runs on which helps debug cloud specific issues in some instances\nConclusion # First shield of defence: make each and every microservice autonomous so the need of integration tests are rare. But if your domain does not allow you to do that then this way of making separate test environment would be cheap and effective in the long term of your startup journey. In the end, each developer having access to a vm or multiple vagrant instances would come into picture but when the money and resources are limited, this approach is also not bad.\n","date":"23 June 2024","externalUrl":null,"permalink":"/posts/ephemeral_env/","section":"Posts","summary":"How we made debugging on cloud easier without costing a fortune. # Preface # Any organization will require good and flexible environment for every developer to test their code before merging it to the main branch.\nThis is normally not required for small-medium monolith application but if the application is too big or consists of a lot microservices and we want to test integration between mutliple microservices. This does not occur often but occurs when you have a feature to implement that will span changes across a lot of microservices. There are other method to test these like writing unit tests for each api of microservice so that it adhers to certain schema and changing the schema as per the changes done and the expected value.\n","title":"Managing on demand ephemeral microservice development environment in a single k8s cluster.","type":"posts"},{"content":"","date":"22 June 2024","externalUrl":null,"permalink":"/tags/container/","section":"Tags","summary":"","title":"Container","type":"tags"},{"content":" Debugging python inside of a docker container # Premises # Lets say you are working on some legacy python project which has very obscure and obsolete dependencies and you now want to debug it. First lets pray to god that, this day does not come to anybody but if you are working on some project without standard practices on package management, this kind of situation is not uncommon.\nThe Problem # Making a local dev envs.\nEven though pyenv and poetry to name a few is able to install different versions of python along with old deps, what if the packages are so old even pip does not have them. A few old python packages do delete old version from pipy instead of yanking them that will cause the old programs to even stop running in the near future. The next problem is bloat if you have multiple of these shitty projects then you would not want your whole system to endure the cost of multiple old and obsolete python versions in the long run. Lets admit it nobody remembers to uninstall the things that we installed just to make things to working, instead we leave it in our system till we get out of space or some other things break due to that. The latter case will be the most troublesome because it might take another full day just to know the reason why your new program is not working and that the problem was you having an old version of something else. The solution # My solution to this problem is to debug inside of a container. You replicate the environment that your system was running on, inside of a container.\nThis is a breeze if you already had your program containerized.\nGet the container that has been running this whole time in production. Either docker pull or form sysadmin. Here is one of the answer on how to load the image to your local docker instance if you don\u0026rsquo;t have access to docker pull image: https://stackoverflow.com/questions/37905763/how-do-i-download-docker-images-without-using-the-pull-command But if you had not done so before, you need answers to following things from your sysadmin or anyone who has access to the running system:\nWhat is the OS? if Its Windows, Goodluck on that. Go the usual pyenv/poetry route. The output of pip freeze and python version. Get hands on requirements.txt. Now containerize your application.\nMake a simple docker image having following steps: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 ## The name and versions should be the one from your production system. ## It should be available in dockerhub as they normally have older images too. FROM debian:\u0026lt;tag\u0026gt; ## Install the required libraries. apt is for debian. RUN apt update \u0026amp;\u0026amp; apt install ..... ## Install debugpy to initialize remote debugging session. RUN pip install debugpy COPY ./requirements.txt /requirements.txt RUN pip install -r requirements.txt ## Some other pip installs you have some custom python packages to install. ## This so that you can get shell access to the docker without doing docker exec anything CMD [\u0026#34;bash\u0026#34;] If you have a bit of complex program requirements like requiring redis or some other things then you can use a docker-compose.yaml file. This was the case in my case. Feel free to take reference from dockercompose below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 services: dev-container: ## I have this because I have access to do docker pull. ## If you have to build from dockerfile, you can give this any name, ## then uncomment the build part below image: us-docker.pkg.dev/\u0026lt;project_id\u0026gt;/us.gcr.io/\u0026lt;image_name\u0026gt;:\u0026lt;image_tag\u0026gt; # build: # context: . # dockerfile: Dockerfile # args: # - GITLAB_DEPLOY_TOKEN=\u0026lt;GITLAB_TOKEN\u0026gt; ## I needed this because I had custom old package to install from gitlab\u0026#39;s package registry. volumes: - ./app:/app - ./service_account.json:/srv/key.json ## this was required to access google specific resource inside the program env_file: - \u0026#34;.env\u0026#34; environment: - GOOGLE_APPLICATION_CREDENTIAL_PATH=/srv/key.json - FLASK_APP=app.app:app ## uncomment this instead of network_mode : host if you have the ports inside the container occupied. # ports: # - 8080:8080 network_mode: \u0026#34;host\u0026#34; stdin_open: true tty: true command: [\u0026#34;bash\u0026#34;] ## my program required redis too. redis-master: image: redis:latest # ports: # - \u0026#34;6379:6379\u0026#34; command: redis-server --appendonly yes --requirepass 123456789 network_mode: \u0026#34;host\u0026#34; env_file: - \u0026#34;.env\u0026#34; Debugpy to rescue # Debugpy a implementation of a Debug Adapter Protocol for python which is protocol developed my microsoft that defines abstract rules of communication between a IDE or text editor and the debugger. (PDB in our instance) Its a remote debugging application developed by microsoft and used in vscode for debugging. We then do docker run --name dev-container --env-file .env --network host -it -v \u0026lt;local_source_code_path\u0026gt;:\u0026lt;path_inside_container\u0026gt; \u0026lt;image_name\u0026gt; Now that your docker container is running, you can attach your current shell to the container: docker attach \u0026lt;container_name\u0026gt; container name is the name you gave while doing docker run. If you had skipped that part, docker will give you a random name. To get that name, you do docker ps to show currently running container and get the random name from that. Running debugpy inside the container # Once you got inside of the shell of the running container, you run the program. python -m debugpy --listen 0.0.0.0:5678 -m flask run --host='0.0.0.0' --port=8080 This is for flask application but if you are running something else modify it accordingly. For more information on debugpy and its options view its github page. Connecting your vscode instance with the debugpy running inside of the container # Open vscode in the dir you mounted or the project dir. Make a new launch.json or make vscode make it automaticaly for you. Click on Create a launch.json file \u0026gt; Python Debugger \u0026gt; Remote Attach. Leave the hostname on localhost and port to 5678 because we set it while launching debugpy in previous step. We are keeping hostname to localhost and port to 5678 because we have --network host while running docker container. If 5678 is somehow occupied in your system then forward the port 5678 to some other available port like 9874 in your system by using -p \u0026lt;local_port\u0026gt;:\u0026lt;container_port\u0026gt; while doing docker run. Now you will get something like this in your launch.json 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 \u0026#34;version\u0026#34;: \u0026#34;0.2.0\u0026#34;, \u0026#34;configurations\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;Python Debugger: Remote Attach\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;debugpy\u0026#34;, \u0026#34;request\u0026#34;: \u0026#34;attach\u0026#34;, \u0026#34;connect\u0026#34;: { \u0026#34;host\u0026#34;: \u0026#34;localhost\u0026#34;, \u0026#34;port\u0026#34;: 5678 }, \u0026#34;pathMappings\u0026#34;: [ { \u0026#34;localRoot\u0026#34;: \u0026#34;${workspaceFolder}\u0026#34;, \u0026#34;remoteRoot\u0026#34;: \u0026#34;.\u0026#34; } ] } ] } Fix the Path mappings with the paths you had while doing docker run in -v part. I had -v ./app:/app so I will change the pathMapping to something like: 1 2 3 4 5 6 7 8 { \u0026#34;pathMappings\u0026#34;: [ { \u0026#34;localRoot\u0026#34;: \u0026#34;${workspaceFolder}\u0026#34;, \u0026#34;remoteRoot\u0026#34;: \u0026#34;.\u0026#34; } ] } Now while the container and the program you want to debug inside is running with debugpy, you set breakpoints like you normally used to do clicking beside the line numbers. Press F5 to launch the debugging session and try to run the program normally. You can send the request to the server if its http server or something like that. Once the code execution hits breakpoint, your program will pause automatically at the breakpoint and you can inspect variables, invoke functions and others. ","date":"22 June 2024","externalUrl":null,"permalink":"/posts/debugpy/","section":"Posts","summary":"Debugging python inside of a docker container # Premises # Lets say you are working on some legacy python project which has very obscure and obsolete dependencies and you now want to debug it. First lets pray to god that, this day does not come to anybody but if you are working on some project without standard practices on package management, this kind of situation is not uncommon.\n","title":"Debugging a python program inside of a container","type":"posts"},{"content":"","date":"22 June 2024","externalUrl":null,"permalink":"/categories/docker/","section":"Categories","summary":"","title":"Docker","type":"categories"},{"content":"Hi, I\u0026rsquo;m Ankit 👋\nI\u0026rsquo;m a platform / SRE engineer at Docsumo, a document-AI company, working from Kathmandu, Nepal. My day job is keeping a fleet of Kubernetes clusters and Python/Go services happy while they chew through millions of documents a month — and building the tooling so other engineers can ship without thinking about the plumbing underneath.\nI like the messy middle of infrastructure: the part where \u0026ldquo;it works on my machine\u0026rdquo; meets \u0026ldquo;it has to work for everyone, at 3 a.m., during a node failure.\u0026rdquo; Most of what I write on this blog comes from that place.\nDownload CV Things I\u0026rsquo;ve been up to at Docsumo # Rebuilt our document-processing pipeline into a distributed system that comfortably moves millions of documents a month. Helped the team move critical workflows onto Temporal — most of our clients run on it now, and it fails far more gracefully than what it replaced. Built on-demand ephemeral dev environments that spin up a full product deployment when you need one and quietly tear it down when you don\u0026rsquo;t — I wrote about how. Moved live production traffic from Ingress to the Gateway API without anyone noticing. Zero downtime — the good kind of boring. Look after observability (Prometheus + Thanos — there\u0026rsquo;s a post) and storage (Longhorn — that one too). Run a self-hosted Redis Sentinel cluster that shrugs off node failures and upgrades. Built a feature-flag service on the OpenFeature spec so blue-green releases stopped being scary. Made credential rotation routine across our cloud workloads, and built sandboxes that can safely run arbitrary Python. Carry the pager, define the SLIs, keep the SLAs honest. How I use AI # I\u0026rsquo;m not training foundation models — I\u0026rsquo;m the person who wires AI into everyday engineering work and actually gets value out of it:\nAgentic CLI tools (Claude Code and friends) are my daily driver for writing code, debugging, and poking at infrastructure — with guardrails in place so nothing touches production without a human saying so. LLMs during incidents — log analysis, hypothesis generation, and the endless one-off scripts that ops work produces. The underlying tech isn\u0026rsquo;t a black box to me either: I did a machine-learning micro-degree at FuseMachines and NAAMI\u0026rsquo;s Winter School in AI, back before any of this was cool. Where I\u0026rsquo;ve worked # Docsumo · Remote / Singapore — joined in 2023 as a DevOps/SRE engineer taming multi-tenant Kubernetes; since 2025 I\u0026rsquo;m the senior platform/SRE engineer. Along the way: migrated the whole org from GitLab to GitHub (and halved the tooling bill), cut database latency by about a third through data-model rework, and set up autoscaling with Keda/HPA that actually tracks load.\nYasok Systems · Kathmandu — backend engineer from 2021 to 2022. Built site-wide search with Python + Meilisearch, made a Node.js ERP backend noticeably faster with Redis caching, and put together an async notification system on AWS SQS + Lambda.\nFuseMachines · Kathmandu — AI fellow in 2021, where the ML micro-degree happened.\nBeyond work # Things I\u0026rsquo;ve built for fun — a 3D renderer, algorithm visualizers, a tank game — live on the projects page.\nI studied Computer Engineering at IOE Pulchowk Campus (2018–2023). During that time I also taught Docker and cloud tooling in a one-month software fellowship — turns out explaining containers to a room full of people is the fastest way to properly understand them yourself.\nCertifications, for the record: AWS Cloud Foundations · FuseMachines ML micro-degree · NAAMI Winter School in AI · IBM Quantum\u0026rsquo;s \u0026ldquo;Qbit by Qbit\u0026rdquo; intro to quantum computing.\nSay hi # The inbox is always open: ankitpaudel20000@gmail.com. I\u0026rsquo;m also around on GitHub, LinkedIn, and Twitter/X. And if you\u0026rsquo;re hiring — the CV button up top has the formal version of this page.\n","externalUrl":null,"permalink":"/about/","section":"Ankit Paudel","summary":"Hi, I’m Ankit 👋\nI’m a platform / SRE engineer at Docsumo, a document-AI company, working from Kathmandu, Nepal. My day job is keeping a fleet of Kubernetes clusters and Python/Go services happy while they chew through millions of documents a month — and building the tooling so other engineers can ship without thinking about the plumbing underneath.\nI like the messy middle of infrastructure: the part where “it works on my machine” meets “it has to work for everyone, at 3 a.m., during a node failure.” Most of what I write on this blog comes from that place.\n","title":"About","type":"page"},{"content":"Things I\u0026rsquo;ve built outside of work — mostly to understand something from first principles, mostly before my professional career. The cards below are live from GitHub.\n3D renderer from scratch # C++ and OpenGL. Renders arbitrary .obj models, and includes a pure-software rasterizer that implements the primitive triangle-drawing algorithms low-level graphics drivers use — written to understand what the GPU is actually doing for you.\nankitpaudel20/simple3drenderer A simple 3d scene renderer in opengl C\u0026#43;\u0026#43; 0 0 Algorithms visualizer # C++ and SDL2. A teaching aid that animates sorting, pathfinding, and spanning-tree algorithms step by step.\nankitpaudel20/Algorithms-Visualizer C\u0026#43;\u0026#43; 2 0 Later I rewrote parts of it in Go, as my first excuse to learn the language properly:\nankitpaudel20/algoviz-go First go project to learn around with go. Go 0 0 Realtime text similarity # Python, Flask, and scikit-learn — my college final project. Uses sentence transformers (plus simpler methods like smooth inverse frequency) to score how similar two pieces of text are, built to help professors avoid repeating exam questions.\nankitpaudel20/realtime-text-similarity-frontend Realtime text Similarity Identification null 1 0 Pocket Tanks clone # C++, SFML, and Box2D. A physics-based artillery game clone — projectile arcs, destructible terrain, the works.\nankitpaudel20/Pocket-Tanks-Clone-in-SFML C\u0026#43;\u0026#43; 0 0 This machine, declared # My entire home environment — shell, editor, tools — as reproducible Nix configuration. It\u0026rsquo;s also how this site\u0026rsquo;s dev shell works.\nankitpaudel20/dotfiles home directory config Nix 0 0 More experiments live on my GitHub profile.\n","externalUrl":null,"permalink":"/projects/","section":"Ankit Paudel","summary":"Things I’ve built outside of work — mostly to understand something from first principles, mostly before my professional career. The cards below are live from GitHub.\n3D renderer from scratch # C++ and OpenGL. Renders arbitrary .obj models, and includes a pure-software rasterizer that implements the primitive triangle-drawing algorithms low-level graphics drivers use — written to understand what the GPU is actually doing for you.\n","title":"Projects","type":"page"}]