Published in Apache Airflow·Dec 13, 2022Migrating Airflow backend database from MySql to PostgreSQLDynamic Yield recently completed the migration of our backend database from MySql to PostgreSQL. This post will detail why we migrated and how, and what you should consider in this process. Background Dynamic Yield engineers are Airflow enthusiastic fans! We have several Airflow clusters for managing various data pipelines (building deep-learning…Airflow8 min readAirflow8 min read
Published in Apache Airflow·Jul 7, 2022Airflow’s Magic LoopA simple optimization that saves us a significant running time — Here at Dynamic Yield, we use several various Airflow clusters for managing a lot of different pipelines. Each of those clusters runs tens of thousands of tasks on a daily basis. …Airflow4 min readAirflow4 min read
Published in ITNEXT·Jun 17, 2022Wrong Spark configuration that cost us $3k/monthSpark version: 3.2.0 Here at Dynamic Yield, we run thousands of Spark applications every day. Over the last two years, we worked hard to upgrade our infrastructure to run those jobs on Kubernetes and we continuously improve our system. One of the innocent configurations that we added a few months…Spark4 min readSpark4 min read
Published in AWS Tip·Dec 29, 2021AWS S3 data-source for GrafanaEnhance visibility to your data — Here at Dynamic Yield we use AWS S3 quite a lot! A few months ago we moved our Spark infrastructure from EC2 (in stand-alone mode) to Kubernetes (cluster mode). …S 34 min readS 34 min read
Published in ITNEXT·Mar 20, 2021HTTP Compression - boost your server’s speedImprove latency and cost by enabling web-server compression — We are all working really hard for making an impact and once in a while there’s an opportunity for making a HUGE impact by a really small effort. After reading this post you’ll be able to make a fast impact on your web-servers and improve your site’s performance! REST API…Web Development7 min readWeb Development7 min read
Published in Towards Data Science·Mar 8, 2021Jupyter Notebook & Spark on KubernetesThe complete guide for setting up your local environment — Jupyter notebook is a well-known web tool for running live code. Apache Spark is a popular engine for data processing and Spark on Kubernetes is finally GA! In this tutorial, we will bring up a Jupyter notebook in Kubernetes and run a Spark application in client mode. We will also…Kubernetes5 min readKubernetes5 min read
Published in The Startup·Jan 30, 2021High Scale Web Server With Kubernetes - Part IIThis is the second part of High scale web server with Kubernetes. We will go over Kubernetes Horizontal Pod Autoscale and how we are using it at Dynamic Yield. Overview While serving a huge amount of requests, we can easily observe that our traffic graph looks like a sine wave with…Kubernetes6 min readKubernetes6 min read
Published in The Startup·Jan 23, 2021High Scale Web Server With KubernetesHere at Dynamic Yield we are serving thousands of HTTP requests per second. Moving our serving services from EC2 to EKS required some tuning to ensure we could serve efficiently without losing any request. I summarized some tips and I hope it might help others with a smooth transition. Overview Kubernetes…Kubernetes5 min readKubernetes5 min read
Published in DynamicYield Tech Blog·Jan 15, 2021HPA for Celery workersCelery is a simple and reliable distributed system to process tasks. At Dynamic Yield, we use Celery to orchestrate and distribute work across pods in Kubernetes. Celery communicates via messages, usually using a broker to mediate between clients and workers. We use RabbitMQ as the broker. In our system, we…Kubernetes4 min readKubernetes4 min read
Dec 29, 2020Restarting Elasticsearch node — the safe wayElasticsearch is being used in a lot of companies as a great search-engine thanks to its speed and scale. At Dynamic Yield, we are using Elasticsearch as part of our recommendations engine and handle thousands of requests per second. Terminology Elasticsearch cluster contains several nodes that can play one or more…Elasticsearch5 min readElasticsearch5 min read
