Open in app

Sign In

Write

Sign In

Itay Bittan
Itay Bittan

123 Followers

Home

About

Published in Apache Airflow

·Dec 13, 2022

Migrating Airflow backend database from MySql to PostgreSQL

Dynamic Yield recently completed the migration of our backend database from MySql to PostgreSQL. This post will detail why we migrated and how, and what you should consider in this process. Background Dynamic Yield engineers are Airflow enthusiastic fans! We have several Airflow clusters for managing various data pipelines (building deep-learning…

Airflow

8 min read

Migrating Airflow backend database from MySql to PostgreSQL
Migrating Airflow backend database from MySql to PostgreSQL
Airflow

8 min read


Published in Apache Airflow

·Jul 7, 2022

Airflow’s Magic Loop

A simple optimization that saves us a significant running time — Here at Dynamic Yield, we use several various Airflow clusters for managing a lot of different pipelines. Each of those clusters runs tens of thousands of tasks on a daily basis. …

Airflow

4 min read

Airflow’s Magic Loop
Airflow’s Magic Loop
Airflow

4 min read


Published in ITNEXT

·Jun 17, 2022

Wrong Spark configuration that cost us $3k/month

Spark version: 3.2.0 Here at Dynamic Yield, we run thousands of Spark applications every day. Over the last two years, we worked hard to upgrade our infrastructure to run those jobs on Kubernetes and we continuously improve our system. One of the innocent configurations that we added a few months…

Spark

4 min read

Wrong Spark configuration that cost us $3k/month
Wrong Spark configuration that cost us $3k/month
Spark

4 min read


Published in AWS Tip

·Dec 29, 2021

AWS S3 data-source for Grafana

Enhance visibility to your data — Here at Dynamic Yield we use AWS S3 quite a lot! A few months ago we moved our Spark infrastructure from EC2 (in stand-alone mode) to Kubernetes (cluster mode). …

S 3

4 min read

AWS S3 data-source for Grafana
AWS S3 data-source for Grafana
S 3

4 min read


Published in ITNEXT

·Mar 20, 2021

HTTP Compression - boost your server’s speed

Improve latency and cost by enabling web-server compression — We are all working really hard for making an impact and once in a while there’s an opportunity for making a HUGE impact by a really small effort. After reading this post you’ll be able to make a fast impact on your web-servers and improve your site’s performance! REST API…

Web Development

7 min read

HTTP Compression - boost your server’s speed
HTTP Compression - boost your server’s speed
Web Development

7 min read


Published in Towards Data Science

·Mar 8, 2021

Jupyter Notebook & Spark on Kubernetes

The complete guide for setting up your local environment — Jupyter notebook is a well-known web tool for running live code. Apache Spark is a popular engine for data processing and Spark on Kubernetes is finally GA! In this tutorial, we will bring up a Jupyter notebook in Kubernetes and run a Spark application in client mode. We will also…

Kubernetes

5 min read

Jupyter Notebook & Spark on Kubernetes
Jupyter Notebook & Spark on Kubernetes
Kubernetes

5 min read


Published in The Startup

·Jan 30, 2021

High Scale Web Server With Kubernetes - Part II

This is the second part of High scale web server with Kubernetes. We will go over Kubernetes Horizontal Pod Autoscale and how we are using it at Dynamic Yield. Overview While serving a huge amount of requests, we can easily observe that our traffic graph looks like a sine wave with…

Kubernetes

6 min read

High Scale Web Server With Kubernetes - Part II
High Scale Web Server With Kubernetes - Part II
Kubernetes

6 min read


Published in The Startup

·Jan 23, 2021

High Scale Web Server With Kubernetes

Here at Dynamic Yield we are serving thousands of HTTP requests per second. Moving our serving services from EC2 to EKS required some tuning to ensure we could serve efficiently without losing any request. I summarized some tips and I hope it might help others with a smooth transition. Overview Kubernetes…

Kubernetes

5 min read

High Scale Web Server With Kubernetes
High Scale Web Server With Kubernetes
Kubernetes

5 min read


Published in DynamicYield Tech Blog

·Jan 15, 2021

HPA for Celery workers

Celery is a simple and reliable distributed system to process tasks. At Dynamic Yield, we use Celery to orchestrate and distribute work across pods in Kubernetes. Celery communicates via messages, usually using a broker to mediate between clients and workers. We use RabbitMQ as the broker. In our system, we…

Kubernetes

4 min read

HPA for Celery workers
HPA for Celery workers
Kubernetes

4 min read


Dec 29, 2020

Restarting Elasticsearch node — the safe way

Elasticsearch is being used in a lot of companies as a great search-engine thanks to its speed and scale. At Dynamic Yield, we are using Elasticsearch as part of our recommendations engine and handle thousands of requests per second. Terminology Elasticsearch cluster contains several nodes that can play one or more…

Elasticsearch

5 min read

Restarting Elasticseach node — the safe way
Restarting Elasticseach node — the safe way
Elasticsearch

5 min read

Itay Bittan

Itay Bittan

123 Followers

Dad | Husband | Principal Software Engineer at Dynamic Yield | Tech Geek | https://www.linkedin.com/in/itaybittan

Following
  • Tim Denning

    Tim Denning

  • Martin Heinz

    Martin Heinz

  • Pinterest Engineering

    Pinterest Engineering

  • Tinder

    Tinder

  • Leah Cole

    Leah Cole

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech