Jupyter notebook is a well-known web tool for running live code. Apache Spark is a popular engine for data processing and Spark on Kubernetes is finally GA! In this tutorial, we will bring up a Jupyter notebook in Kubernetes and run a Spark application in client mode. We will also use a cool sparkmonitor widget for visualization. Additionally, our Spark application will read some data from AWS S3 that will simulate that locally with localstack S3.

This is the second part of High scale web server with Kubernetes. We will go over Kubernetes Horizontal Pod Autoscale and how we are using it at Dynamic Yield.

Overview

While serving a huge amount of requests, we can easily observe that our traffic graph looks like a sine wave with a high rate at midday and a lower rate at night. The difference is relatively big, around 2–3 times more requests in the rush hours. Moreover, there are special occasions such as Black Friday, Cyber Monday, sale campaigns, etc, that our traffic can raise up to x3.

Using Kubernetes’s elasticity…

Here at Dynamic Yield we are serving thousands of HTTP requests per second. Moving our serving services from EC2 to EKS required some tuning to ensure we could serve efficiently without losing any request. I summarized some tips and I hope it might help others with a smooth transition.

Overview

Kubernetes can decide to proactively replace pods (replicas) of our application, whenever:

• Kubernetes node is replaced (if the node is spot instance). If you have running pods on that node — they will be terminated and new pods will start in another node instead.
• Kubernetes nodes scale-in. Pods in this node…

Celery is a simple and reliable distributed system to process tasks. At Dynamic Yield, we use Celery to distribute work across pods in Kubernetes.

Celery communicates via messages, usually using a broker to mediate between clients and workers. We use RabbitMQ as the broker.

In our system, we have ~30 different celery tasks such as:

• Reading data from Elasticsearch.
• Indexing data to Elasticsearch.
• Reading data from S3.
• Writing data to Redis.
• etc

There’s a huge variety between those tasks: some of them can run for seconds while others can take hours, depending on the data (size) being processed and the…

Elasticsearch is being used in a lot of companies as a great search-engine thanks to its speed and scale. At Dynamic Yield, we are using Elasticsearch as part of our recommendations engine and handle thousands of requests per second.

Terminology

Elasticsearch cluster contains several nodes that can play one or more roles: master, data, ingestion, etc.

Indices are being stored in the data nodes.

Each index can have one or more shards, usually determined by the index size.

Each shard holds part of the index data (documents), so if for example, we have an index with two shards, each shard holds…