Adam Yells at Cloud

SRERC

2022-11-18T21:34:00-06:00

My collection of shareable code, snippets, documentation, etc. https://github.com/abkierstein/srerc

I decided to host a few sharable tips & tricks I’ve picked up outside of this blog in the srerc repo.

Check it out for references like…

Blog Init

2021-01-17T15:17:00-06:00

Yet another tech blog…

I’ve wanted to start a blog as a place to document things I couldn’t easily find answers to on the internet. I’d also like to use this opportunity to document personal projects and potentially rant about this and that. This post falls under documenting a personal project as there’s nothing here I’m covering that isn’t mostly covered in references below.

Blog backend

I specifically searched for a static hosting technology for the blog so I could serve something out of s3 and not worry about a complex site backend. A short amount of research later and jekyll seemed to be:

Widely used
Well documented
Allows pages to be written in markdown

So far, I’m fairly happy with jekyll. There’s a bit of learning curve to it and I claim no mastery of it yet but this blog is as good enough as a reason as any to kick the tires on it.

Stack Overview

Jekyll: static website technology
S3: AWS hosted static object store for jekyll contents
Cloudfront: AWS hosted distribution network used to front S3 and for SSL termination.

As of writing this post, the blog and all of its infrastructure are publicly visible on my Gitlab. Along with the desire to host a static site in s3, I wanted the site to have a valid SSL cert and of course it was going to be defined in Terraform.

While I didn’t find an end-to-end guide on how to achieve my goal here, the blogs below helped me out a good amount:

I may write up a full guide on how I’ve implemented the blog once I get a better handle on Jekyll.

Getting Indexed

I’m showing my hand a little bit here but this is the first site I’ve hosted for public consumption. I didn’t even know I needed to opt-in to google’s indexing… The link below was a huge help in getting this setup:

victor2code.github.io

This view into a site’s performance and usage is super interesting. I didn’t expect to get so much data or insight into the site and am kinda tickled about it. The screenshot below shows some of the information provided.

AWS Elasticsearch Maximum Shards Per Index

2021-01-11T19:30:00-06:00

The intent of this post is to help answer the question: How many shards per index can a multi-az/zone-aware AWS Elasticsearch with ‘routing.allocation.total_shards_per_node’ enabled handle? Heck of a question, I’m not sure of another way to phrase that as a one-liner…

As you may already know visiting this post, with the elasticsearch index setting ‘routing.allocation.total_shards_per_node’, assigning more shards than can be routed in the cluster will result in an index going yellow (replica shards can’t be allocated) or even red (if primary shards can’t be allocated).

Background: I use AWS ES as the E in an ELK stack. My log indices can get REALLY busy and since AWS ES severly limits options on manually routing/balancing shards, using ‘routing.allocation.total_shards_per_node’ is one way I can distribute R/W load among the nodes of the cluster. I also operate this cluster AZ aware across 3 AZs, so AWS ES auto-configures zone/shard awareness and I’ve yet to find a good white paper on how AWS ES specifically implements this shard awareness in opendistro this is about as good as I’ve found.

Problem: In a multi-AZ AWS ES cluster, what is the maximum number of shards I can request in an index and have AWS allocate them appropriately?

This deserves some testing…

Goal: Create a repeatable formula that tells me exactly how many shards I can request from a multi-az aware ES cluster.

Approach: Create real multi-AZ zone aware clusters, and with an index setting of ‘routing.allocation.total_shards_per_node: 1’, create indices with more and more shards until an index when yellow. At that point, I know with a cluster of size X, an index can have Y - 1 shards and allocate primary and replicas just fine.

Here’s an example CURL to create an index

curl --request PUT 'http://<yourclusteraddresshere>:<port>/index-foo' \
--header 'Content-Type: application/json' \
--data-raw '
{
    "settings" : {
        "index" : {
            "number_of_shards" : 5,
            "routing.allocation.total_shards_per_node": 1,
            "number_of_replicas" : 1

        }
    }
}
'

Time to break out the spreadsheet…

First attempt I went a little off the deep end with my theory… It didn’t hold up on trial 2 but a new formula was derived which did!

Result:

(Number of Nodes / 2) * Shards Per Node

# Or what I assume actually is...
(Number of Nodes / (1 primary shard + N replica shards)) * Shards Per Node

So given a cluster of 40 nodes, I could request a maximum of 20 shards before the index went yellow. Please note that increasing shards adds additional lucine indices for ES to keep track of, and there’s a balancing point I won’t get into regarding optimal & consistent sizes for ALL indices in your cluster.

Documentation dump:

Hello, World!

2021-01-09T18:44:58-06:00

Example post

#!/bin/sh
echo "Hello, World!"