 

Metadata and meta tags are underestimated SEO optimization tools. Newer web developers may completely gloss over this powerful technique to improve the discoverability of the websites for which they are responsible. One might discover the lacking of meta tags in a website via tools like <Anchor href="https://www.checkbot.io/">checkbot.io</Anchor>, or one might realize when they share a link to their website, an image doesn’t show up with the link on Twitter or Facebook like other websites! Why is that?

![Vercel OG Image Generation](https://res.cloudinary.com/bytesizedpieces/image/upload/v1672883067/article/og-dynamic-image/og_image_generation_kzk5dv.png)

In this article, we will focus on a few critical concepts before diving into NextJS new open graph dynamic image updates. In order to truly grasp the incredible benefits of this released feature, we need to have a solid understanding of the following:

1.  Metadata meaning and meta tags
2.  Open graph protocol
3.  Shortcomings with the standard method of adding image meta tags
4.  Open graph dynamic images with NextJS

## Metadata meaning and meta tags

To first understand why we care about the recent NextJS release, we need to have a background for the notion of metadata, what it means, and how meta tags fall under the metadata scope.

![What is Metadata?](https://res.cloudinary.com/bytesizedpieces/image/upload/v1672882529/article/og-dynamic-image/metadata_k5fl6r.png)

### What is Metadata?

**Generally speaking, metadata is defined as data that describes data.** A more helpful way to understand metadata is by examining its application in specific areas. In this case, let’s understand metadata when applied to web development.

**In web development, metadata consists of unseen HTML elements that serve the purpose of communicating and clarifying relevant website information for search engines like Google**, ultimately making it a critical role in effective Search Engine Optimization.

Metadata still feels a bit abstract, and this is where meta tags come in to clear up the picture and give it a bit more structure.

![What is a Meta Tag?](https://res.cloudinary.com/bytesizedpieces/image/upload/v1672882529/article/og-dynamic-image/metatag_rktwkv.jpg)

Meta tags help us accomplish the goal of communicating and clarifying the relevant website information for our search engines. **Meta tags are the actual snippets of text that describe a page’s content.** They don’t appear on the page itself, but they are present in the page’s source code for crawlers to parse.

How do we create a meta tag on a website? Let’s examine an example of the usage of meta tags in <Anchor href="http://bytesizedpieces.com/">ByteSizedPieces</Anchor>!

```jsx
<Head>
    <title>ByteSizedPieces blog posts about ReactJS web development</title>
    <meta name="description" content="A web development blog focused on interview preparation, javascript, css, react, accessibility, web development tools, mobile development, and the tech industry." />
</Head>
```

In the above example, we have a few details to make note of.

1.  The `<head>` element we have utilized serves as a container for our metadata and is placed between the `<html>` tag and the `<body>` tag. As such, any metadata should be positioned as children of the `head` element!

2. The `meta` element is our meta tag! Note how we have some attributes, names, and content which tell the search engine crawlers that the meta tag is specifically intending to provide a description of our website, as well as the supplemental content which is the actual description.

<Anchor href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta/name">
  Meta tags can have several names and provide a range of information.
</Anchor> We can see a full list of the meta tags to a site via inspection of developer
tools.

Looking at Medium, we can see many meta tags set within the `head` element! This explains why Medium does so well in the SEO space!

![Medium Meta Tags](https://res.cloudinary.com/bytesizedpieces/image/upload/v1672882564/article/og-dynamic-image/medium_metatag_brjmef.png)

## Open Graph Protocol

The open graph protocol is another metadata extension that gives websites a unique stance in the social graph. More specifically, whenever you share a URL on social media, Open Graph meta tags (OGP tags for short), inform what supplemental information comes with that URL. It could be the title, description, or in many cases an image.

On Twitter, a supplemental image influences engagement statistics by leaps and bounds. Retweets increased by 150%, clicks increased by 20%, and favorites increased by 90%.

![Engagement increases with images!](https://res.cloudinary.com/bytesizedpieces/image/upload/v1672882702/article/og-dynamic-image/engagement_increase_a54yra.jpg)

The numbers do not lie, and images as metadata are important!

How do we typically communicate the image with the URL? Well, open graph is intended to be generalized across social media platforms. However, one needs to stay attentive to potential unique handling. <Anchor href="https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/abouts-cards">Twitter Cards</Anchor>, for instance, look similar to Open Graph tags, but still deviate from the rulesets of OGP!

To avoid complications, we focus our attention on the general OGP conventions. To turn your own web page into a graph object, you would define a meta tag that follows the rulesets defined by the social media platform. For Facebook, the meta tag for an image would look like the following.

```
<meta property="og:image" content="" />
```

In conjunction with the importance of image metadata with shared URLs on social media, we should note the impact an aesthetic image makes versus one with low quality or visual relevance to the text.

This leads me to my final review topic prior to exploring NextJS OG Dynamic Image release.

# Shortcomings with the standard method of adding image meta tags

We have established that images drive better conversion rates and the quality of images matters. The unfortunate fact is, with large blogs or e-commerce sites, a lot of effort goes into asset creation. The standard method of adding image meta tags requires individually creating images per page. This has proven to be inefficient and a waste of developer and designer time. If only things could be standardized!

![Massive Waste of Time Sign](https://res.cloudinary.com/bytesizedpieces/image/upload/v1672882725/article/og-dynamic-image/waste_of_time_zv887c.jpg)

In comes Vercel with their new amazing release for NextJS. Now we can see the beauty behind OG Dynamic Images.

## Open graph dynamic images with NextJS

First we answer the following question.

### What are Open Graph Dynamic Images?

Open graph or OG dynamic images enable social images to be generated on the fly per developer specification. These images can be configured to include dynamic properties, unlocking asset customization as well. Finally, the image generation runs on the <Anchor href="https://vercel.com/docs/concepts/functions/edge-functions">vercel edge</Anchor>, to deliver rapid results based on user local, and is combined with caching for faster results.

Now that we understand what OG dynamic images are, let’s go over how we can create a new API for OGImage and leverage it in our website.

Before we begin, a gentle reminder that this feature is only available to projects that have been brought to version 13 of NextJS, and is still experimental.

### Steps to leverage OG Dynamic Images

1.  Install <Anchor href="https://www.npmjs.com/package/@vercel/og">@vercel/og</Anchor> npm package
2.  Create a new API for OGImage
3.  Use new API

## Install vercel/og npm package

```
npm i --save @vercel/og
```

replace npm with `yarn` or `pnpm` as needed.

## Create a new API for OG Image

First let’s decide what we are going to create.

Our asset is intended for to blog articles. When we share a link on social media to our blog post, ideally we would like to see an image with a background color, our blogs name, the title of the article and the category! OG Dynamic Images will come in handy here, as we will be able to set the dynamic properties for our images, and never have to look back!

Under `pages/api/*` we will want to create a new file `og.js`

Here we will import the `ImageResponse` object from `@vercel/og` . This object is responsible for processing our provided HTML and CSS, and convert that into an appropriate image which the endpoint will serve.

Recall the mention that this process is experimental, therefore we need to enable the experimental edge temporarily via:

```js
export const config = {
  runtime: 'experimental-edge',
};
```

Finally, let’s create a `handler` that takes in a request object, this process should be familiar to any <Anchor href="https://nextjs.org/docs/api-routes/introduction">NextJS endpoint</Anchor>.

Now our `pages/api/og.js` file should look like the following:

```js
// pages/api/og.js

import { ImageResponse } from '@vercel/og';

// enable experimental edge

export const config = {  
  runtime: 'experimental-edge',  
};

export default function handler(req: NextRequest) {}
```

Now we want to define some HTML and CSS and pass it to our imported `ImageResponse` object. For sake of simplicity, we will statically set filler text for the intended dynamic article title and category.

Below is the following rendering:

![Image Generated at the Vercel Edge with Static Fields](https://res.cloudinary.com/bytesizedpieces/image/upload/v1672882754/article/og-dynamic-image/og_image_edge_hljizx.png)

And below is how we did it via HTML and CSS passed to `ImageResponse`.

```js
// pages/api/og.js

import { ImageResponse } from '@vercel/og';

// enable experimental edge
export const config = {  
  runtime: 'experimental-edge',  
};

export default function handler(req: NextRequest) {  
  return new ImageResponse(  
    <div  
      style={{  
        backgroundColor: "#6800A3",  
        backgroundSize: "100%",  
        height: "100%",  
        width: "100%",  
        color: "#F9F9F9",  
        display: 'flex'  
      }}  
    >  
      <div  
        style={{  
          padding: "20px",  
          display: 'flex',  
          flexDirection: 'column',  
          alignItems: 'center',  
          justifyContent: 'center',  
          width: '100%'  
        }}  
      >  
        <h1  
          style={{  
            fontWeight: 900  
          }}  
        >  
          ByteSizedPieces  
        </h1>  
        <h2>Article title should go here</h2>  
        <div style={{  
          'width': '100%',  
          display: 'flex',  
          justifyContent: 'center',  
          alignItems: 'center',  
          textAlign: 'center'  
        }}>  
          <p style={{  
            textAlign: 'center',  
            backgroundColor: '#A341DB',  
            borderRadius: '50px',  
            padding: '5px 10px'  
          }}>Category here</p>  
        </div>  
      </div>  
    </div>  
  );  
}
```

If you want to play with this example, vercel offers an OG Image Playground. I have created a <Anchor href="https://og-playground.vercel.app/?share=jVRtT4MwEP4rTY3hy5LBXowjc8leNPGbURO_7AujN-hklJTihgv_3ZaN0boaBwnp9R7u7rnn2gMOGQHs4zGhX8sU1U8uygQeDofGRmgVhJ8RZ0VK5ixh3EdLfHN377rT_hJ3bLA3-g0K5bnurQGJgUaxsLp2lIjY6gnPSZ9G6jWchOZZEpQ-ctYJ7J3GUVXH1aTZ0BnaOCKUBYTQNFKJem62N7Jc5DF8amdBOYSCslQiZMHFNjUxQUKj9FnANlcASAVwE7ApckHX5ZxJVypbZAWdmuSoHp25tmw1vpJx7Om_2ijL0mW6j5MoI9fVnW1QIyxCs1KAUpi8UAgh1zJ2Y88ooDeZckHDBJCgQn7zmBUJQRGTg8BBwns6XCpkrdKpWTsN7atluaqj_-oiYC-mCtT69cYbnRlnf7TZEsRIYjljzs20P_AWswsg4wT4a0BooUoeylH9DTlPsjPM9shTCFPYyTwQEDFennTIdBm6Uof22GjWcY07mGVq0HPsH3AtDfblZdDBx8ON_YEyCKyKCPvrIMmhg2HLNvS9zNRdI3a1JeOo4XvcroBgX_ACquoH">shareable playground here</Anchor>.

Now we get to the fun part, which are the dynamic traits to the generated image!

Our handler always receives a `Request` object as an argument. From the `req` we will obtain the `url` and retrieve provided `searchParams` .

searchParams are query params set when we use our endpoint via a meta tag. The query params can be dynamic or static depending on the developer. If this isn’t making too much sense, hang tight, when we use the endpoint we are creating it will click!

Let’s expect for now from our query params, a title and category field. We will set fallbacks should something unexpected happen and they are not set correctly.

Then we will just use those fields in our HTML and CSS passed to `ImageResponse`!

```js
// pages/api/og.js

import { ImageResponse } from '@vercel/og';

// enable experimental edge  

export const config = {  
  runtime: 'experimental-edge',  
};

export default function handler(req: NextRequest) {  
    //  get searchParams  
    const { searchParams } = new URL(req.url);  const { title = 'New Article', category = '' } = searchParams;

    return new ImageResponse(  
        <div  
      style={{  
        backgroundColor: "#6800A3",  
        backgroundSize: "100%",  
        height: "100%",  
        width: "100%",  
        color: "#F9F9F9"  
      }}  
    >  
      <div  
        style={{  
          padding: "20px"  
        }}  
      >  
        <h1  
          style={{  
            fontWeight: 900  
          }}  
        >  
          ByteSizedPieces  
        </h1>  
        <h2>{title}</h2>  
        <div style={{  
          'width': '100%',  
          display: 'flex',  
          justifyContent: 'center'  
        }}>  
          <p style={{  
            width: '150px',  
            backgroundColor: '#A341DB',  
            borderRadius: '50px'  
          }}>{category}</p>  
        </div>  
      </div>  
    </div>  
  );  
}
```

Note that we adjusted the image to leverage the title and category variables now!

We have completed the first big part for OG Dynamic Images. The endpoint is set to expect two dynamic fields, a title and category, and will consume the provided HTML and CSS and return an image rendered with those fields.

You can test if the endpoint is working by visiting your running local instance of the app at ```https://localhost:3000/api/og?title=test&category=react```

Note that in the test I have supplied the query params `title` and `category`. We observe an element that when inspected in the Developer Tools, reveals an `img` element. Neat!

Now the final detail comes to bring this full circle.

## Use new API

Define a meta tag in the `head` for the page you would like the asset to display. For me that would be within `pages/posts/[slug.js]`

```jsx
<Head>
    <meta property="og:image"
      content=\`https://bytesizedpieces.com/api/og?title=${getPostShortTitle(frontMatter) || ''}&category=${getPostCategory(frontMatter)}\`
    />
</Head>
```

A few details to note here that are very important for this to work.

You _must_ specify the following attributes:

- property as `og:image`
- content with value set to the _deployed_ instance for your app

Finally, check out how I provided the query params `title` and `category` to the value for the `content` attribute!

frontmatter is a representation of parsed YAML metadata each of my articles begin with. Thus I will achieve at each `bytesizedpieces.com/posts/*` route a dynamically generated asset at the edge, with built in caching!

Amazing!

Vercel is a no brainer. The improvements to NextJS are ever enticing. <Anchor href="https://vercel.com/ambassadors/codezysg">If you are interested in joining Vercel, try it out!</Anchor>

## Custom Fonts in OG Dynamic Image

One final mention, note that the supplied image contains the <Anchor href="https://fonts.google.com/specimen/Poppins">Poppins</Anchor> typeface. If you would like to utilize a custom font, NextJS have accommodated this use case.

By making use of `fetch` you can supply a `URL` instance with the path to your custom fonts, and await the result in the handler. <Anchor href="https://vercel.com/docs/concepts/functions/edge-functions/og-image-examples#using-a-custom-font">Refer to the following documentation to see how this is accomplished.</Anchor>

## Testing

Test open graph images on Facebook, LinkedIn and Twitter separately, or take advantage of an open graph debugger.

1.  <Anchor href="https://developers.facebook.com/tools/debug/">
      Facebook open graph Debugger.
    </Anchor>
2.  <Anchor href="https://cards-dev.twitter.com/validator">
      Twitter open graph Debugger.
    </Anchor>
3.  <Anchor href="https://www.linkedin.com/post-inspector/">
      Linkedin open graph Debugger.
    </Anchor>
4.  <Anchor href="https://metatags.io/">
      All social media, open graph Debugger
    </Anchor>
5.  <Anchor href="https://og-playground.vercel.app/">
      OG Dynamic Image Playground
    </Anchor>

### Documentation to learn more

- <Anchor href="https://vercel.com/docs/concepts/functions/edge-functions/og-image-generation">
    API Documentation for og image generation
  </Anchor>
- <Anchor href="https://vercel.com/docs/concepts/functions/edge-functions/og-image-examples">
    Vercel Examples
  </Anchor>
- <Anchor href="https://vercel.com/new/clone?repository-url=https://github.com/vercel/examples/tree/main/edge-functions/vercel-og-nextjs&project-name=vercel-og-nextjs&repository-name=vercel-og-nextjs">
    Deploying your first social card
  </Anchor>

---

Remember, developers are creatures that turn coffee into code. So I'd very much appreciate if you <Anchor slug="overcome-imposter-syndrome" action="buy_coffee_overcome-imposter-syndrome_click" href={"https://www.buymeacoffee.com/stefanagloginic"} styles={{ 'display': 'inline-flex', 'alignItems': 'center' }}>bought me a coffee! <TinyIcon alt="buy me a coffee icon" styles={{
    width: '23px',
    height: '23px',
    marginLeft: '3px'
}} src={"https://res.cloudinary.com/bytesizedpieces/image/upload/v1639952578/assets/buymeacoffee_udskx4.png"} /></Anchor> I will be posting very frequently on my findings and learnings in the tech industry and beyond. Join my newsletter if you would like to stay tuned!

Thanks for reading again! ❤️



Metadata and meta tags are underestimated SEO optimization tools. Newer web developers may completely gloss over this powerful technique to improve the discoverability of the websites for which they are responsible.

Learn how to use NextJS Version 13 release for Open Graph Dynamic Images by understanding how metadata and metatags lead to better SEO and engagement with users

NextJS Open Graph Dynamic Image

Understanding Open Graph Dynamic Image Meta Tags with NextJS

 
 

 ![Webpack bundle analyzer (package bloat)](https://res.cloudinary.com/bytesizedpieces/image/upload/v1667153748/article/medium-to-mdx/bundle_analyzer_yorhev.png)

The tech industry never stops expanding with alternate solutions. We see a beautiful landing page and get sucked into wanting to integrate these tools into our workflow. The problem with these solutions is that they sometimes can bloat our projects where this could be avoided. <Anchor href="https://www.sanity.io/">Sanity.io</Anchor> was one such product we recently came across that enraptured us.

Sanity offers a viable solution that unlocks rapid content release through a seamless and easy UI. Technical and non-technical individuals can work together, with little effort spent to custom-develop the resources needed. We envisioned potentially integrating Sanity into the ByteSizedPieces blog.

### ByteSizedPieces Background Context

<Anchor href="https://www.bytesizedpieces.com/posts/how-i-built-my-blog">
  ByteSizedPieces was developed
</Anchor> in a way that it would be lightweight, and cost-effective but still performant.
Unfortunately, the website wasn’t user-friendly, or considerate of the time when
it came to adding new content.

The workflow for ByteSizedPieces expected a single individual to make a contribution and it took a lot of time to prepare articles for release. Articles were written on Medium for the ease of the editor and laboriously copied over into the expected <Anchor href="https://mdxjs.com/">MDX</Anchor> syntax.

This was not the best setup, especially considering ByteSizedPieces is run by a capable developer. Allowing time sucks like this to go on for long is unacceptable!

This is where we considered integrating Sanity into the ByteSizedPieces <Anchor href="https://nextjs.org/">NextJS</Anchor> project. We were however reluctant to add yet another heavy dependency to the project when it felt like the problem could be addressed via a script.

## Sanity vs custom script/small package
A quick check of Sanity’s stats on <Anchor href="https://bundlephobia.com/">Bundlephobia</Anchor> showed a large minified **_bundle size of approximately 270 bytes._**

![Sanity Bundlephobia](https://res.cloudinary.com/bytesizedpieces/image/upload/v1667153761/article/medium-to-mdx/sanity_bundlephobia_rugdu4.png)

## Alternative to Sanity (generalize for other tools)
For every fully fleshed out product, there is a simpler solution that isn't pretty to look at but could realistically get the job done.

Let’s see how we can better match the priorities of ByteSizedPieces.

Because we use Medium as my editor, perhaps there already existed a solution that could transfer content on Medium to MDX syntax. This is the point at which we discovered <Anchor href="https://www.npmjs.com/package/medium-to-markdown">medium-to-markdown</Anchor>.

The difference in size between Sanity and medium-to-markdown is _monumental_. Observe that the **_bundle size of the minimized medium-to-markdown package is approximately 260kB_**.

![medium-to-markdown npm package bundlephobia breakdown](https://res.cloudinary.com/bytesizedpieces/image/upload/v1667153885/article/medium-to-mdx/medium_to_md_ydjeuu.png)

Realize this observed bundle size discrepancy makes perfect sense. Solutions like Sanity pull in many other dependencies to create that aesthetic and intuitive UI that would become the admin portal for your content-driven website. Medium-to-markdown hones in on a specific problem and doesn’t offer much flexibility. This approach allows it to be such a light package.

For ByteSizedPieces, Sanity was overkill and would cause unnecessary bloat. <Anchor href="https://tsh.io/blog/reduce-node-modules-for-better-performance/">Here is a fantastic article</Anchor> that breaks down why we should care about bloat for projects, and how we might go about reducing the size of our `node_modules`.

In a nutshell, recall ByteSizedPieces considers performance and cost-effectiveness. **If we are not careful about unnecessary bloat, then our application becomes less performant over time**. We additionally **incur a greater cost for infrastructure management of the resources**.

![A Venn diagram where effective, efficient, and satisfying make for usability](https://res.cloudinary.com/bytesizedpieces/image/upload/v1667154085/article/medium-to-mdx/usability_vd_hn7kog.jpg)

## Medium-to-markdown package evaluation
With all this taken into consideration, we tried to proceed with this more straightforward solution for the time being. Let’s look into the medium-to-markdown package.

After some examination, we, unfortunately, discovered the package didn’t work end-to-end for the preferred ByteSizedPieces workflow. Our blog posts are intended only to be published on <Anchor href="https://bytesizedpieces.com/">bytesizedpieces.com</Anchor>, and we learned that the script was only capable of parsing public, non-paywalled content on Medium.

Not a problem! We decided to take a look at how this script parses the content, and expand its capabilities ourselves.

## Adapted medium-to-mdx script
We have the capability to evaluate how the script performs it's conversions, and adapt it to our needs.

Our priorities for our customized medium-to-mdx script are:

* Enabling the script to consume a draft link
* Parse mdx
* Setup rules for the custom elements ByteSizedPieces uses
* Add YAML template for gray matter to parse for article metadata
* Write converted contents to templates/ where articles are stored
* Generate new article route for sitemap

The medium-to-markdown package mainly relies on 3 dependencies to parse and translate medium content to markdown syntax. The 3 packages are <Anchor href="https://www.npmjs.com/package/turndown">Turndown</Anchor> (the library to convert HTML to Markdown), <Anchor href="https://www.npmjs.com/package/cheerio">cheerio</Anchor> (parses markup and provides an API for traversing/manipulating the resulting data structure), and <Anchor href="https://www.npmjs.com/package/turndown-plugin-gfm">turndown-plugin-gfm</Anchor> (a turndown intended plugin that adds GitHub Flavored Markdown extensions).

The repository was simple enough to comprehend.

We have a series of <Anchor href="https://github.com/dtesler/medium-to-markdown/blob/master/lib/mdConverters.js">converters</Anchor> that just focus on the DOM elements we would encounter in Medium which we would like to parse. For each of those converters, a <Anchor href="https://github.com/dtesler/medium-to-markdown/blob/master/lib/convertFromUrl.js#L14-L16">turndownService appropriate rule</Anchor> gets added. <Anchor href="https://github.com/dtesler/medium-to-markdown/blob/master/lib/convertFromUrl.js#L19-L41">A few more rules get added to the turndownService</Anchor> specific to encountering an anchor in Medium, and <Anchor href="https://github.com/dtesler/medium-to-markdown/blob/master/lib/convertFromUrl.js#L45-L64">two more rules for images, and codeblocks</Anchor>.

<Anchor href="https://github.com/dtesler/medium-to-markdown/blob/master/lib/convertFromUrl.js#L73">
  convertFromUrl
</Anchor> is the function to invoke, provided the URL of the public non-paywalled
medium article. The package requests the URL, targets the body with cheerio then
the article of the Medium retrieved content, and <Anchor href="https://github.com/dtesler/medium-to-markdown/blob/master/lib/convertFromUrl.js#L85">
  invokes the turndownService
</Anchor> configured with the set rules. The turndownService will proceed to do its
magic. The DOM tree gets parsed, and on a match of elements for which rules were
set, the turndownService will invoke the set of handlers to map the elements to the
appropriate replacement.

Now that we understand how the package is accomplishing the conversion from medium content to markdown, we need to consider how we can adjust it to our use case.

We recognize that the primary issue is unpublished content on Medium. Thankfully, Medium provides us with a way to expose these unpublished articles via a draft link.

![Share the draft link in the dropdown of the unpublished article on medium](https://res.cloudinary.com/bytesizedpieces/image/upload/v1667154131/article/medium-to-mdx/draft_link_twr9gf.png)

The draft link is provided to another user with a Medium account, so we probably need to log in to a viable Medium account, and _then_ attempt to visit the draft URL in the same session.

This is starting to sound like we need a headless browser. For this, we decided to use <Anchor href="https://www.npmjs.com/package/puppeteer">puppeteer</Anchor>.

![Puppeteer](https://res.cloudinary.com/bytesizedpieces/image/upload/v1667154161/article/medium-to-mdx/puppeteer_cwzmzb.jpg)

## Headless Browser Puppeteer Login To Medium Flow

1.  Tell puppeteer to start the browser
2.  Visit the home of medium
3.  Select the sign-in button on medium.com
4.  Sign in via Twitter option
5.  Have the page object input the correct Twitter credentials
6.  Visit the draft URL provided

### Tell puppeteer to start the browser

For simplicity, I have removed the irrelevant code. To start, we have a `navigateToMediumUrlWithLogin` function, which will invoke `loginToMedium` and then after that navigate to the draft URL. loginToMedium will start the browser and page via puppeteer and return those relevant values.

```js
async function startBrowser() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    return { browser, page };
}

...

async function loginToMedium(url) {
    const browserPageObj = await startBrowser();    ...
}

...

export const navigateToMediumUrlWithLogin = async (url) => {
    let browserPageObj = null;
    try {
        browserPageObj = await loginToMedium('https://medium.com/');
    } catch (e) {}
};
```

### Visit the home of Medium

Next, we need to visit Medium using the `page` we received from `startBrowser` . Our `loginToMedium` will now be adjusted to accommodate this next most intent.

```js
async function loginToMedium(url) {
    const browserPageObj = await startBrowser();
    const { page } = browserPageObj;
    page.setViewport({width: 1366, height: 768});
    console.log('Navigating to: ', url);
    await page.goto(url);
    ...
}
```

### Select the sign-in button on medium.com

Still focusing on `loginToMedium` we now need to select the sign-in button on medium.com.

Here is how we accomplished selecting the sign-in button:

```js
async function loginToMedium(url) {
    ...

    const [ anchor ] = await page.$x("//a[contains(text(), 'Sign In')]");
    if (!anchor) {
        return;
    }

    console.log('Selecting sign in prompt at: ', url);

    await anchor.click();

    await page.waitForTimeout(200);

    ...
}
```

### Select login via the Twitter button

Now we need to select _how_ we want to log in. Let’s go with Twitter.

```js
async function loginToMedium(url) {
   ...

   const [div] = await page.$x("//div\[contains(text(), 'Sign in with Twitter')\]");

   if (!div) {
        return;
    }

    console.log('Selecting Sign In With Twitter Prompt on popup at: ', url);

    await div.click();

    await page.waitForTimeout(2000);

    ...
}
```

### Have the page object input the correct Twitter credentials

We are almost done with the flow. We now target the input fields for the Twitter username and password and populate those fields for the account we prefer.

We return the browserObject with the page at the result after signing in.

```js
async function loginToMedium(url) {
    ...

    console.log('Populating Twitter credentials, username first');
    // for twitter login populate username

    await page.click('#username_or_email');
    await page.keyboard.type(credentials.TWITTER_USERNAME);

    console.log('Populating Twitter credentials, password next');

    // for twitter login populate password
    await page.click('#password');
    await page.keyboard.type(credentials.TWITTER_PASSWORD);

    console.log('Attempting to sign in via Twitter profile');

    await page.click("#allow");

    await page.waitForNavigation();

    return browserPageObj;

    ...
}
```

Now that `loginToMedium` is complete, we can proceed with visiting the draft URL we were provided.

### Visit the draft URL provided

We divert our focus back to the`navigateToMediumUrlWithLogin` function. It will use the returned `browserPageObj` to visit the provided URL which is our draft URL, then finally return the `browserPageObj` that should be in the correct context and ready for parsing.

```js
export const navigateToMediumUrlWithLogin = async (url) => {
    let browserPageObj = null;

    try {
        browserPageObj = await loginToMedium('https://medium.com/');
    } catch (e) {
        console.log('Something happened while trying to login to medium', e.message);
        return browserPageObj;
    }

    await browserPageObj.page.waitForTimeout(2000);

    console.log('Now attempting to visit url: ', url);

    await browserPageObj.page.goto(url);

    return browserPageObj;
};
```

For this, we have a `getMarkdownFromPage` function, that we will utilize to pull the HTML from the puppeteer page, use the turndownService to convert to markdown, and write the contents of that markdown to our desired file.

Here is where we tie all of this together. Our script will effectively be calling `convertUrlByLogin` to retrieve the markdown and write the contents to a file.

```js
// retrieve the correct page after successful navigation
// to medium draft url (which requires login)
// from there seek out the article element and use
// turndownService to convert to markdown
export const convertUrlByLogin = async (url) => {
    const browserPageObj = await navigateToMediumUrlWithLogin(url);
    return _getMarkdownFromPage(browserPageObj);
}
```

What is neat about pursuing this approach ourselves instead of relying on the medium-to-markdown package, is that we can custom configure our rules to the turndownService. So we did add our own rulesets so the new elements we map during match occurrences are ByteSizedPieces appropriate.

Perhaps you noticed, why stop at markdown? ByteSizedPieces uses MDX. Although MDX enables the usage of markdown, it enables the usage of React components too!

Our turndownService rules can thankfully provide the React components since they are provided as strings anyways. MDX will be able to correctly handle once our script writes the contents to a file.

Here is an example of a rule set for processing Medium images since ByteSizedPieces uses a custom component for images.

```js
{ // NOTE: This entire rule is bytesizedpieces specific
  ruleName: 'images',
  filter: (node, options) => {
      return (
          node.nodeName === 'IMG' &&
          node.getAttribute('src') &&
          !node.getAttribute('src').endsWith('?q=20')
      )
  },
  replacement: (content, node) => {
    const parentWrapperDiv = node.parentNode;
    let parentNode = null;
    if (parentWrapperDiv) {
        parentNode = parentWrapperDiv.parentNode;
    }

    const captionLabelIdSibling = parentNode ? parentNode.nextSibling : undefined;
    let captionLabelId = '';
    let innerText = '';
    if (captionLabelIdSibling && captionLabelIdSibling.nodeName === 'FIGCAPTION') {
        innerText = captionLabelIdSibling.textContent;
    }

    if (innerText.length > 0) {
        captionLabelId = innerText.toUpperCase().split(' ').join('_');
    }

    return '<Image captionLabelId="' + captionLabelId + '" alt="' + innerText + '" src="' + node.getAttribute('src') + '" />';
}
```

For each article we add some YAML at the top of our files to define some metadata that <Anchor href="https://www.npmjs.com/package/gray-matter">gray-matter</Anchor> parses for us.

Knowing this, we added some logic to insert a YAML template at the start of our newly created file. Later we intend to make the script smarter to attempt to autofill some of these fields, but a template works nicely for the time being.

Here is how the final script looks.

```js
const generateMdFromMediumArticle = async (fileNameArg, mediumPostUrlArg) => {
    if (!fileNameArg) {
        throw new Error('Need to provide file name to convert as argument');
        return;
    }

    if (!mediumPostUrlArg) {
        throw new Error('Need to provide url for medium article to convert as argument');
        return;
    }
    const PATH_FOR_CONVERTED_MD = `${POST_TEMPLATES_FOLDER_NAME}/${fileNameArg}.mdx`;

    const markdown = convertUrlByLogin(mediumPostUrlArg);
    if (markdown && markdown.length > 0) {
        const prettierConfig = await prettier.resolveConfig('./prettierrc.js');

        const formattedMarkdown = prettier.format(markdown, {
            ...prettierConfig,
            parser: 'mdx'
        });

        writeFileSync(PATH_FOR_CONVERTED_MD, `${YAMLTemplate} \\n \\n ${formattedMarkdown}`);
        console.log('Success, file written and should be found at: ', PATH_FOR_CONVERTED_MD);
        return;
    }
```

Finally, here is what I added to `package.json`

`“mediumToMdx”: “node ./scripts/convert-mdx-from-medium-url.mjs && npm run postbuild”`

Note that we developed the script in such a way that when it starts up, it prompts the user for the name of the file they would like to create (which will become the route by default for the article) as well as the URL to the medium link. These arguments get passed on to `generateMdFromMediumArticle` to take it from there.

As a final callout, note `postbuild` being added after the conversion script runs. We wanted to kill two birds with one stone by first writing the new file, then generating an update to the ByteSizedPieces `sitemap` for search engines to reference for indexing purposes.

Sitemap generation is yet another script we added to increase efficiency in the development of the blog, and will be covered in another article.

Let’s see this in action in this very article! Observe how I retrieve the draft link from Medium, start up my script, and show the final markdown transferred result.

![medium-to-mdx script](https://res.cloudinary.com/bytesizedpieces/image/upload/v1667154318/article/medium-to-mdx/medium-to-md-final_rgybng.gif)

## Conclusion

So! What do we gain from this effort?

**Transferring an article to ByteSizedPieces was laborious and took approximately an hour worth of time.**

Adding this custom script **_cut the effort of transferring an article to ByteSizedPieces to 5 minutes._**

The script is not perfect, after the file is generated we need to still go in and make a few edits. The effort to edit and validate the article comprises the majority of the 5 minutes. The script itself runs in ~20 seconds.

Those 55 minutes saved enable us to divert efforts elsewhere for more meaningful content and changes!

The final lesson here is to keep an open mind when deciding what to custom-develop and when we should be using tools. Always bear in mind your original goals and stay true to them.

This script took about 4 hours to get working end-to-end. Though it can use improvements that will happen in time, it works beautifully and saved us from adding a heavy package to ByteSizedPieces.


The tech industry never stops expanding with alternate solutions. We see a beautiful landing page and get sucked into wanting to integrate these tools into our workflow. The problem with these solutions is that they sometimes can bloat our projects where this could be avoided.

Use bundlephobia to assess package size and potential project bloat. Learn how to differentiate between real products like Sanity versus how to create your own solution, in this example learn how to create a Medium to mdx script

When to use a custom solution versus your own

How to decide between the use of custom solutions and ones that bloat your website


## Understand types of caches and policies to better build your own system

Cache systems demand a delicate balance for applications, as they have many <Anchor href="https://www.bytesizedpieces.com/posts/cache-pro-con">pros and cons</Anchor>. When done right, caching can be performant, increase application speed and decrease costs. However, the integration of cache systems poses the risk of application reliability downgrade.

![ByteSizedPieces on Cache Policies and Types](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663569821/article/cache-types/cache_policies_and_types_kbypck.png)

Engineers need to be well educated in the approaches that could be taken when <Anchor href="https://www.bytesizedpieces.com/posts/cache-system-design">designing a cache system</Anchor>. Part of the cache system relies on the appropriate selection of caching policies and implementation approaches. The choices an engineer makes should fit the needs of the application behavior.

A cache typically sits in between applications and databases. Its sole purpose is to provide with lightning speed the responses applications request. For the cache to do its job correctly, it needs clearly defined rulesets for _how_ it will obtain the data necessary to remain relevant. With that said, there are **varying types of caching systems governed by carefully selected cache policies, pertaining to data reads and writes.**

**Cache read and write policies** define when the cache and main memory should be updated to maintain the relevance and reliability of the data when it comes to applications.

The event for which the system or application makes a request to retrieve data from a cache, but that specific data is not currently in the cache memory is known as a <Bold styles={{ 'variant': 'colors.primary' }}>cache miss</Bold>.

![Cache miss](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663569853/article/cache-types/cache_miss_fc9sn2.png)

A <Bold styles={{ 'variant': 'colors.primary' }}>cache hit</Bold> is when the system or application makes a request to retrieve data from a cache and the data is in fact present and served to the application.

![Cache hit](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663569851/article/cache-types/cache_hit_c8c3cw.png)

What the cache strives for, are more **cache hits** that contain **accurate representations of the data in the main memory**. This is where the read-and-write policies largely come into play.

**The read and write policy the cache system follows largely relates to the frequency of cache misses and cache hits.**

### Cache Read Policy

<Bold styles={{ 'variant': 'colors.primary' }}>Cache read policies</Bold> depict how the cache is to behave on data retrieval attempts from the application. Should the cache be referenced and the data is not present, then the main memory takes on the responsibility of serving the information needed. This is an occurrence we want to minimize, especially if our application is read-heavy.

### Cache Write Policy

The way in which the cache gets updated when data changes are goverened by the <Bold styles={{ 'variant': 'colors.primary' }}>cache write policy</Bold>.

There are three primary cache write policies.

1.  write-through
2.  write-back
3.  write-around

### Write-through cache writing policy

<Bold styles={{ 'variant': 'colors.primary' }}>Write-through</Bold> is a cache writing policy in which data is written into the cache and the corresponding main memory location simultaneously.

![Write-through cache](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663569887/article/cache-types/write_through_qynj75.png)

**Advantages of write-through**

- Simple
- Reliable
- Aids with data recovery
- Resolves inconsistency problem

The write-through policy is simple and reliable as our data is in constant sync with the main memory location. Any writes to the data will result in an adjustment to the cache, thereby eliminating the risk of a dated cache. In the circumstance of power outages or system failures, we have the cache to aid with the recovery of our data as well. The cache can serve as a backup.

**Disadvantages of write-through**

- High traffic of writes = worsened performance
- Increased latency

A write-through policy puts into question the advantage of having a cache for write operations. Why? The point of a cache is to _avoid_ going to the main memory, and this policy goes to the main memory one-to-one when it comes to writes.

In the circumstance of high traffic, writing to two sources is not performant, and is vulnerable to latency. Therefore, the **write-through cache policy is appropriate for applications that do not have a high write volume.**

### Write-back cache writing policy

<Bold styles={{ 'variant': 'colors.primary' }}>Write-back</Bold> is a cache writing policy in which data is written only to the cache. The data gets written in the <Bold><Italics>background</Italics></Bold> into the corresponding location in the main memory at specified intervals or under specific conditions. The completion confirmation for the write operation remains unblocked when it comes to writes to the backing store or main memory, as this process happens in the background.

![Write-back cache](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663569882/article/cache-types/write_back_h8qskq.png)

**Advantages of write-back**

- Low latency
- High throughput for write-intensive applications

Writing to the cache is much faster than deferring writes to memory. Therefore the write-back solution can manage many more write requests and complete the requests with low latency.

**Disadvantage of write-back**

- Risk of data availability/data loss
- Complex

Should the cache fail, we run the risk of suffering from data loss in the event that the data had not been written yet in the background to the backing store.

A write-back cache is more complex to implement than a cache using the write-through policy, as the cache needs to track which data is present solely in the cache but not yet in the main memory. Any updated data not yet reflected in the cache typically gets marked as **dirty** to indicate the need for a write to the backing store. The data in these locations get written back to the backing store when they are evicted from the cache. The main memory update on the removal rule is an effect referred to as a **lazy write**. As a result of this defined rule, <Anchor href="https://www.youtube.com/watch?v=xU0ICkgTLTo&ab_channel=Udacity">a read miss in a write-back cache often results in two memory access</Anchor>: one to write the replaced data from the cache back to the store, and one to retrieve the needed data.

The write-back cache is beneficial when it comes to applications with high write volume.

### Write-around cache writing policy

The <Bold styles={{ 'variant': 'colors.primary' }}>write-around policy</Bold> has writes go straight to the database first. The cache gets updated after the write to the main memory is complete, or the corresponding key in the cache is deleted to trigger an update the next time a data retrieval from the cache gets attempted.

![Write-around cache](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663569848/article/cache-types/write_around_f5u36k.png)

**Advantages of write-around**

- Up-to-date database
- Fault tolerance
- Avoidance of cache flooding

**Disadvantage of write-around**

- Potentially higher read latency
- Potential higher miss rate

**The write-around policy works for those applications that write fairly often, and don’t tend to re-read recently written data.** The outcome when using this policy results in a cache with values that tend to get read often.

A detail alluded to thus far but not explicitly discussed yet is how we determine when fields in the cache are in need of replacement, given the cache is limited in space.

## Cache Replacement Policies

A <Bold styles={{ 'variant': 'colors.primary' }}>cache replacement policy</Bold> or <Bold styles={{ 'variant': 'colors.primary' }}>cache eviction algorithm</Bold> governs the circumstances under which data is subject to being overwritten. The cache eviction policy is necessary because caches tend to be substantially smaller than the main memory to stay cost-effective and efficient.

There are several popular methodologies for determining when to replace data stored in a cache system.

**Some cache replacement policies**

- First in first out (FIFO)
- Last in first out (LIFO)
- Least recently used (LRU cache)
- Least frequently used (LFU cache)
- Most recently used (MRU cache)

### First in First Out (FIFO)

If one is familiar with <Anchor href="https://www.geeksforgeeks.org/queue-data-structure/">Queues, the FIFO cache policy</Anchor> will envelop the same behavior.

![FIFO (First in First Out)](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663570008/article/cache-types/fif_b6zhlj.webp)

In a **First in First Out or** **FIFO replacement policy**, any item that first got added to the cache is also the first item to be evicted when there is not enough remaining space. The cache evicts blocks according to the order in which they were added, without regard to how often or how many times they were accessed before.

### Last in First Out (LIFO)

If one is familiar with the <Anchor href="https://www.geeksforgeeks.org/stack-data-structure/">Stack as a data structure</Anchor>, a LIFO or last in first out cache likewise follows the behavior of the Stack.

![LIFO (Last in First Out)](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663570009/article/cache-types/lifo_ppur8q.jpg)

In a **Last in First Out or LIFO replacement policy**, any item that last got added to the cache is also the first item to be evicted when there is not enough remaining space.

### Least Recently Used (LRU cache)

Now, this style of cache is more interesting and common in a real-world setting. <Anchor href="https://leetcode.com/problems/lru-cache/">LRU Caches are common interview questions</Anchor> and are highly applicable.

**The Least Recently Used or (LRU Cache)** keeps track of the items added to the cache, and _when_ they get used. This means that the item that has been retrieved last is the first to get evicted from the cache upon a new entry.

![LRU (Least Recently Used) Cache](https://res.cloudinary.com/bytesizedpieces/image/upload/v1663570010/article/cache-types/lru_cache_ppdkul.png)

Any item that gets retrieved or added to the cache gets brought to the forefront, to indicate the recency for which it was used. If an item gets added, we can assume by the natural ordering that follows from our addition/retrieval rulesets that the last item in the cache is the least recently used one. Therefore the last item can be evicted from the cache on insertions to a cache that is void of space.

### Least Frequently Used (LFU cache)

Though it sounds similar to the least recently used cache, the frequency focuses more on how many times a certain item is referred to from the cache! Instead of keeping tabs on how _recently_ a block was accessed, we store a _value_ depicting how many times it was accessed.

**The Least Frequently Used or (LFU Cache)** counts how often an item in a cache is needed. Those items in the cache that are used least often are discarded first.

### Most Recently Used (MRU cache)

For the **Most Recently Used cache or MRU cache**, you guessed it, the item that is _most_ recently used is the first to be replaced in a cache!

You might be wondering why we would ever want to evict the most recently used items from a cache.

A most recently used cache or MRU cache applies to situations for which the older an item is, the more likely it is to be accessed. Occasionally, if you have just seen something then it’s unlikely you’ll want to see it again!

For a more literal example of the benefit of this style of cache. Imagine an individual is assessing the quality of recently traded devices. You just examined a phone and determine that it is of higher quality. Would you want to examine the phone again anytime soon? Probably not!

The way the most recently used cache or MRU cache approach works is by keeping a list of the items, for which the item that is most recently viewed becomes the first to be evicted should a new item be inserted.

## Summary

Caching systems need to follow rulesets that will determine how effective the cache is at doing its job.

The <Bold styles={{ 'variant': 'colors.primary' }}>cache hit ratio</Bold> is a measurement of how many requests a cache is **_able_** to fulfill successfully, in comparison to the number of requests it receives.

Therefore, a cache needs to seek to improve its **cache hit ratio**.

The <Bold styles={{ 'variant': 'colors.primary' }}>cache miss ratio</Bold> is a measurement of how many requests a cache is **_unable_** to fulfill successfully, in comparison to the number of requests it receives.

A cache should seek to minimize the **cache miss ratio** while ensuring that the data remains relevant.

So, how do we accomplish this?

Caches follow **read and write policies**, where a few examples of write policies are:

1.  write-through
2.  write-back
3.  write-around

Each type of policy comes with trade-offs that need consideration. Application needs should be taken into consideration, and those are determined by evaluating the behavior of the application.

Ask questions like:

> Does the application read data often, or write often?

Or

> Does the application have written data become read immediately, or after some time?

Or how about

> Does recently read application data get re-read often?

Every application has an applicable read-and-write policy. Each policy accommodates different needs. The disadvantages of each need to be carefully evaluated and monitored.

The next detail to consider is that caches are a small subset when compared to the size of main memory.

Because of the limited size of a cache, the cache also needs to follow a policy that governs when stored data is subject to be replaced with incoming fresh data. This type of policy is better known as the <Bold styles={{ 'variant': 'colors.primary' }}>cache replacement policy</Bold>.

Some cache replacement policies:

- First in first out (FIFO)
- Last in first out (LIFO)
- Least recently used (LRU cache)
- Least frequently used (LFU cache)
- Most recently used (MRU cache)

---
Remember, developers are creatures that turn coffee into code. So I'd very much appreciate if you <Anchor slug="overcome-imposter-syndrome" action="buy_coffee_overcome-imposter-syndrome_click" href={"https://www.buymeacoffee.com/stefanagloginic"} styles={{ 'display': 'inline-flex', 'alignItems': 'center' }}>bought me a coffee! <TinyIcon alt="buy me a coffee icon" styles={{
    width: '23px',
    height: '23px',
    marginLeft: '3px'
}} src={"https://res.cloudinary.com/bytesizedpieces/image/upload/v1639952578/assets/buymeacoffee_udskx4.png"} /></Anchor> I’m a new writer and I will be posting very frequently on my findings and learnings in the tech industry and beyond. Join my newsletter if you would like to stay tuned!

Thanks for reading again! ❤️

Cache systems demand a delicate balance for applications, as they have many pros and cons. When done right, caching can be performant, increase application speed and decrease costs. However, the integration of cache systems poses the risk of application reliability downgrade

Different types of caches exist to serve the needs of an application by its behavior. Many write policies for caches have trade offs to consider as well as cache replacement policies to dictate when data in the cache is overwritten. Caches need to improve cache hit ratio over cache miss ratio to make applications more efficient.

Types of caches and policies

Understand types of caches and policies to better build your own system


import { FormattedMessage } from 'react-intl';

import Anchor from '../components/article/Anchor';
import TinyIcon from '../components/article/TinyIcon';
import Bold from '../components/article/Bold';

![ByteSizedPieces on Caching System Design](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247862/article/cache-system-design/1_c2_bOIcMfZAsKG37v9h8pQ_gvvjt8.png)

We live in a world governed by fast results. Caching is one such technique that enables rapid responses online. This begs the question for software engineers:

> How do we determine the best design for our cache such that it fits the needs of our application while staying reliable?

When it comes to system design for caching, there are two approaches a software engineer needs to consider. There is the **in-memory cache system**, versus the **distributed cache system**.

1.  in-memory cache system
2.  distributed cache system

## In-memory cache system

An <Bold styles={{ 'variant': 'colors.primary' }}>in-memory cache</Bold> is a data storage layer reliant on a **_single_** computer's main memory (ex: random access memory, or RAM). The in-memory cache sits in between applications and databases and provides responses with lightning speed. The system accomplishes rapid responses by storing data from prior made requests or by storing data copied directly from databases. The RAM used for temporary storage is known as the cache.

![Simple Caching Diagram](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247883/article/cache-system-design/1_YzAi-UVHleh_p13HECHU7g_mlkslz.png)

The important piece to note in my explanation of this type of caching design system is the specification that the **in-memory cache is reliant on a _single_ device’s memory*.*** If you’re questioning which device the cache lives on, this could be a client-side cache (in the browser or hosting application of the client) or a server-side cache (some arbitrary server that sits between the application and the database).

We now need to take into account the <Anchor href="https://www.bytesizedpieces.com/posts/cache-pro-con">pros and cons of using a cache</Anchor>.

A few pros to using a cache:

- Latency reduction
- Accelerated data retrieval
- Increase IOPS (Input/Output operations per second)

However, it’s the detriments to caching that we want to focus on when considering the in-memory caching system. The potential risks associated with relying on an in-memory cache encourage a software engineer to consider whether a _single_ device will be sufficient for their application needs, or if they need a more involved system.

The primary concerns a good engineer needs to consider with in-memory caching, and really any system design-related problem:

1.  **Scalability**
2.  **Availability**
3.  **Performance**

### Scalability

![Maintaining Scalability](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247897/article/cache-system-design/0_Ckx7mftvBRz1m2I9_t8h5sr.jpg)

Memory on a single device is a limited resource. Consider the burden we may be placing on our single device if we intend to cache large datasets or have our device single-handedly manage an application with a high volume of traffic. We will inevitably see performance-related problems begin to bubble up with an overburdened cache. When that time comes, one can infer it is time to scale their cache system to better meet application demand.

### Availability

It may occur that our connection to the cache is severed. Without our cache, the system becomes unresponsive, leading to a poor user experience.

![Application crashing leads to poor user experiences](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247912/article/cache-system-design/0_d6xFr9m_TgyN4nhI_qdaavi.png)

Meeting the availability requirements means focusing on ensuring that the data in the cache is not lost or inaccessible during hardware failures.

### Performance

As an application grows, so does the demand for a caching system. If the caching system is neglected, application performance begins to see a negative impact.

**High performance is the first priority for any caching system**, as performance is the entire point of creating a cache in the first place!

![High performance is the point of a cache system!](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247924/article/cache-system-design/0_CA20baXGE86W4Scf_kshajy.png)

If we start to see issues arise in any of these three points, we need to consider making our system more robust and scalable. In comes our distributed caching system design to the rescue!

## Distributed Cache System

A <Bold styles={{ 'variant': 'colors.primary' }}>distributed cache</Bold> is a design system that pools together the memory resources of **_multiple_** networked computers into a single in-memory data store. A distributed cache enables us to break out of the limitations of a single computer. This strategy of sharing data across multiple computers by accumulating their processing power is also known as a **distributed architecture** or a **cluster**.

![Distributed System](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247941/article/cache-system-design/1_0dP1bSBl-_YyydKmLAiOmA_nrpr5e.png)

An analogy to consider for a distributed cache is to think of the system as a brain and the individual computers as the limbs that listen for the brain’s command. Each server or limb doesn’t know of the others, nor does it have to. Each is responsible for its own function and no other. Recognize with this analogy that each individual computer in the overarching system or the “big picture” is abstracted out, and managed at a higher level.

An operation of this scale comes with its set of challenges. The primary challenge with a distributed cache design system is determining what rules govern the split of the data. Determining **partitions** for datasets is challenging as the never-ending trade-offs with each approach challenge the engineer's critical thinking.

One concept for implementing a distributed cache system is _sharding_.

<Bold styles={{ 'variant': 'colors.primary' }}>Sharding</Bold> involves breaking up the data into horizontal or vertical partitions, also known as the horizontal or vertical scaling of a system. Each partition is referred to as a shard or an individual server.

Now sharding in databases may be familiar. Database sharding and caching systems both leverage the same approaches and balance similar tradeoffs when considering data segmentation.

![Vertical vs Horizontal Sharding](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247955/article/cache-system-design/0_JAK-kt-2uykDQsSi_rzb4qr.png)

A few data sharding approaches:

1.  Geo-based sharding
2.  Range-based sharding
3.  Hash-based sharding

Each data sharding approach has its own advantage and disadvantage.

### Geo-based sharding

![Geo-based Sharding by user location](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247969/article/cache-system-design/1_znKEKppKFzKLEFSBmuGCQg_mvoxds.png)

This concept of sharding is based on location, more specifically, the user's location. **The user gets matched to a server or node closest to their location.**

The benefit of geo-based sharding is a **reduction in latency**. However, the population distribution by location varies, leading to **unbalanced partitions**, making this a downside to this approach.

### Range-based sharding

![Range based sharding example](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662247984/article/cache-system-design/1_V_BpHDQ3twnZ_zXufHe9ag_oyp8pt.png)

Range-based sharding sections off data by a **range of the key value pair** determined by the engineers.

One popular example where range-based sharding comes in handy, which also conveniently is a popular interview question for software engineers, is for the handling of word searches.

> Given a list of millions of documents, how would you find all documents that contain a list of words?

This question is so common in the interview setting, that it was even covered in the famous <Anchor href="https://amzn.to/3CRW26S">Cracking the Coding Interview Series</Anchor> book. Any software engineer that doesn’t own this book is putting themselves at a disadvantage. All of the fundamental concepts of the software engineer's job are covered, with ample thought-provoking questions per topic, caching included. Each question is subsequently answered with correct and optimal solutions by the author herself, a prior Google engineer that conducted hundreds of technical interviews and was on the panel.

Now I won’t go into how this question specifically is implemented, you can refer to the <Anchor href="https://amzn.to/3CRW26S">Cracking the Coding Interview book</Anchor> for that! However, you can imagine how the alphabet e.g: \[A-Z\] could serve as an appropriate range to define our partitions.

The nature of range-based sharding makes it **easy to add more partitions** or machines as it does not require extensive intervention from engineers. However, the disadvantage to sharding poses the **risk of a potential hotspot caused by non-uniformly distributed data**. One segment of the range could become responsible for far more hits than others.

### Hash-based sharding

![Hash-based sharding](https://res.cloudinary.com/bytesizedpieces/image/upload/v1662248000/article/cache-system-design/1_8UldDC0xTrVVfGdF36wJNg_yew1lh.png)

Hash-based sharding runs the input keys for the cache system through a <Anchor href="https://prog.world/choosing-a-hash-function-in-the-data-sharding-problem/">hashing algorithm</Anchor>. The hash value is used to compute the appropriate partition, thereby allocating the result to the appropriate bucket. This is demonstrated through the above image, as we have values in the dataset that are larger than our defined ranges. Our hashing function maps these values to fit in the appropriate slots.

A good hash algorithm will **distribute data evenly across partitions**. However, it is **blind to relations in the data**, thereby missing the opportunity to improve performance by trying to predict and pre-load future queries.

### Cache Writing Policies

Once an engineer has determined how best to partition the data given the use case, the rest of the cache is about the implementation details and engagement of the cache with the application and database.

A few of the diagrams in this post demonstrate that the application can still end up going to the database in instances where the cache does not contain the relevant data. And in the system design decisions have been made for some caching behaviors, particularly around how writes get handled.

These decisions came from the following questions.

1.  How will the cache be obtaining the data it needs to serve to the application?
2.  How does the cache avoid containing irrelevant data, better known as **cache inconsistency**, in the occurrence of writes?

Techniques to avoid irrelevance of cached data, or avoidance of cache inconsistency is coined by **_cache writing policy_** or **the expected behavior of a cache while a write operation is being performed**. A cache’s write policy plays a central and critical part in the varying characteristics exposed by the cache.

Cache writing policies are covered in an upcoming article that will focus on the design process of caches as well as the implementation details of a caching system.

### Summary

An **in-memory cache design system** is a data storage layer reliant on a **_single_** computer’s main memory. Given the in-memory cache depends on a single device, it is potentially a **temporary solution** for an application that does not involve unreasonable demand on a single device.

The way we justify unreasonable demand on a single device serving as our in-memory cache design system is by evaluating:

1.  **Scalability**
2.  **Availability**
3.  **Performance**

Once one begins to see tell tale signs of the single device becoming overburdened with the workload, one needs to expand their singular device to become a node aiding a more sophisticated network of devices. This network of devices is known as a **distributed cache system**.

A distributed cache system requires thorough thinking of the engineers. Working with a network of devices poses the challenge of how best to partition or **shard** the data in order to effectively share the resources of the multitude of devices.

---

Remember, developers are creatures that turn coffee into code. So I'd very much appreciate if you <Anchor slug="overcome-imposter-syndrome" action="buy_coffee_overcome-imposter-syndrome_click" href={"https://www.buymeacoffee.com/stefanagloginic"} styles={{ 'display': 'inline-flex', 'alignItems': 'center' }}>bought me a coffee! <TinyIcon alt="buy me a coffee icon" styles={{
    width: '23px',
    height: '23px',
    marginLeft: '3px'
}} src={"https://res.cloudinary.com/bytesizedpieces/image/upload/v1639952578/assets/buymeacoffee_udskx4.png"} /></Anchor>

We post relevant topics in the tech industry with a specific focus on web development biweekly on Monday’s. Join the ByteSizedPieces newsletter to stay on top of **_free_** relevant topics!

Thanks for reading again! ❤️


We live in a world governed by fast results. Caching is one such technique that enables rapid responses online. This begs the question for software engineers: "How do we go about deciding how best to design a cache that fits the needs of our application?"

Article discussing how one can make sure their cache design system is reliable. Learn how an in-memory cache design system becomes a distributed cache design system and how sharding poses a challenge in distributed systems.

Caching System Design

How to make sure your cache design system is reliable for your website


import { FormattedMessage } from 'react-intl';

import Anchor from '../components/article/Anchor';
import Italics from '../components/article/Italics';
import Bold from '../components/article/Bold';
import BigQuotes from '../components/article/BigQuotes';
import TinyIcon from '../components/article/TinyIcon';

Caching intimidates developers of all skill levels. However, despite its challenges, caching is an important concept a developer should devote the time to wrap their minds around.

![Cache associated words list](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792358/article/cache-pro-con/caching_words_qos1ma.jpg)

When applied correctly caching can bring about dramatic improvement to applications and servers of the like.

Caching has several pros and famous cons that make it an area that software engineers tend to approach with reluctance. Prior to diving into the types of caches, there are cases to caching a developer first needs to understand.

A software engineer should understand the following about a cache.

* What is a cache?
* Why is caching important?
* Pros of caching data
* Cons of caching data

## What is a Cache in Software Development?
A **cache is a sophisticated data structure intended to store information, usually data, temporarily in a computing environment.** Simply put, a cache is a temporary storage structure for information that has already been seen.

![Time Sensitive Cache](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792358/article/cache-pro-con/caching_time_ogsykd.png)

## Why is Caching Important?

Caching data is important because it enables applications or servers to quickly retrieve data that has already been seen. This means caching **helps speed up application performance while additionally increasing efficiency.**

A cache is valuable as it enables us to salvage precious computing power. Instead of re-retrieving  data, we have already seen we retrieve it from the cache. **Talented software engineers distinguish themselves from others by seeking and acting on opportunities that optimize their work in areas that add value. Caching is one such opportunity for high value.**

![Application optimization shown via graphs](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792499/article/cache-pro-con/application_optimization_via_graph_jctfru.jpg)

Compute power and memory strike a delicate balance. Oftentimes the cross-comparison of CPU preservation vs memory tests an engineer's situational awareness. Caches aren't *always* the optimal solution, but knowing when to use a cache and how best to implement a cache system makes for a powerful developer!

## Pros of Caching in Software Engineering
![Pros of caching](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792516/article/cache-pro-con/pros_of_caching_syvyct.jpg)

**Advantages of caches:**

* Retrieve content faster for applications, ex: websites
* Improve application performance
* Accelerate data retrieval (decrease time duration for queries)
* Reduce latency of database querying
* Reduce the load on backend services
* Reduce database cost
* Reduce CPU usage
* Enable predictable performance
* Increase IOPS (Input/Output operations per second)
* Improve user experience (faster load times = happy consumer)
* Content is still available in network issues
* Save device energy

Believe it or not, this long list contains just a few of the reasons a cache should be considered in software engineering efforts.

Let's expand on some of the  above points for why caching should be considered for applications.

### Caches reduce the load on backend services
Caches  redirect parts of the read load from the backend database to the in-memory layer. **Because caches are stored in memory, caches naturally reduce the load on databases.** The latency of database querying improves because our **databases have fewer queries to address.**

Fewer database queries tie into the point that caching decreases database expenses as well. **Less database querying means less cost!**

![Less database cost follows with less database querying](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661795500/article/cache-pro-con/cost_reduction_rxrnvj.webp)

Finally, if we reduce the returns to the database, we decrease the strain on our databases, meaning they're better protected! Databases are subject to crashing at times of unexpected spikes. **Traffic spikes are naturally balanced out by caches, as fewer queries end up going back to the database.**

### Enable predictable performance for applications
Traffic spikes in application usage can come in expected time periods. An example involves increased traffic to college basketball pages for sports websites when March Madness comes around yearly. Another example of traffic spikes engineers can anticipate applies to e-commerce sites during Black Friday or scheduled sales. **To combat traffic spikes while reducing costs, developers can choose to lean on a cache.**

![Traffic spikes can be catastrophic for applications](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792644/article/cache-pro-con/catastrophic_traffic_spiike_yhlptp.jpg)

Increased load on databases results in higher latency to retrieve data in web development. The application performance rapidly becomes unpredictable as engineers encounter these uncommon scenarios. **A high throughput in-memory cache mitigates the risk as caches preclude the increase in traffic from making a mark on databases, ensuring applications behave in predictable manners.**

### Improve Application Performance
Those with Computer Science or Engineering backgrounds know that <Anchor href="https://www.bytesizedpieces.com/posts/analyze-big-o-notation#where-can-we-apply-big-o-notation">CPU Usage and Memory Usage are constantly at odds.</Anchor> We learned that storage tends to be slower than memory, as a result, we need to exercise caution when relying on storage.

**The engineer's aspiration involves spending as little computing power as possible while achieving the fastest results without too much reliance on storage.**

![Tradeoff evaluations software engineers balance for applications](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792657/article/cache-pro-con/tradeoff_evals_for_swe_lr6pfe.png)

Fulfillment of this goal may be tough if an engineer lacks the understanding of RAM vs Disk. Let's dive into the components we need to understand, which in turn will help us see how powerful a concept caching is.

## Random access memory (RAM) vs Disk
**RAM (Random access memory) has a conveniently fast path to the computer's CPU. The disk is much slower given our CPU has to determine where the long-term data is actually being stored.** As a result, memory access is substantially faster than reads from disk.

Databases tend to use disks to store our data. The line of thought for disk storage versus memory comes down to the longevity of our data persistence. If the data is something we want to keep for a long period of time, it's better stored on disk. This is why database reads can be slow and should actively be avoided if possible!

Reduction in the necessity of database reads and reliance on memory means we get our data faster, which in turn means our application performs better. **Significantly faster data access improves the overall performance of the application.**

![Memory vs Storage](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792669/article/cache-pro-con/memory_vs_storage_jzf7xv.png)

**In-memory systems tie into an increase in IOPS or Input/Output operations per second.** One instance of a single in-memory data store used as a cache has the potential to serve hundreds of thousands of requests per second. Let's break that down into ms to see the dramatic time improvement caching brings to the table.

Your average RAM chip accesses memory in 10 nanoseconds. Disk Access Time is the sum of Access Time and Data Transfer Time.

<br />

`Disk Access Time = Access Time + Data Transfer Time`

<br />

**Access Time is defined as the amount of setup time needed to read from the right segment of the data block.**

**Data Transfer Time is defined as the time it takes to transfer the found data between the system and the disk.**

![The complex structure for the disk that needs to go through for data retrievals](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792683/article/cache-pro-con/simplified_disk_structure_eyfavz.gif)

Disk Access Time is the sum of Access Time and Data Transfer Time, resulting in an average 25–100 milliseconds time duration for disk data retrieval. Let's evaluate disk retrievals via the best-case scenario, 25 milliseconds.

Recall earlier I mentioned your average RAM chip accesses memory in 10 nanoseconds. We already can see a major difference when we compare 10 nanoseconds for memory access to 25 milliseconds for disk access (1 nanosecond is 0.000001 milliseconds)!

But let's press on. We are comparing 100,000 requests made to disk versus memory.

**100,000 requests to memory would take 1,000,000 nanoseconds or just 1 millisecond.**

**100,000 requests to disk would take 2,500,000 milliseconds.**

Compare 1 millisecond to 2.5 million milliseconds! That is the difference we are discussing when we propose using caches (memory access) to database access (disk).

*Recall additionally that I chose the best case scenario for the time it takes to retrieve data from the disk. This means that retrievals of data from disk can only get worse.*

Caching when done right can turn a bog of a system into a snappy little vixen, and we just saw the proof in numbers!

![Fast vs slow application](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792691/article/cache-pro-con/fast_v_slow_application_kpt1im.jpg)

## Cons of caching in software engineering

**Some disadvantages to caching:**

* Storage consumption
* Risk of outdated content (stale data)
* Incorrect or corrupt cached data
* Security risk
* Cache invalidation is difficult
* Overhead and Complexity
* When the device turns off cache is lost

Notice that although the list is shorter for disadvantages to caching, they are serious. If the software engineer responsible for implementing a cache doesn't strike the right balance, we can be in trouble with an application misrepresenting our data or posing security vulnerabilities.

How about we discuss some of these disadvantages of caching data?

**The risk of outdated content or stale data**

The premise of caching revolves about retaining prior retrieved data. We force data we pulled a certain time period back to "stick around" for longer. In doing so, software developers pose the risk of presenting data that is old. We don't necessarily know that this data stored is dated because we do not go back to the database to retrieve what's been updated.

Let's proceed with an example of **how our cached data can become outdated or stale.**

Popular E-commerce sites like Amazon or Warby Parker get millions of page views daily. These visits could mean an abundance of database queries for products that the software engineers decide could be cached.

![E-Commerce Sites need to handle high traffic](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792707/article/cache-pro-con/e_commerce-site-traffic-handle_g2yygz.png)

Let's say the software engineers decide to update their cache system once a day.

It should go without saying that at least one of these products is subject to change within 24 hours. We can also agree that any alteration to a product should show up immediately on an E-commerce site. Imagine a product goes on sale 2 hours after the cache has been updated. A consumer wouldn't see that sale on the product until 22 hours pass and the cache gets updated!

![Potential Caching Architecture](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792723/article/cache-pro-con/potential_cache_architecture_srxzgf.png)

With the e-commerce example, we observe the necessity for caution when using caches. **Software engineers need to be considerate about choosing when the cache is deemed invalid and in need of updating.**

My above scenario now becomes one revolving around the concept of <Bold><Anchor href="https://yihui.org/en/2018/06/cache-invalidation/">cache invalidation.</Anchor></Bold> Software engineers need to consider the following question.

<BigQuotes>What is the acceptable worst-case scenario for determining cache invalidation timing?"</BigQuotes>

Perhaps instead of 24 hours, our team of software engineers for e-commerce sites decides to bring the cache invalidation timing down to a matter of a few seconds.

Now we ask again the same question.

If the product manager updates the pricing of a product, is it acceptable that a consumer receives that update a few seconds later?

Now we get to the famously challenging part of caching, as the software engineer's situational awareness is put to the test.

Consider the following cases for the second cache invalidation idea: Is the product a time-sensitive sale, or always an offering on the website? Do these particular products have high visibility and lots of traffic that could impact large sums of users?

Perhaps if the product is time sensitive, or has the potential of being time-sensitive, then maybe even a few seconds is not good enough for cache invalidation. If the product is always an offering on the website, a few seconds is not the end of the world. In cases of high visibility and traffic, perhaps a few seconds would not be doable for purging the cache. It becomes far more likely that a large volume of users will receive this outdated information.

The demand for in-depth reasoning, careful consideration, and meticulous information gathering of cases makes decision-making for caches a genuine challenge for software engineers.

![Caching is hard, but worth going to the end for!](https://res.cloudinary.com/bytesizedpieces/image/upload/v1661792733/article/cache-pro-con/caching_is_hard_tsrdls.png)

### Overhead and Complexity
The balance software developers need to strike to avoid stale data while improving application efficiency and performance is in fact overhead. Caching can become far more complex if they go by methods besides timestamps, which makes it harder for developers to maintain the code written and understand the source of problems if they arise.

Engineers typically rely on a variation of problem-solving methodologies to avoid these potential issues, but naturally, such alternative methods introduce even more complexity. Again, situational awareness is key. Engineers need to understand the implications of the decisions they make, for the sake of both users and their own coworkers!

## Summary
Bear in mind that a little can get you a long way. Don't let what-if scenarios bog you down and deter you from huge performance gains.

**The main benefits of caching are:**

* Reduction of load on backend services
* Predictable Performance
* Application Performance Improvements

**The potential cons associated with caching:**

* Outdated/Stale data
* Poor choices for cache invalidation
* Overhead and Complexity

Happy caching everyone!

---

Remember, developers are creatures that turn coffee into code. So I'd very much appreciate if you <Anchor slug="overcome-imposter-syndrome" action="buy_coffee_overcome-imposter-syndrome_click" href={"https://www.buymeacoffee.com/stefanagloginic"} styles={{ 'display': 'inline-flex', 'alignItems': 'center' }}>bought me a coffee! <TinyIcon alt="buy me a coffee icon" styles={{
    width: '23px',
    height: '23px',
    marginLeft: '3px'
}} src={"https://res.cloudinary.com/bytesizedpieces/image/upload/v1639952578/assets/buymeacoffee_udskx4.png"} /></Anchor> I’m a new writer and I will be posting very frequently on my findings and learnings in the tech industry and beyond. Join my newsletter if you would like to stay tuned!

Thanks for reading again! ❤️

Caching intimidates developers of all skill levels. However, despite its challenges, caching is an important concept a developer should devote the time to wrap their minds around. When applied correctly caching can bring about dramatic improvement to

Describes what a cache is, summarizes the pros and cons of caching in software engineering, with a specific focus on caching data in web or mobile applications to improve performance.

Cache Pros and Cons

Pros and Cons of Caching Data in Software Development

Emails are central to any successful business operation. Emails are widely used and accessible to a large audience. This makes emails the perfect solution for communication between businesses and consumers. That being said, setting up a reliable mailing system can seem daunting! Setting up your own SMTP server

Article summarizing how to use Firebase Mail Trigger Extension and SendGrid to automate emails by adding documents in Firebase Firestore databases. Covers testing, debugging and potential SMTP URI connection issues that may occur during the setup process.

Email automation with Firebase Mail Extension and SendGrid

Email Automation made easy with Firebase Trigger Mail Extension and SendGrid


import { FormattedMessage } from 'react-intl';

import Anchor from '../components/article/Anchor';
import Bold from '../components/article/Bold';
import BigQuotes from '../components/article/BigQuotes';
import TinyIcon from '../components/article/TinyIcon';

Animating a loader is a topic I felt I should write about given I remember being so confused about how to get a loader when starting out with frontend development.

![Animated Loader](https://res.cloudinary.com/bytesizedpieces/image/upload/v1656084931/article/a-how-to-guide-on-making-an-animated-loading-image-for-a-website/animated_loader_gif_n6b5x0.gif)

I’d ask myself questions like…

<BigQuotes>Do I create a gif?</BigQuotes>

Personally, I wouldn’t recommend using a gif. Depending on your loader, it may be hard to make the background transparent. Gifs also don’t scale smoothly, have a limited image quality given their color limit, and are difficult to change.

<BigQuotes>Do I try to learn how to animate an SVG?</BigQuotes>

Yes, you can do this if you have a more complex animation in mind. There is a <Anchor href={"https://www.svgator.com/"}>cool tool called SVGator</Anchor> that can help you achieve your desired effect with ease! You just start by providing SVGator with the static image you want to become animated. <Anchor href={"https://www.youtube.com/watch?v=SPcPL4hWvIo&ab_channel=SatoriGraphics"}>Here is a great video to refer to if you wanted to give SVGator a try.</Anchor>

<BigQuotes>Do I use CSS to animate an SVG or PNG?</BigQuotes>
That works too! The easiest way if you ask me to accomplish getting an animated loader, is to find a static SVG you like (needs to be a circle in this example) and apply some CSS to it. There are some negatives to this approach like performance and browser compatibility (pesky IE!) so bear this in mind when selecting this approach. Let’s get to an example, shall we?

## Animating an SVG Image
I’m picking the <Anchor href={"https://fontawesome.com/search?q=spinner&s=solid%2Cbrands"}>spinner SVG</Anchor> from the <Anchor href={"https://fontawesome.com/"}>font awesome 6 packages.</Anchor> Now, there are two ways we can go about animating our SVG. If we follow along to <Anchor href={"https://fontawesome.com/v6/docs/web/style/animate"}>font-awesome documentation</Anchor>, we can animate our spinner like so:

`<i class="fa-solid fa-spinner fa-spin"></i>`

<Bold>fa-spin</Bold> is the class we want to focus on as it enables the animation to take place, by making the icon spin 360° clockwise. Now, I didn't want to install the entire font-awesome package, so I achieved this behavior in a different manner.

I installed the icons on my local device, then created an SVGs folder in my project.

![folder structure for svgs](https://res.cloudinary.com/bytesizedpieces/image/upload/v1656085355/article/a-how-to-guide-on-making-an-animated-loading-image-for-a-website/folder_rrpafw.png)

I only copied in the folder the SVGs that I wanted to leverage in my project. I then enabled my project to directly import the SVGs and render them. With this approach, I get the added advantage that I can modify the fill attribute myself and effectively control the color of the SVG as well! To achieve this, I installed <Anchor href={"https://www.npmjs.com/package/@svgr/webpack"}>svgr for webpack</Anchor> and modified my next.config.js to look like the following (Note that I am using the <Anchor href={"https://vercel.com/solutions/nextjs"}>NextJS</Anchor> framework for my project):

```jsx
/** @type {import('next').NextConfig} */
module.exports = {
    reactStrictMode: true,
    webpack(config) {
        config.module.rules.push({
            test: /\.svg$/,
            use: ["@svgr/webpack"]
        });
        return config;
    }
};
```

Now, I have the capability to import and render my SVGs like so:

```jsx
// SVGS
import Loader from '../svgs/loader.svg';

...

<Loader id="loader" width={30} fill={COLORS.blues.mediumA.color} />
```

The last step is to animate the loader!

I added the following CSS targets and keyframes:

```css
#loader {
    animation: spin 1s linear infinite;
}
@keyframes spin {
    0% {
        transform: rotate(0deg);
    }
    100% {
        transform: rotate(360deg);
    }
}
```

This CSS tells my loader to use the keyframe named spin, from 0% to 100% of the animation, I want my loader to start at 0 degrees and end up at 360 degrees by the end of the sequence.

We want this to happen within 1s and keep going infinitely in a linear fashion.

The final result looks like *this*!

![My final animated loader demonstrated](https://res.cloudinary.com/bytesizedpieces/image/upload/v1656085547/article/a-how-to-guide-on-making-an-animated-loading-image-for-a-website/animated_loader_hghpbl.gif)

I hope you found this helpful and happy coding!

Remember, developers are creatures that turn coffee into code. So I'd very much appreciate if you <Anchor slug="overcome-imposter-syndrome" action="buy_coffee_overcome-imposter-syndrome_click" href={"https://www.buymeacoffee.com/stefanagloginic"} styles={{ 'display': 'inline-flex', 'alignItems': 'center' }}>bought me a coffee! <TinyIcon alt="buy me a coffee icon" styles={{
    width: '23px',
    height: '23px',
    marginLeft: '3px'
}} src={"https://res.cloudinary.com/bytesizedpieces/image/upload/v1639952578/assets/buymeacoffee_udskx4.png"} /></Anchor> I’m a new writer and I will be posting very frequently on my findings and learnings in the tech industry and beyond. Join my newsletter if you would like to stay tuned!

Thanks for reading again! ❤️

Animating a loader is a topic I felt I should write about given I remember being so confused about how to get a loader when starting out with frontend development...

Article providing a guide on how a frontend web developer can go about making animated loading images. Gifs, CSS animations for PNGS or SVGS are covered which uses font awesome as well for icons.

Guide on making animated loading images

A How-to Guide on Making an Animated Loading Image for a Website


import { FormattedMessage } from 'react-intl';

import Anchor from '../components/article/Anchor';
import Bold from '../components/article/Bold';
import Italics from '../components/article/Italics';
import TinyIcon from '../components/article/TinyIcon';

![chrome devtools image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452257/article/chrome-devtools-for-beginners/devtools_lst44h.jpg)

When first starting off with web development, it’s easy to get overwhelmed by the many tools out there. Developer tools are certainly handy, but you only need a couple in your arsenal to be an effective developer. Chrome DevTools is one tool that is absolutely crucial to any web developer. In fact, I would say Chrome DevTools is my #1 tool, it helps me get the job done 99% of the time. So without further ado, let’s dive into what makes Chrome DevTools so useful! And why I recommend it to anyone involved with web development.

*What are the Chrome DevTools? How do I access it?*

I’m glad you asked! <Bold>Chrome DevTools is a set of useful web developer tools that can be accessed directly in your chrome browser.</Bold> One way you can access it is by left clicking on the element you’re interested in viewing, selecting *Inspect*, and bam! The DevTools will open either appended to the bottom of your window, or you can set it to append to the left or right of the window respectively. Another way to bring it up is Option + ⌘ + J (on macOS), or Shift + CTRL + J (on Windows/Linux).

![chrome devtools console image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452245/article/chrome-devtools-for-beginners/incognito_x00h4g.png)

Let’s start breaking down the parts to the DevTools you have available here bit by bit.
<Bold>Elements Tab:</Bold> gives a breakdown of the <Anchor href="https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction">DOM</Anchor> elements composing your page. As a developer, you may want to see *if* your element is rendering, *where* it is rendering, or maybe you want to mock things out before going back to the code.

![chrome devtools elements breakdown image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452258/article/chrome-devtools-for-beginners/elements_r7izzw.png)

<Bold><Italics>Using the Elements tab, here are just a few ways we can manipulate the DOM…you can:</Italics></Bold>

* Insert a new DOM element at a certain level in the tree

![chrome devtools elements insertion image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452313/article/chrome-devtools-for-beginners/addElement_s8p4t2.png)

* Change some styling to the elements

![chrome devtools elements style change image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452536/article/chrome-devtools-for-beginners/styling_hsfcho.png)

* Add/Change attributes, tag names, or inner text

![chrome devtools elements attributes change image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452313/article/chrome-devtools-for-beginners/add_change_attrs_ujje7e.png)

* Remove DOM nodes entirely from the page by selecting it in the elements tab, and hitting the backspace button
* Do a deeper dive into the <Anchor href="https://www.w3schools.com/css/css_boxmodel.asp">Box Model</Anchor>, see how it is set up for the elements. You can examine the margin, border, padding, and content size of the particular element.

![chrome devtools elements styles inspection image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452150/article/chrome-devtools-for-beginners/examine_r9v8dd.png)

<Bold>Console tab:</Bold> This is where a lot of magic happens. You can test out your Javascript here and you can see the logs you’ve added for your website. You can view standard information logs, verbose logs, warnings, and errors. *Remember, verbose logs should not be exposed in production!*

![chrome devtools console usage image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452148/article/chrome-devtools-for-beginners/console_df6doo.png)

<Bold>Network tab:</Bold> Allows you to view the network activity of your webpage. Don’t worry! I’ll give an example. Let’s say I go on Youtube, and type something in the search input…

![chrome devtools network usage image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452147/article/chrome-devtools-for-beginners/network_jnw4hr.png)

Interesting! We can see a GET request going out. You can see the Request URL, that the Status Code was 200 or successful, and dig even further into the headers. If you click on Response, you can see what the service has come back with, to answer the GET request.

![chrome devtools network response review image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452145/article/chrome-devtools-for-beginners/networkRes_vgxte0.png)

So it seems this is a GET request for suggestions based on what I type! One of the suggestions is “you got this travis”. Yep, Travis definitely does! You’ve got this too!
The network tab is quite useful when you have queries that are failing as well. When used the same way, you can track the Status Code, the details of the request you are making, or the response error code from the API to see what the culprit of the problem is.
Another key feature in the Network tab is the Waterfall. This gives you a breakdown of the length of time it took for each query to complete, and can help track down particularly slow endpoints or resources that could be making our website less performant.

![chrome devtools network all requests image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452144/article/chrome-devtools-for-beginners/recordedNetwork_vmwnhb.png)

<Bold>Performance tab:</Bold> The performance tab is a tool that enables us to capture a profile of our website and evaluate its performance. We can source if any bottlenecks are present, and at what stage they occur. We can also identify why a timeout may be occurring.
See the grayed out button at the near top left corner? You can use this to start a recording and establish your profile.
Let’s go with an example. When you load Youtube, you want to make sure you can see the initial loading skeletal state and how long that state is shown for, until we see some actual content populating.

![chrome devtools performance tab image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452139/article/chrome-devtools-for-beginners/performance_audukg.png)

After stopping the recording, you can see a breakdown of the initial page load.

![chrome devtools performance tab breakdown image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452774/article/chrome-devtools-for-beginners/recording_jl6ncs.png)

Notice that for 122ms we are presented with the skeletal template of Youtube, as it is fetching the resources necessary to provide us some real content. Not bad!

![chrome devtools performance tab frames image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452826/article/chrome-devtools-for-beginners/recordingView_h0m0j7.png)

The performance tab allows us to really get into the nitty gritty details of our websites, to make them really shine for our users.

<Bold>Application tab:</Bold> The application tab allows us to investigate our service workers if there are any set up, local and session storage details, cookies and more.

For <Anchor href="https://developer.chrome.com/docs/devtools/storage/localstorage/">Local Storage</Anchor> or <Anchor href="https://developer.chrome.com/docs/devtools/storage/sessionstorage/">Session Storage</Anchor>, we can see the key value pairings the website has setup in our Chrome browser. Please note that session storage has a similar UI! We can edit the values, or delete the pairing if we are testing certain conditions or handlings of our website.
I’ll provide an example to provide some clarity on the use case of this section. Let’s say I have a website, where after the user views it three times, I want to present them with an interstitial to subscribe. I also want to keep presenting it until the subscription flow is completed by our user. Instead of loading my website over and over until I reach the required interstitial count, I can just go to Local Storage and edit the value immediately to force the condition.

![chrome devtools local storage image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452922/article/chrome-devtools-for-beginners/localstorage_mrv29u.png)

Cool! So the Local Storage section of the Application tab has effectively helped me save some time.
The last thing I wanted to cover about the Chrome DevTools, is the capability of <Bold>testing out our websites responsiveness.</Bold>

![chrome devtools responsive simulation image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641452970/article/chrome-devtools-for-beginners/header_txeqs8.png)

We can see at the top left corner of the tool an icon that shows a small screen overlapping a larger screen. This is a quick access toggle tool, which serves the purpose of switching between a wider desktop view and a mobile web view. If we wanted to scope the view to pixel specific, you should see a new header with preset phone device types. You can even manually input the width and the height in the input fields to match your exact desired conditions if you wanted!

![chrome devtools responsive tab image](https://res.cloudinary.com/bytesizedpieces/image/upload/v1641453008/article/chrome-devtools-for-beginners/mobile_ofqrps.png)

---

Remember, developers are creatures that turn coffee into code. So I'd very much appreciate if you <Anchor slug="overcome-imposter-syndrome" action="buy_coffee_overcome-imposter-syndrome_click" href={"https://www.buymeacoffee.com/stefanagloginic"} styles={{ 'display': 'inline-flex', 'alignItems': 'center' }}>bought me a coffee! <TinyIcon alt="buy me a coffee icon" styles={{
    width: '23px',
    height: '23px',
    marginLeft: '3px'
}} src={"https://res.cloudinary.com/bytesizedpieces/image/upload/v1639952578/assets/buymeacoffee_udskx4.png"} /></Anchor> I’m a new writer and I will be posting very frequently on my findings and learnings in the tech industry and beyond. Join my newsletter if you would like to stay tuned!

Thanks for reading again! ❤️

When first starting off with web development, it’s easy to get overwhelmed by the many tools out there. Developer tools are certainly handy, but you only need a couple in your arsenal to be an effective developer. Chrome DevTools is one tool that is absolutely crucial to any web developer. In fact

Article summarizing Chrome DevTools and it's importance for web development debugging and testing purposes. It covers the Elements, Console, Sources, Network, Performance, Memory, Application, and screen size testing tools for frontend web development.

TOOLS

Understanding Open Graph Dynamic Image Meta Tags with NextJS

Metadata and meta tags are underestimated SEO optimization tools. Newer web developers may completely gloss over this powerful technique to improve the discoverability of the websites for which they are responsible....Read more

How to decide between the use of custom solutions and ones that bloat your website

The tech industry never stops expanding with alternate solutions. We see a beautiful landing page and get sucked into wanting to integrate these tools into our workflow. The problem with these solutions is that they sometimes can bloat our projects where this could be avoided....Read more

Understand types of caches and policies to better build your own system

Cache systems demand a delicate balance for applications, as they have many pros and cons. When done right, caching can be performant, increase application speed and decrease costs. However, the integration of cache systems poses the risk of application reliability downgrade...Read more

How to make sure your cache design system is reliable for your website

We live in a world governed by fast results. Caching is one such technique that enables rapid responses online. This begs the question for software engineers: "How do we go about deciding how best to design a cache that fits the needs of our application?"...Read more

Pros and Cons of Caching Data in Software Development

Caching intimidates developers of all skill levels. However, despite its challenges, caching is an important concept a developer should devote the time to wrap their minds around. When applied correctly caching can bring about dramatic improvement to...Read more

Email Automation made easy with Firebase Trigger Mail Extension and SendGrid

A How-to Guide on Making an Animated Loading Image for a Website

Animating a loader is a topic I felt I should write about given I remember being so confused about how to get a loader when starting out with frontend development......Read more

Chrome DevTools for Beginners

Join our newsletter to read delightful bytesizedpieces every Monday!

Understanding Open Graph Dynamic Image Meta Tags with NextJS

Metadata and meta tags are underestimated SEO optimization tools. Newer web developers may completely gloss over this powerful technique to improve the discoverability of the websites for which they are responsible.....css-1ncj5p5{color:var(--theme-ui-colors-red-color);}Read more

How to decide between the use of custom solutions and ones that bloat your website

The tech industry never stops expanding with alternate solutions. We see a beautiful landing page and get sucked into wanting to integrate these tools into our workflow. The problem with these solutions is that they sometimes can bloat our projects where this could be avoided....Read more

Understand types of caches and policies to better build your own system

Cache systems demand a delicate balance for applications, as they have many pros and cons. When done right, caching can be performant, increase application speed and decrease costs. However, the integration of cache systems poses the risk of application reliability downgrade...Read more

How to make sure your cache design system is reliable for your website

We live in a world governed by fast results. Caching is one such technique that enables rapid responses online. This begs the question for software engineers: "How do we go about deciding how best to design a cache that fits the needs of our application?"...Read more

Pros and Cons of Caching Data in Software Development

Caching intimidates developers of all skill levels. However, despite its challenges, caching is an important concept a developer should devote the time to wrap their minds around. When applied correctly caching can bring about dramatic improvement to...Read more

Email Automation made easy with Firebase Trigger Mail Extension and SendGrid

A How-to Guide on Making an Animated Loading Image for a Website

Animating a loader is a topic I felt I should write about given I remember being so confused about how to get a loader when starting out with frontend development......Read more

Chrome DevTools for Beginners

Join our newsletter to read delightful bytesizedpieces every Monday!

Metadata and meta tags are underestimated SEO optimization tools. Newer web developers may completely gloss over this powerful technique to improve the discoverability of the websites for which they are responsible....Read more