News Feed

Our goal is to design a news feed application that contains a list of feed posts users can interact with.Requirements:

Requirements

Out of scope (for now): ⁠Mobile support is a B feature.

High-level design: ⁠

Data model

API ⁠Here are basic functionalities we need:

The most interesting API to talk about will be the HTTP API to fetch a list of feed posts because the pagination approach is worth a discussion.

Pagination considerations:

There are two common ways to return paginated content, each with its own pros and cons.

1) Offset-based pagination

An offset-based pagination API accepts `size` and `page` as parameters. So for  parameters of {size: 5, page: 2}, the SQL query would be like: ⁠SELECT * FROM posts LIMIT 5 OFFSET 0;  // page1 ⁠SELECT * FROM posts LIMIT 5 OFFSET 5;  // page2

Some downsides of offset-based pagination is that for data that updates frequently, the current page window might be inaccurate after some time. Imagine a user has fetched the first 5 posts in your feed. After sometime, 5 more posts were added. If the users scroll to the bottom of the feed and fetches page 2, the same posts in the original page 1 will be fetched, and the user will see duplicate posts.

Another downside of offset-based pagination is that you cannot change the page size for subsequent queries since the offset is a product of the page size and the page being requested.

2) Cursor-based pagination

Cursor-based pagination works by returning a pointer/position to the last item in the results. On subsequent queries, the server passes the cursor to the database, and the database knows to return items after that cursor.

This avoids the inaccurate page window problem because new posts added over time do not affect the offset, which is determined by a fixed cursor.

A cursor-based pagination API uses `size` and `cursor` parameters.

For an infinite scrolling news feed where new posts can be added frequently to the top of the feed, newly fetched posts are appended to the end of the feed, or/and table size can get grow large really quickly, cursor-based pagination is the most suitable for news feed use cases.

Rendering approach:

Traditional web applications have multiple choices on server-side rendering or/and client-side rendering. News feed applications are somewhere in-between, there's a good amount of static content but they also require interaction.  In fact, modern UI JavaScript frameworks like React and Vue, along with meta frameworks like Next.js enable this.

Implementing Infinite Scroll:

Use the  to monitor when the marker element is entering or exiting another element () or intersect by a specified amount. (See Intersection Observer API in MDN doc).

Virtualized Lists

With infinite scrolling, all the loaded feed items are on one single page. As the user scrolls further down the page, more posts are appended to the DOM and with feed posts having complex DOM structure, the DOM size rapidly increases. And the feed items list can easily grow really long quickly, the number of feed items can be a cause of performance issues in terms of DOM size, rendering, and browser memory usage.

Virtualized lists is a technique to render only the posts that are within the viewport to improve browser paining.

Faster performance

Code splitting (javascript) is a technique to split code needed on a page into separate files so that they can be loaded in parallel or when they're needed. For a news feed application, there's only a single page, so page-level code splitting is not too relevant, however lazy loading can still be super useful for other purposes.

Also, lazy loading dpendencies helps speed up rendering news feed posts. ⁠For example, non-crucial features that can be lazy loaded are: image pickers and background images.

Optimistic Updates

Optimistic update is a performance technique where the client immediately reflect the updated state after a user interaction that hits the server and optimistically assume that the server request succeeds, which should be the case for most requests. This gives users instant feedback and improves the perceived performance. If the server request fails, we can revert the UI changes and display an error message.

For a news feed, optimistic updates can be applied for reaction interactions by immediately showing the user's reaction and an updated total count of the reactions.

Icon Rendering

There are a few ways to render icons. For instance, use icon-fonts, svg, inline-svg, sprite sheet, and etc. Inline SVGs seem to be the trend now a days.

There are more topics to cover such as image rendering, timestamps, accessibility, and so on. I should come back for them later.