Module 20 · Lesson 2 of 10Free

Design: a photo-sharing app like Instagram

Level

Intermediate

Topic

End-to-end design problems

Reading time

30 min

Asha uploads a 2 MB outfit photo to Kartly. Her phone loses signal after the upload finishes. She retries, and her followers see two posts—one with a broken image.

How can one photo become one usable post, even when uploads, processing and feed delivery fail independently?

We’ll design Kartly’s new photo-sharing feature, not describe Instagram’s internal architecture. Kartly is already at its many-services stage. We’ll start with a small implementation inside that system, then add complexity only where this feature needs it.

Separate the bytes from publication

The first boundary separates a file from a post. A file is stored bytes. A post is a database record that says who owns those bytes and whether readers may see them.

Version 1 uses direct uploads, covered in Direct uploads to object storage. The API handles small control requests. Object storage handles large transfers.

A CDN is a network of nearby servers that caches image responses. We choose it because many followers request identical bytes. We keep originals and generated images in private object storage; only approved delivery paths can read them.

For now, the API resizes the image during the completion request. A successful response means the post is published. This works for a controlled rollout, but ties the phone’s request to image-processing time.

Product answers set the boundaries

These are design assumptions, not existing Kartly measurements. Existing customers, catalog traffic and orders remain unchanged.

We exclude video, stories, direct messages and automated recommendation ranking. Caption editing is also outside this first version. Those features introduce different processing or ordering rules.

Publication correctness is stricter than feed freshness. A delayed photo is acceptable during an incident. A feed item pointing to unfinished bytes is not.

Size reads and writes separately

For this lesson’s adoption scenario, assume Kartly’s 5 million daily customers use the photo feature. Assume 1 million photos arrive daily. Each customer opens ten feed pages.

The following estimates use decimal storage units. They exclude replication, backups and protocol overhead.

The estimates justify separate scaling paths. More API instances do not reduce image-transfer costs. More processing workers do not make an expensive feed query cheaper.

Make one upload one intention

The Photo API owns the publishing contract. The client chooses a idempotency key before its first request and stores it locally until the operation finishes.

This builds on Idempotency: making retries safe. The key identifies an intention, not a file’s contents. Posting the same photo tomorrow is a different intention.

The API owns object names

POST /v1/photo-posts
Authorization: Bearer customer-token
Idempotency-Key: publish-asha-017
Content-Type: application/json

{
  "caption": "My festival outfit",
  "upload": {
    "bytes": 2000000,
    "content_type": "image/jpeg",
    "sha256_base64": "base64-encoded-sha256"
  }
}

HTTP/1.1 201 Created
Location: /v1/photo-posts/p_17
Content-Type: application/json

{
  "id": "p_17",
  "state": "UPLOADING",
  "upload": {
    "method": "PUT",
    "url": "https://uploads.kartly.example/signed-upload",
    "required_headers": {
      "Content-Type": "image/jpeg",
      "x-amz-checksum-sha256": "base64-encoded-sha256"
    },
    "expires_in_seconds": 600
  }
}

The API chooses a unique staging key. It never accepts a client-selected storage path.

The unique database constraint is (author_id, request_key). A retry returns the same post ID. Changed caption or upload metadata with the same key returns 409 Conflict. Store those original request fields for comparison.

An S3 presigned URL authorizes a particular operation against an object. It can be reused before expiry, and another upload to the same key replaces the object. It is not a single-use ticket. ^{1Download and upload objects with presigned URLsAmazon Web Services, Amazon S3 User Guide}

Our design signs the checksum header and requires the original checksum on every retry. S3 supports checksum validation with Signature Version 4 presigned uploads. ^{1Download and upload objects with presigned URLsAmazon Web Services, Amazon S3 User Guide}

That prevents later retries from substituting different bytes under the same upload contract. It does not prove the file is safe. The server still needs to decode it.

Completion is a separate request

After the PUT succeeds, the client sends:

POST /v1/photo-posts/p_17/complete
Authorization: Bearer customer-token
Content-Type: application/json

{}

Version 1 checks ownership, reads the stored object, validates it and generates feed-sized variants. It then publishes the post and returns its representation.

Malformed requests return 400. Oversized upload declarations return 413. Missing authentication returns 401. Requests against another author’s writable resource return 404 to avoid confirming its existence.

The upload contract solves duplicate identity. It does not solve a slow completion request.

Move processing behind a durable job

Version 2 changes completion to return 202 Accepted with state: PROCESSING. The phone uses polling against the post resource until publication or failure.

Publication is an explicit transition

The post follows a state machine. PROCESSING is not a weaker form of published. Feed queries exclude it entirely.

In one database transaction, completion changes UPLOADING to PROCESSING and inserts the processing job. If either operation fails, neither commits.

Workers claim jobs with a short transaction. They record a lease deadline and increment a generation number. They release the database transaction before reading or decoding image bytes.

PostgreSQL’s SELECT FOR UPDATE locks the selected rows against competing writers until transaction end. We use that behavior only for short ownership and finalization transactions. ^{213.3. Explicit LockingPostgreSQL Global Development Group, PostgreSQL 17 Documentation}

Keep the authoritative records compact

The Photo service owns these tables. Existing account records supply author IDs; cross-service account validation is outside this schema.

CREATE TABLE photo_posts (
  id TEXT PRIMARY KEY,
  author_id TEXT NOT NULL,
  request_key TEXT NOT NULL,
  caption TEXT NOT NULL,
  input_key TEXT NOT NULL UNIQUE,
  input_bytes BIGINT NOT NULL CHECK (input_bytes > 0),
  input_sha256 TEXT NOT NULL,
  input_type TEXT NOT NULL,
  state TEXT NOT NULL CHECK (
    state IN ('UPLOADING','PROCESSING','PUBLISHED','FAILED','DELETED')
  ),
  manifest JSONB,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ,
  UNIQUE (author_id, request_key),
  CHECK (state <> 'PUBLISHED' OR
         (manifest IS NOT NULL AND published_at IS NOT NULL))
);

CREATE INDEX photo_posts_author_recent
ON photo_posts (author_id, published_at DESC, id DESC)
WHERE state = 'PUBLISHED';

CREATE TABLE photo_jobs (
  post_id TEXT NOT NULL REFERENCES photo_posts(id),
  kind TEXT NOT NULL CHECK (kind IN ('PROCESS','FANOUT','DELETE')),
  generation BIGINT NOT NULL DEFAULT 0,
  leased_until TIMESTAMPTZ,
  attempts INT NOT NULL DEFAULT 0,
  available_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  progress JSONB NOT NULL DEFAULT '{}',
  done BOOLEAN NOT NULL DEFAULT false,
  PRIMARY KEY (post_id, kind)
);

CREATE INDEX photo_jobs_due
ON photo_jobs (available_at) WHERE NOT done;

The author index serves profile pages and feed retrieval by author. The job index finds due work; claim logic also checks lease expiry. We start with a database-backed work queue, rather than operate another broker immediately.

Validate bytes before trusting metadata

The processor decodes the stored bytes in a resource-limited environment. It ignores the declared image type when choosing a decoder. It rejects excessive dimensions, strips location metadata and generates fixed-size variants.

Here is the small policy function both implementations use after decoding. The decoder supplies these facts; request fields do not.

from dataclasses import dataclass

@dataclass(frozen=True)
class DecodedImage:
    width: int
    height: int
    frames: int

def variant_widths(image: DecodedImage) -> tuple[int, ...]:
    if image.width <= 0 or image.height <= 0:
        raise ValueError("invalid dimensions")
    if image.width * image.height > 40_000_000:
        raise ValueError("pixel limit exceeded")
    if image.frames != 1:
        raise ValueError("only still images are supported")
    return tuple(sorted({min(image.width, n) for n in (320, 640, 1080)}))

type DecodedImage = Readonly<{
  width: number;
  height: number;
  frames: number;
}>;

function variantWidths(image: DecodedImage): number[] {
  const { width, height, frames } = image;
  if (!Number.isSafeInteger(width) || !Number.isSafeInteger(height) ||
      width <= 0 || height <= 0) {
    throw new Error("invalid dimensions");
  }
  if (width * height > 40_000_000) {
    throw new Error("pixel limit exceeded");
  }
  if (frames !== 1) {
    throw new Error("only still images are supported");
  }
  return [...new Set([320, 640, 1080].map(n => Math.min(width, n)))]
    .sort((a, b) => a - b);
}

The pixel limit is this lesson’s illustrative policy. A small compressed file can still require substantial decoding memory. The policy complements decoder resource limits; it cannot replace them.

Each processing attempt writes variants under new, attempt-specific keys. Its manifest records keys, dimensions, types and byte sizes. Retries never overwrite a variant already selected for publication.

Finalization decides which attempt wins

After writing variants, the processor begins a short transaction:

1. Lock its job row and verify the recorded generation still matches.
2. Lock the post row.
3. If the post is still PROCESSING, save the manifest and change it to PUBLISHED.
4. Insert a FANOUT job in the same transaction.
5. Mark the processing job done and commit.

All code touching both rows locks the job first, then the post. Consistent lock order reduces deadlock risk. PostgreSQL documents deadlock detection and recommends consistent acquisition order. ^{213.3. Explicit LockingPostgreSQL Global Development Group, PostgreSQL 17 Documentation}

A worker whose lease expired might still finish. A newer generation prevents that stale worker from selecting its manifest. A deleted post also fails the state check. Its unused variant files become cleanup work.

The reader now has a complete image, but finding the right readers is a separate problem.

Start the feed with indexed reads

Version 2 still builds feeds on demand. The Feed API fetches the viewer’s follow list, retrieves recent published posts and orders them newest first.

Use publication time, not upload creation time. A photo that takes longer to process should enter the feed when it becomes readable.

The Graph service owns follows:

CREATE TABLE follows (
  follower_id TEXT NOT NULL,
  author_id TEXT NOT NULL,
  PRIMARY KEY (follower_id, author_id),
  CHECK (follower_id <> author_id)
);
CREATE INDEX follows_by_author ON follows (author_id, follower_id);

The primary key serves “who do I follow?” The reverse index serves distribution to an author’s followers.

For a small rollout, the Feed API can merge indexed author queries. Later, batching and shard-local queries reduce network calls. Do not issue one sequential service request per followed author.

The Interaction service stores likes with primary key (post_id, viewer_id). Comments use a client-generated ID unique per author. It also indexes (post_id, created_at, id) for comment pages.

These records remain separate from feed entries. Liking a photo must not rewrite every follower’s feed. Like counts may lag; the viewer’s own like state comes from the unique relationship.

A cursor fixes the ordering boundary

A feed response contains post IDs, captions, variant metadata and a next cursor. It does not embed image bytes.

{
  "items": [
    {
      "id": "p_17",
      "author_id": "asha",
      "caption": "My festival outfit",
      "published_at": "2026-10-25T10:00:00Z",
      "image": {
        "path": "/photos/p_17/attempt-4/640.webp",
        "width": 640,
        "height": 800
      }
    }
  ],
  "next_cursor": "opaque-signed-cursor"
}

The cursor carries a fixed upper boundary and the last scanned (published_at, id) pair. The ID breaks timestamp ties. This follows Pagination: offset vs cursor-based.

Newly published posts above that boundary wait for refresh. Delayed distribution can still cause an older item to be missed during one scrolling session. We promise stable ordering, not a frozen snapshot of all future arrivals.

Read-time assembly is easy to repair because posts remain the source. At full adoption, repeated assembly becomes expensive.

Popular authors change distribution costs

The underlying alternatives are developed in Feed generation: fan-out on write vs on read. Here, the important boundary is that feed entries are disposable references, not copies of authoritative post state.

Distribution must survive partial progress

The Feed service owns this derived table:

CREATE TABLE feed_entries (
  viewer_id TEXT NOT NULL,
  post_id TEXT NOT NULL,
  author_id TEXT NOT NULL,
  published_at TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (viewer_id, post_id)
);
CREATE INDEX feed_entries_page
ON feed_entries (viewer_id, published_at DESC, post_id DESC);

The distribution worker reads followers in ID order. It writes a bounded batch with duplicate inserts ignored. Only after the destination acknowledges the batch does it advance the job’s progress cursor.

A crash before cursor advancement repeats a batch. The primary key absorbs duplicates. Advancing first would permanently skip followers after a crash.

This is at-least-once distribution with duplicate-safe inserts, not exactly-once execution. The durable publication job applies the Transactional outbox principle without requiring a separate broker.

New follows read recent author posts while background backfill catches up. Unfollows take effect during final filtering, even if old inbox rows remain.

Merge references, then check permission

The Feed API merges inbox candidates with large-author candidates. It deduplicates IDs, orders them and checks current eligibility.

This small function shows the merge after batched authoritative checks. Its input rank is an integer publication timestamp. Production cursors also carry the fixed upper boundary.

from dataclasses import dataclass

@dataclass(frozen=True)
class Candidate:
    id: str
    published_us: int

def feed_page(
    inbox: list[Candidate],
    pulled: list[Candidate],
    eligible_ids: set[str],
    before: tuple[int, str],
    limit: int = 20,
) -> list[Candidate]:
    if not 1 <= limit <= 100:
        raise ValueError("invalid limit")
    unique = {p.id: p for p in inbox + pulled}
    ordered = sorted(
        unique.values(),
        key=lambda p: (p.published_us, p.id),
        reverse=True,
    )
    return [
        p for p in ordered
        if (p.published_us, p.id) < before and p.id in eligible_ids
    ][:limit]

type Candidate = Readonly<{ id: string; publishedUs: number }>;
type Rank = readonly [number, string];

function feedPage(
  inbox: readonly Candidate[],
  pulled: readonly Candidate[],
  eligibleIds: ReadonlySet<string>,
  before: Rank,
  limit = 20,
): Candidate[] {
  if (!Number.isInteger(limit) || limit < 1 || limit > 100) {
    throw new Error("invalid limit");
  }
  const unique = new Map<string, Candidate>();
  for (const p of [...inbox, ...pulled]) unique.set(p.id, p);
  const ordered = [...unique.values()].sort((a, b) =>
    b.publishedUs - a.publishedUs ||
    (a.id < b.id ? 1 : a.id > b.id ? -1 : 0));
  return ordered.filter(p =>
    (p.publishedUs < before[0] ||
      (p.publishedUs === before[0] && p.id < before[1])) &&
    eligibleIds.has(p.id)
  ).slice(0, limit);
}

IDs use a restricted ASCII alphabet, and timestamp integers must fit TypeScript’s safe integer range. Both implementations therefore use the same ordering.

The function sorts a bounded candidate batch. If filtering leaves too few items, the API scans another batch up to a work limit. The cursor advances past the last scanned candidate, not merely the last visible one.

Large follow lists still make the pull side expensive. Cap supported follows for this version and measure that path before expanding it.

Deletion crosses both delivery paths

A feed inbox can contain yesterday’s permissions. It must never become the authorization database.

OWASP recommends permission checks on every request, including access to static resources. Checking the feed endpoint alone does not protect image URLs. ^{3Authorization Cheat SheetOWASP, OWASP Cheat Sheet Series}

For launch, accounts are public. Deletion still needs a precise contract:

• The owner’s delete transaction sets DELETED and inserts durable cleanup work.
• Feed and profile reads exclude deleted posts using current authoritative state.
• Media delivery checks current post state before serving bytes, including cache hits.
• Cleanup removes feed references, purges cached variants and deletes stored objects.

The media check runs in a controlled delivery layer before the CDN serves a response. Storage origins reject direct public access. A random-looking object path is not permission.

S3 presigned URLs are bearer credentials: anyone possessing one can use its granted access. Keep upload URLs out of logs and never reuse them as permanent viewing URLs. ^{1Download and upload objects with presigned URLsAmazon Web Services, Amazon S3 User Guide}

The correctness path is now complete. Production operations must keep its separate stages visible.

Operate stages, not one success rate

The final system has three independently scaled paths: publication, feed assembly and image delivery.

Trace the write and read paths

Publication:

1. The API creates one upload record.
2. The phone transfers bytes directly to storage.
3. Completion atomically records processing work.
4. A processor stores validated variants.
5. Finalization publishes the manifest and records distribution work.
6. Distribution inserts duplicate-safe feed references.

Reading:

1. The Feed API retrieves inbox and large-author candidates.
2. It checks follows and current post eligibility in batches.
3. It returns ordered metadata and a cursor.
4. The phone requests an appropriately sized variant.
5. The delivery layer authorizes the request before the CDN serves it.

The phone loads images near the viewport instead of downloading a whole feed in advance. Variant dimensions also let it reserve space before bytes arrive.

Detect the failure users experience

Never convert an unavailable feed store into unlimited database scans. That can turn a partial failure into a wider outage. A bounded fallback may return fewer items.

Grow from measured constraints

At ten times this scenario, increase processor capacity only if CPU or processing age requires it. Partition feed entries by viewer because each page belongs to one viewer.

At one hundred times, a single post database may no longer meet write and maintenance targets. Partition posts by author and keep each post’s jobs beside its authoritative row. This preserves local publication transactions.

Changing an author between pull and push modes needs overlap. Backfill the destination mode, read both temporarily and deduplicate before retiring the old mode. An abrupt switch could hide recent posts.

Publishing photos makes duplicate display inconvenient; moving money makes duplicate execution much more costly, which is the next lesson’s problem.

Your turn

Common mistakes

• Publishing when the upload finishes. Stored bytes still need validation and usable variants.
• Using feed entries as post records. Deletion and permission changes would require perfectly updating every copy.
• Advancing distribution progress before inserts succeed. A crash would skip followers permanently.
• Treating a lease as proof the old worker stopped. A generation check must reject stale finalization.
• Choosing a follower threshold without a capacity budget. Distribution time depends on posting rate, recipient count and available worker capacity.

Check your understanding

Summary