ArchGen

Scalable Photo-Sharing Platform Architecture

Focus: Media Storage Flow vs Metadata Flow + Redis Timeline + Fan-out on Write. Key areas: iOS/Android, React (Web), HTTP/2 + TLS.

Use this as a block diagram of the system when explaining architecture.

Preview
Edit this example
Diagram caption: Scalable Photo-Sharing Platform Architecture (Media Storage Flow vs Metadata Flow + Redis Timeline + Fan-out on Write) has 4 layers: Client & Edge Layer, Core Services Layer, Async Fan-out & Processing Layer, Data & Storage Layer.

Prompt

System architecture for a scalable photo-sharing platform like Instagram. The diagram should clearly separate the Media Storage flow (uploading images to AWS S3 served via CloudFront CDN) from the Metadata flow (storing user/post info in a sharded PostgreSQL/Cassandra cluster). Highlight the 'Feed Generation Service' using a Redis Cluster for caching user timelines. Include a 'Fan-out on Write' mechanism for delivering posts to followers' feeds asynchronously via a Message Queue (RabbitMQ/Kafka).
Highlights
  • Layer details · Client & Edge Layer: Modules include Mobile/Web Clients, AWS CloudFront CDN (Media Delivery), API Gateway / Load Balancer (Metadata & Control Plane).
  • Module responsibilities · Client & Edge Layer / Mobile/Web Clients: Initiate media upload and post creation; Render timelines and post details; Authenticate and call backend APIs
  • Layer details · Core Services Layer: Modules include Auth & Identity Service, Media Upload Service (Media Storage Flow), Post & Metadata Service (Metadata Flow), Social Graph Service.

Overview

Scalable Photo-Sharing Platform Architecture (Media Storage Flow vs Metadata Flow + Redis Timeline + Fan-out on Write) has 4 layers: Client & Edge Layer, Core Services Layer, Async Fan-out & Processing Layer, Data & Storage Layer.

Layer details

Show all (4)
  • Client & Edge Layer: Modules include Mobile/Web Clients, AWS CloudFront CDN (Media Delivery), API Gateway / Load Balancer (Metadata & Control Plane).
  • Core Services Layer: Modules include Auth & Identity Service, Media Upload Service (Media Storage Flow), Post & Metadata Service (Metadata Flow), Social Graph Service.
  • Async Fan-out & Processing Layer: Modules include Message Queue / Event Bus (Fan-out Backbone), Fan-out on Write Workers, Media Processing Workers.
  • Data & Storage Layer: Modules include Amazon S3 (Media Object Store) — Media Storage Flow, Sharded PostgreSQL (Canonical Metadata) — Metadata Flow, Cassandra Cluster (Denormalized Views) — Metadata Flow, Redis Cluster (Timeline Cache) — Feed Highlight.

Module responsibilities

Show all (14)
  • Client & Edge Layer / Mobile/Web Clients: Initiate media upload and post creation; Render timelines and post details; Authenticate and call backend APIs
  • Client & Edge Layer / AWS CloudFront CDN (Media Delivery): Serve media globally with low latency; Reduce S3 origin load for reads; Enforce controlled access patterns (optional)
  • Client & Edge Layer / API Gateway / Load Balancer (Metadata & Control Plane): Expose stable transactional APIs; Protect services from abuse and bursts; Route requests to domain services
  • Core Services Layer / Auth & Identity Service: Authenticate users and issue tokens; Authorize access to services; Enforce session security policies
  • Core Services Layer / Media Upload Service (Media Storage Flow): Authorize and coordinate direct-to-S3 uploads; Validate upload policies and metadata; Trigger async processing and persist media references
  • Core Services Layer / Post & Metadata Service (Metadata Flow): Persist canonical post/user metadata; Expose post read/write APIs; Emit domain events for downstream fan-out
  • Core Services Layer / Social Graph Service: Maintain follower relationships at scale; Provide follower sets for fan-out; Enforce privacy constraints
  • Async Fan-out & Processing Layer / Message Queue / Event Bus (Fan-out Backbone): Decouple write path from timeline updates; Provide durable asynchronous processing; Smooth bursts from viral creators
  • Async Fan-out & Processing Layer / Fan-out on Write Workers: Deliver posts to followers' feeds asynchronously; Update Redis timelines efficiently; Ensure eventual consistency with retries
  • Async Fan-out & Processing Layer / Media Processing Workers: Produce delivery-optimized renditions; Store derived artifacts back to S3; Publish MediaProcessed events
  • Data & Storage Layer / Amazon S3 (Media Object Store) — Media Storage Flow: Durably store media objects at scale; Support high-throughput uploads and reads; Enable lifecycle and replication strategies
  • Data & Storage Layer / Sharded PostgreSQL (Canonical Metadata) — Metadata Flow: Maintain canonical transactional metadata; Provide strong consistency for critical writes; Scale horizontally through sharding
  • Data & Storage Layer / Cassandra Cluster (Denormalized Views) — Metadata Flow: Support high-write throughput denormalized models; Enable fast lookups for social graph and indexes; Provide eventual consistency read models
  • Data & Storage Layer / Redis Cluster (Timeline Cache) — Feed Highlight: Cache user timelines for low-latency feed reads; Absorb traffic bursts and hotspots; Enable fast fan-out timeline writes

Key flows

Show all (3)
  • Media Storage flow: clients request a pre-signed upload URL from the Media Upload Service, upload media directly to Amazon S3, asynchronous workers generate optimized renditions back into S3, and CloudFront serves the media globally with edge caching.
  • Metadata flow: post/user/social graph data is written via the Post & Metadata Service into sharded PostgreSQL for canonical transactional records and Cassandra for denormalized high-scale read/write views; PostCreated is emitted via an outbox to the message queue.
  • Fan-out on Write & feed serving: PostCreated events are consumed by fan-out workers, which fetch follower sets and append post_id pointers into followers’ Redis Cluster timelines; the Feed Generation Service reads Redis timelines, hydrates post metadata, and returns low-latency feeds.