Cloud Schema Registry Architecture

=============================================================================

Strategic architecture for a Central Schema Registry Service to enable

schema sharing across teams and organizations at scale (50+ engineers).

Status: Architecture Planning Phase

Target: Future v2.0+ feature

=============================================================================

Problem Statement

As FakeDataDSL adoption scales to 50+ engineers:

  • Git-based schema sharing becomes cumbersome
  • Schema versioning conflicts increase
  • Cross-team schema discovery is difficult
  • Contract testing requires manual coordination

Solution: Central Schema Registry

A centralized service for publishing, versioning, and discovering schemas across teams and organizations.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    Schema Registry Service                   │
│  (REST API + gRPC + WebSocket for real-time updates)        │
├─────────────────────────────────────────────────────────────┤
│  Features:                                                   │
│  - Schema publishing (User v1.0, Order v2.3)                │
│  - Version management (semantic versioning)                  │
│  - Schema discovery (search, browse, tags)                    │
│  - Contract testing (Frontend ↔ Backend validation)          │
│  - Access control (team-based permissions)                   │
└─────────────────────────────────────────────────────────────┘
                              │
                              │ HTTP/gRPC
                              │
        ┌─────────────────────┴─────────────────────┐
        │                                             │
┌───────▼────────┐                          ┌────────▼───────┐
│  Backend Team  │                          │  Frontend Team │
│  Publishes:    │                          │  Consumes:     │
│  User v1.0     │                          │  User v1.0     │
│  Order v2.3    │                          │  (for mocks)   │
└────────────────┘                          └────────────────┘

Core Features

1. Schema Publishing

# Backend team publishes schema
FakeDataDSL::Registry::Cloud.publish(
  schema: user_schema,
  version: "1.0.0",
  team: "backend",
  tags: ["user", "authentication"],
  description: "User authentication schema"
)

2. Schema Discovery

# Frontend team discovers schemas
schemas = FakeDataDSL::Registry::Cloud.search(
  tags: ["user"],
  team: "backend",
  version: ">=1.0.0"
)

# Load schema for mocking
user_schema = FakeDataDSL::Registry::Cloud.fetch("User", version: "1.0.0")

3. Ref() with Remote Lookup

# Ref() automatically resolves from registry if not found locally
# user.dsl
Order:
  user_id: Ref(User.id)  # Looks up User schema from registry if not local

4. Contract Testing

# Frontend generates mocks using exact backend schema
backend_schema = FakeDataDSL::Registry::Cloud.fetch("User", team: "backend", version: "1.0.0")
mock_data = backend_schema.generate_many(100)

# Validate frontend handles backend schema correctly
expect(frontend_component).to_handle_schema(backend_schema)

API Design

REST API

POST   /api/v1/schemas                    # Publish schema
GET    /api/v1/schemas/:name              # Get schema
GET    /api/v1/schemas/:name/versions     # List versions
GET    /api/v1/schemas?team=backend       # Search schemas
PUT    /api/v1/schemas/:name/:version     # Update schema
DELETE /api/v1/schemas/:name/:version     # Deprecate schema

gRPC API (for high-performance clients)

service SchemaRegistry {
  rpc PublishSchema(PublishRequest) returns (PublishResponse);
  rpc GetSchema(GetSchemaRequest) returns (Schema);
  rpc SearchSchemas(SearchRequest) returns (SearchResponse);
  rpc WatchSchema(WatchRequest) returns (stream SchemaUpdate);
}

Implementation Phases

Phase 1: Local Registry Cache (MVP)

  • Client-side caching of fetched schemas
  • HTTP client for schema fetching
  • Fallback to local schemas if registry unavailable
# lib/fake_data_dsl/registry/cloud.rb
module FakeDataDSL
  module Registry
    class Cloud
      def self.fetch(name, version: nil, team: nil)
        # Try cache first
        # Then HTTP fetch
        # Fallback to local registry
      end
    end
  end
end

Phase 2: Full Registry Service

  • REST API server (Ruby/Rails or Go)
  • Schema storage (PostgreSQL with JSONB)
  • Version management
  • Search and discovery
  • Authentication/authorization

Phase 3: Advanced Features

  • Real-time schema updates (WebSocket)
  • Schema diff visualization
  • Contract testing framework
  • Schema analytics (usage, versions, teams)

Data Model

CREATE TABLE schemas (
  id UUID PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  version VARCHAR(50) NOT NULL,
  team VARCHAR(100) NOT NULL,
  dsl_content TEXT NOT NULL,
  compiled_json JSONB NOT NULL,
  tags TEXT[],
  description TEXT,
  published_at TIMESTAMP NOT NULL,
  deprecated_at TIMESTAMP,
  UNIQUE(name, version, team)
);

CREATE INDEX idx_schemas_name ON schemas(name);
CREATE INDEX idx_schemas_team ON schemas(team);
CREATE INDEX idx_schemas_tags ON schemas USING GIN(tags);

Client Integration

Configuration

FakeDataDSL.configure do |config|
  config.registry_url = "https://registry.company.com"
  config.registry_token = ENV["REGISTRY_TOKEN"]
  config.registry_team = "backend"
  config.registry_cache_ttl = 3600 # 1 hour
end

Automatic Resolution

# Ref() automatically resolves from registry
registry = FakeDataDSL::Registry.new
registry.load_file("user.dsl")  # Local schema

# Order.dsl references User - automatically fetched from registry
registry.load_file("order.dsl")  # Ref(User.id) resolves from registry

Security & Access Control

  • Authentication: API tokens per team
  • Authorization: Team-based access control
  • Schema Privacy: Private schemas (team-only) vs public schemas
  • Rate Limiting: Prevent abuse

Deployment Options

Option 1: SaaS (Hosted Service)

  • FakeDataDSL team hosts registry
  • Free tier + paid tiers
  • Multi-tenant architecture

Option 2: Self-Hosted

  • Docker container
  • Kubernetes deployment
  • On-premises installation

Option 3: Hybrid

  • Local registry for internal schemas
  • Cloud registry for public/shared schemas

Success Metrics

  • Adoption: 50+ engineers using registry
  • Performance: <100ms schema fetch latency
  • Reliability: 99.9% uptime
  • Usage: 1000+ schema versions published

Migration Path

  1. Phase 1: Add client library (backward compatible)
  2. Phase 2: Deploy registry service (optional)
  3. Phase 3: Teams migrate schemas to registry
  4. Phase 4: Registry becomes primary source

Open Questions

  1. Schema Format: Store DSL or compiled JSON?

    • Recommendation: Both (DSL for human editing, JSON for fast loading)

  2. Versioning Strategy: Semantic versioning or timestamp-based?

    • Recommendation: Semantic versioning (1.0.0, 1.1.0, 2.0.0)

  3. Conflict Resolution: What if two teams publish same schema name?

    • Recommendation: Namespace by team (backend/User vs frontend/User)

  4. Backward Compatibility: How to handle breaking changes?

    • Recommendation: Major version bumps (1.0.0 → 2.0.0) + deprecation warnings

References

  • JSON Schema Registry patterns
  • Docker Registry architecture
  • NPM registry design
  • Maven Central Repository