WebFiltering — Ruby Client for Web Filtering Database API

Overview

The webfiltering gem provides a lightweight Ruby client for querying and downloading domain classification data purpose-built for network-level content filtering. It connects to a DNS-level web filtering database containing over 100 million domains organized into 59 content categories, enabling firewalls, secure web gateways, DNS resolvers, and proxy servers to enforce acceptable-use policies without external API calls at query time.

Content filtering at the network layer is a core requirement for organizations governed by regulations such as the Children's Internet Protection Act (CIPA), which mandates that schools and libraries receiving federal E-rate funding block access to obscene or harmful material. Enterprise IT departments enforce similar policies to reduce exposure to phishing, malware distribution, and productivity drains like gambling or social media during working hours. The approach described by NIST's Cybersecurity Framework recommends layered defenses in which URL filtering serves as one of the earliest lines of protection, catching threats before they reach endpoint software.

This gem abstracts the HTTP transport, authentication, and response parsing so that Ruby developers can integrate domain lookups and bulk downloads into security appliances, reporting dashboards, or DevOps automation scripts with minimal boilerplate.

Installation

Add the gem to your Gemfile:

gem 'webfiltering'

Then run:

bundle install

Or install it directly:

gem install webfiltering

Quick Start

require 'webfiltering'

client = WebFiltering::Client.new(api_key: 'YOUR_API_KEY')

# Classify a single domain
result = client.classify('example.com')
puts result.category       # => "Technology & Computing"
puts result.confidence      # => 0.94
puts result.subcategories   # => ["Software", "Developer Tools"]

# Batch classification
domains = ['reddit.com', 'espn.com', 'malware-site.xyz']
results = client.classify_batch(domains)
results.each do |r|
  puts "#{r.domain}: #{r.category} (#{r.threat_level})"
end

Usage

Real-Time Domain Lookup

The simplest use case is classifying a domain on the fly. The API returns a primary category drawn from a 59-category taxonomy designed specifically for content filtering, along with a confidence score and optional subcategories.

client = WebFiltering::Client.new(api_key: ENV['WF_API_KEY'])

# IAB content taxonomy classification
result = client.classify('youtube.com', taxonomy: 'web_filtering')
puts result.to_h
# {
#   domain: "youtube.com",
#   category: "Streaming Media",
#   confidence: 0.97,
#   threat_level: "none",
#   subcategories: ["Video Sharing"]
# }

Bulk Database Download

For offline or on-premise deployments where latency matters, the gem supports downloading the full categorized domain dataset. This is the recommended approach for DNS filtering appliances that need sub-millisecond lookup times, as described in best practices published by the Internet Engineering Task Force (IETF).

client = WebFiltering::Client.new(api_key: ENV['WF_API_KEY'])

# Download full dataset as CSV
client.download_database(
  format: 'csv',
  output: '/var/lib/webfilter/domains.csv',
  categories: :all
)

# Download only security-relevant categories
client.download_database(
  format: 'json',
  output: '/var/lib/webfilter/threats.json',
  categories: ['Malware', 'Phishing', 'Command and Control', 'Spam']
)

Category Lookup

Query the full taxonomy to understand available categories and their hierarchical relationships. This is useful for building policy configuration interfaces or generating compliance documentation.

client = WebFiltering::Client.new(api_key: ENV['WF_API_KEY'])

# List all 59 categories
categories = client.list_categories
categories.each { |c| puts "#{c.id}: #{c.name} (#{c.domain_count} domains)" }

# Check if a domain is in a specific category
blocked = client.in_category?('facebook.com', 'Social Media')
puts blocked  # => true

Streaming Updates

Keep your local database current by polling for incremental updates. New domains and reclassifications are published continuously.

client = WebFiltering::Client.new(api_key: ENV['WF_API_KEY'])

# Get changes since a timestamp
updates = client.updates_since('2025-12-01T00:00:00Z')
puts "#{updates.added.count} new domains"
puts "#{updates.changed.count} reclassified domains"
puts "#{updates.removed.count} delisted domains"

Features

• 59 Content Categories — Purpose-built taxonomy covering adult content, gambling, malware, phishing, social media, streaming, weapons, drugs, and other policy-relevant verticals. Categories are aligned with the filtering requirements defined under CIPA and common enterprise acceptable-use policies.

• 100 Million Domains — One of the largest commercially available domain classification datasets, refreshed regularly with newly registered domains and updated threat intelligence.

• Threat Detection — Domains associated with malware distribution, phishing kits, command-and-control infrastructure, and social engineering are flagged with a separate threat level indicator, following risk assessment guidelines from the OWASP Foundation.

• Offline Mode — Download the full dataset for air-gapped or latency-sensitive deployments. The dataset can be loaded into any SQL or NoSQL store, or consumed as flat files by DNS filtering middleware.

• Incremental Sync — After the initial download, retrieve only deltas to keep local copies current without re-downloading the entire database.

• Multiple Output Formats — CSV, JSON, and SQL dump formats supported out of the box.

Use Cases

K-12 and Library Content Filtering

Schools and public libraries subject to CIPA must deploy technology protection measures that block access to visual depictions that are obscene or harmful to minors. The web filtering database provides a ready-made classification layer that integrates with open-source DNS forwarders like Pi-hole or commercial appliances from vendors such as Cisco Umbrella. Compliance auditors can verify coverage by cross-referencing the 59-category taxonomy against CIPA requirements.

Enterprise Network Security

Corporate IT teams deploy URL filtering as part of a defense-in-depth strategy recommended by frameworks like NIST SP 800-53. By blocking known malware distribution and phishing domains at the DNS layer, organizations prevent threats from reaching endpoints altogether. The database also supports productivity policies by categorizing social media, gaming, and streaming sites.

ISP Parental Controls

Internet service providers offer optional parental-control features to subscribers. A pre-categorized domain database allows the ISP to implement filtering at the resolver level with negligible latency impact, avoiding the need to inspect HTTP payloads or break TLS connections. The W3C Web Annotation standards inform metadata structures used in the classification pipeline.

Managed Security Service Providers (MSSPs)

MSSPs building multi-tenant filtering platforms benefit from a bulk-downloadable dataset that can be partitioned per customer with custom policy overlays. The gem's batch classification API supports the throughput required for managing thousands of client networks simultaneously.

Cybersecurity Threat Intelligence

Security operations centers integrate URL filtering feeds into SIEM platforms and threat intelligence pipelines. When an employee clicks a link in a phishing email, DNS-level filtering can block the connection before the browser even begins the TLS handshake. The database flags domains associated with malware distribution, ransomware command-and-control servers, cryptojacking scripts, and credential-harvesting kits. These threat categories are maintained using automated scanning combined with manual review, following the threat intelligence lifecycle described by NIST SP 800-150. By correlating DNS query logs with the category database, SOC analysts can identify patterns such as a workstation repeatedly attempting to reach gambling or adult sites — often indicators of compromised machines controlled by browser-hijacking malware.

Regulatory Compliance Reporting

Organizations in regulated industries need to demonstrate that their content filtering controls are effective. The gem supports generating compliance reports by querying classification history and policy enforcement statistics. School districts can produce CIPA compliance evidence showing that all 59 categories of restricted content are actively filtered, while financial services firms can document that employees cannot access known phishing or social engineering sites. The structured JSON responses make it straightforward to build automated compliance dashboards that satisfy auditor requirements.

Configuration

WebFiltering.configure do |config|
  config.api_key     = ENV['WF_API_KEY']
  config.base_url    = 'https://www.webfilteringdatabase.com/api'
  config.timeout     = 30
  config.retries     = 3
  config.format      = :json
end

Error Handling

begin
  result = client.classify('example.com')
rescue WebFiltering::AuthenticationError => e
  puts "Invalid API key: #{e.message}"
rescue WebFiltering::RateLimitError => e
  puts "Rate limited, retry after #{e.retry_after} seconds"
rescue WebFiltering::ApiError => e
  puts "API error: #{e.code} - #{e.message}"
end

Related Services

Our platform extends beyond web filtering into several complementary areas of content intelligence and data enrichment:

• Website Categorization API — Classify any URL or text into IAB content taxonomy categories with machine-learning-powered real-time classification across 700+ categories.

• Product Categorization AI — Categorize product titles and descriptions into Google Shopping, Shopify, and Amazon taxonomies for ecommerce applications.

• Categorized Domain Dataset — A bulk-downloadable database of domains classified under IAB taxonomy, suitable for ad-tech contextual targeting, brand safety, and large-scale analytics.

• Content Moderation API — Detect hate speech, profanity, spam, and toxic content in user-generated text with real-time AI moderation.

• Anonymization API — Automatically anonymize faces and sensitive objects in images and video streams for GDPR-compliant data processing.

• Redaction API — Remove personally identifiable information from documents at scale, supporting compliance with GDPR, CCPA, and HIPAA.

References

• Children's Internet Protection Act (CIPA) — Federal requirements for content filtering in schools and libraries.
• NIST Cybersecurity Framework — Risk management guidelines including network-layer URL filtering.
• OWASP Web Security Testing Guide — Security testing methodologies relevant to threat detection.
• IETF DNS-Based Filtering — Standards for DNS infrastructure and filtering mechanisms.
• IAB Tech Lab Content Taxonomy — Industry-standard content categorization taxonomy for digital advertising.
• W3C Web Standards — Web annotation and metadata standards informing classification pipelines.

License

MIT License. See LICENSE for details.