Class: Wp2txt::CategoryFetcher

Inherits:
Object
  • Object
show all
Defined in:
lib/wp2txt/multistream.rb

Overview

Fetches category members from Wikipedia API Uses SQLite-based CategoryCache for efficient repeated access

Constant Summary collapse

API_ENDPOINT = 
"https://%s.wikipedia.org/w/api.php"
MAX_LIMIT = 
500
RATE_LIMIT_DELAY = 
0.1

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(lang, category, max_depth: 0, cache_expiry_days: nil, cache_dir: nil) ⇒ CategoryFetcher

Returns a new instance of CategoryFetcher.



1167
1168
1169
1170
1171
1172
1173
1174
1175
 
# File 'lib/wp2txt/multistream.rb', line 1167

def initialize(lang, category, max_depth: 0, cache_expiry_days: nil, cache_dir: nil)
  @lang = lang.to_s
  @category = normalize_category_name(category)
  @max_depth = max_depth
  @cache_expiry_days = cache_expiry_days || Wp2txt::DEFAULT_CATEGORY_CACHE_EXPIRY_DAYS
  @cache_dir = cache_dir
  @cache = nil
  @visited_categories = Set.new
end

Instance Attribute Details

#cache_expiry_days ⇒ Object (readonly)

Returns the value of attribute cache_expiry_days.



1165
1166
1167
 
# File 'lib/wp2txt/multistream.rb', line 1165

def cache_expiry_days
  @cache_expiry_days
end

#category ⇒ Object (readonly)

Returns the value of attribute category.



1165
1166
1167
 
# File 'lib/wp2txt/multistream.rb', line 1165

def category
  @category
end

#lang ⇒ Object (readonly)

Returns the value of attribute lang.



1165
1166
1167
 
# File 'lib/wp2txt/multistream.rb', line 1165

def lang
  @lang
end

#max_depth ⇒ Object (readonly)

Returns the value of attribute max_depth.



1165
1166
1167
 
# File 'lib/wp2txt/multistream.rb', line 1165

def max_depth
  @max_depth
end

Instance Method Details

#cache ⇒ Object

Get the category cache instance Creates one if caching is enabled but cache not yet initialized



1186
1187
1188
1189
1190
1191
 
# File 'lib/wp2txt/multistream.rb', line 1186

def cache
  return @cache if @cache
  return nil unless @cache_dir

  @cache = CategoryCache.new(@lang, cache_dir: @cache_dir, expiry_days: @cache_expiry_days)
end

#enable_cache(cache_dir) ⇒ Object

Enable caching of category member lists

Parameters:

  • cache_dir (String) —

    Directory for cache files



1179
1180
1181
1182
 
# File 'lib/wp2txt/multistream.rb', line 1179

def enable_cache(cache_dir)
  @cache_dir = cache_dir
  @cache = CategoryCache.new(@lang, cache_dir: cache_dir, expiry_days: @cache_expiry_days)
end

#fetch_articles ⇒ Object

Fetch all article titles in the category (and subcategories if depth > 0)



1213
1214
1215
1216
1217
1218
 
# File 'lib/wp2txt/multistream.rb', line 1213

def fetch_articles
  @visited_categories = Set.new
  @articles = []
  fetch_category_members(@category, 0)
  @articles.uniq
end

#fetch_preview ⇒ Object

Preview mode - returns statistics without full article list



1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
 
# File 'lib/wp2txt/multistream.rb', line 1194

def fetch_preview
  @visited_categories = Set.new
  subcategories = []
  total_articles = 0

  fetch_category_stats(@category, 0, subcategories)

  total_articles = subcategories.sum { |s| s[:article_count] }

  {
    category: @category,
    depth: @max_depth,
    subcategories: subcategories,
    total_subcategories: subcategories.size - 1,
    total_articles: total_articles
  }
end