Class: AlExtPosts::ExternalPostsGenerator

Inherits:

Jekyll::Generator

Object
Jekyll::Generator
AlExtPosts::ExternalPostsGenerator

show all

Defined in:: lib/al_ext_posts.rb

Constant Summary collapse

PAGE_EXTENSIONS = Extensions dropped from a URL segment before it is turned into a title, so .../my-post.html reads as "My Post" rather than "My Post Html".

%w[html htm xhtml shtml php asp aspx jsp cfm md markdown txt].freeze

FALLBACK_TITLE = Last-resort title for a URL with neither a usable path nor a host.

'External post'.freeze

Instance Method Summary collapse

#build_slug(source_name, url, title) ⇒ Object
Build a filesystem-safe post slug from the title, falling back to the source name + last URL segment when the title is missing, blank, or made up entirely of non-word characters.
#create_document(site, source_name, url, content, src = {}) ⇒ Object
#decode_url_segment(segment) ⇒ Object
Percent-decode a single URL segment, keeping the raw text whenever the escapes are malformed or decode to invalid bytes.
#fallback_slug(source_name, url) ⇒ Object
Fallback slug built from the source name and the last URL segment.
#fetch_content_from_url(url) ⇒ Object
#fetch_from_rss(site, src) ⇒ Object
#fetch_from_urls(site, src) ⇒ Object
#generate(site) ⇒ Object
#humanize_segment(segment) ⇒ Object
Split a URL segment on its separators and capitalize each word, leaving words that are already cased alone: "google-gemini-io-2024" becomes "Google Gemini Io 2024".
#metadata_for_post(src, post) ⇒ Object
#metadata_value(post, key) ⇒ Object
#parse_published_date(published_date) ⇒ Object
#process_entries(site, src, entries) ⇒ Object
#resolve_title(title, url) ⇒ Object
Title to publish for an ingested item.
#slugify(value) ⇒ Object
Drop every character that is not a word character, space, or hyphen in a single pass, then translate the remaining spaces to hyphens.
#strip_page_extension(segment) ⇒ Object
Drop a trailing web-page extension, keeping the segment untouched when the extension is unknown (a version number, say) or is all there is.
#title_from_url(url) ⇒ Object
Turn the last meaningful path segment of a URL into a human-readable title: "https://blog.google/technology/ai/gemini-update-2024/" becomes "Gemini Update 2024".
#url_host(url) ⇒ Object
Host of a URL, without a leading "www.".
#url_path_segments(url) ⇒ Object
Path segments of a URL, percent-decoded and stripped of blanks.

Instance Method Details

#build_slug(source_name, url, title) ⇒ `Object`

Build a filesystem-safe post slug from the title, falling back to the source name + last URL segment when the title is missing, blank, or made up entirely of non-word characters. Guards against a nil title (e.g. an RSS entry or fetched page with no ), which previously raised a NoMethodError and aborted the whole build. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 91 92 93 94 95 96</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 91 def build_slug(source_name, url, title) return fallback_slug(source_name, url) unless title.to_s.match?(/\w/) slug = slugify(title) slug.empty? ? fallback_slug(source_name, url) : slug end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="create_document-instance_method"> #create_document(site, source_name, url, content, src = {}) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 57 def create_document(site, source_name, url, content, src = {}) # The slug is still derived from the raw title so existing post URLs keep # their source-name fallback; only the published title is filled in. slug = build_slug(source_name, url, content[:title]) title = resolve_title(content[:title], url) path = site.in_source_dir("_posts/#{slug}.md") doc = Jekyll::Document.new( path, { :site => site, :collection => site.collections['posts'] } ) doc.data['external_source'] = source_name doc.data['title'] = title doc.data['feed_content'] = content[:content] doc.data['description'] = content[:summary] doc.data['date'] = content[:published] doc.data['redirect'] = url # Apply default categories and tags from source configuration if src['categories'] && src['categories'].is_a?(Array) && !src['categories'].empty? doc.data['categories'] = src['categories'] end if src['tags'] && src['tags'].is_a?(Array) && !src['tags'].empty? doc.data['tags'] = src['tags'] end doc.content = content[:content] site.collections['posts'].docs << doc end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="decode_url_segment-instance_method"> #decode_url_segment(segment) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Percent-decode a single URL segment, keeping the raw text whenever the escapes are malformed or decode to invalid bytes. Scrubbed either way so the callers below can match against it without raising. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 173 174 175 176 177 178</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 173 def decode_url_segment(segment) decoded = URI.decode_www_form_component(segment) (decoded.valid_encoding? ? decoded : segment).scrub('') rescue ArgumentError segment.scrub('') end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="fallback_slug-instance_method"> #fallback_slug(source_name, url) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Fallback slug built from the source name and the last URL segment. Only computed on the fallback path so the common (titled) case does no extra string work. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 101 102 103</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 101 def fallback_slug(source_name, url) "#{slugify(source_name)}-#{url.split('/').last}" end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="fetch_content_from_url-instance_method"> #fetch_content_from_url(url) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 234 def fetch_content_from_url(url) html = HTTParty.get(url).body parsed_html = Nokogiri::HTML(html) title = parsed_html.at('head title')&.text&.strip || '' description = parsed_html.at('head meta[name="description"]')&.attr('content') description ||= parsed_html.at('head meta[name="og:description"]')&.attr('content') description ||= parsed_html.at('head meta[property="og:description"]')&.attr('content') body_content = parsed_html.search('p').map { |e| e.text } body_content = body_content.join() || '' { title: title, content: body_content, summary: description # Note: The published date is now added in the fetch_from_urls method. } end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="fetch_from_rss-instance_method"> #fetch_from_rss(site, src) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 33 34 35 36 37 38 39 40 41 42 43</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 33 def fetch_from_rss(site, src) xml = HTTParty.get(src['rss_url']).body return if xml.nil? begin feed = Feedjira.parse(xml) rescue StandardError => e puts "Error parsing RSS feed from #{src['rss_url']} - #{e.message}" return end process_entries(site, src, feed.entries) end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="fetch_from_urls-instance_method"> #fetch_from_urls(site, src) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 197 198 199 200 201 202 203 204</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 197 def fetch_from_urls(site, src) src['posts'].each do |post| puts "...fetching #{post['url']}" content = fetch_content_from_url(post['url']) content[:published] = parse_published_date(post['published_date']) create_document(site, src['name'], post['url'], content, metadata_for_post(src, post)) end end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="generate-instance_method"> #generate(site) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 20 21 22 23 24 25 26 27 28 29 30 31</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 20 def generate(site) if site.config['external_sources'] != nil site.config['external_sources'].each do |src| puts "Fetching external posts from #{src['name']}:" if src['rss_url'] fetch_from_rss(site, src) elsif src['posts'] fetch_from_urls(site, src) end end end end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="humanize_segment-instance_method"> #humanize_segment(segment) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Split a URL segment on its separators and capitalize each word, leaving words that are already cased alone: "google-gemini-io-2024" becomes "Google Gemini Io 2024". </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 193 194 195</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 193 def humanize_segment(segment) segment.to_s.gsub(/[^[:alnum:]]+/, ' ').split.map { |word| word.sub(/\A[[:lower:]]/, &:upcase) }.join(' ') end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="metadata_for_post-instance_method"> #metadata_for_post(src, post) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 206 207 208 209 210 211 212 213</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 206 def metadata_for_post(src, post) metadata = src.dup %w[categories tags].each do |key| value = metadata_value(post, key) metadata[key] = value if value && !(value.respond_to?(:empty?) && value.empty?) end metadata end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="metadata_value-instance_method"> #metadata_value(post, key) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 215 216 217 218 219 220 221</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 215 def metadata_value(post, key) if post.respond_to?(:key?) post[key] || post[key.to_sym] elsif post.respond_to?(key) post.public_send(key) end end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="parse_published_date-instance_method"> #parse_published_date(published_date) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 223 224 225 226 227 228 229 230 231 232</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 223 def parse_published_date(published_date) case published_date when String Time.parse(published_date).utc when Date published_date.to_time.utc else raise "Invalid date format for #{published_date}" end end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="process_entries-instance_method"> #process_entries(site, src, entries) ⇒ <tt>Object</tt> </h3><table class="source_code"> <tr> <td> <pre class="lines"> 45 46 47 48 49 50 51 52 53 54 55</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 45 def process_entries(site, src, entries) entries.each do |e| puts "...fetching #{e.url}" create_document(site, src['name'], e.url, { title: e.title, content: e.content, summary: e.summary, published: e.published }, metadata_for_post(src, e)) end end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="resolve_title-instance_method"> #resolve_title(title, url) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Title to publish for an ingested item. Fetches degrade in ways that leave no title at all (an unreachable host, a page without <title>, an RSS item with a blank one), and publishing that empty string produced a blank but clickable row in the blog index plus a stream of Jekyll "Empty <code>slug</code> generated" warnings. The source is listed in _config.yml by the user, so the entry is kept: derive a readable title from the URL and warn that the fetch degraded. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 120 121 122 123 124 125 126</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 120 def resolve_title(title, url) return title if title.to_s.match?(/[[:word:]]/) derived = title_from_url(url) Jekyll.logger.warn('ExternalPosts:', "No title found for #{url} - using #{derived.inspect} derived from the URL.") derived end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="slugify-instance_method"> #slugify(value) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Drop every character that is not a word character, space, or hyphen in a single pass, then translate the remaining spaces to hyphens. Equivalent to the previous <code>gsub(' ', '-').gsub(/[^\w-]/, '')</code> pair, but avoids the intermediate string and the second regexp scan. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 109 110 111</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 109 def slugify(value) value.to_s.downcase.strip.gsub(/[^\w -]/, '').tr(' ', '-') end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="strip_page_extension-instance_method"> #strip_page_extension(segment) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Drop a trailing web-page extension, keeping the segment untouched when the extension is unknown (a version number, say) or is all there is. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 182 183 184 185 186 187 188</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 182 def strip_page_extension(segment) extension = File.extname(segment) return segment unless <a href="#PAGE_EXTENSIONS-constant" title="AlExtPosts::ExternalPostsGenerator::PAGE_EXTENSIONS (constant)">PAGE_EXTENSIONS</a>.include?(extension.delete_prefix('.').downcase) stripped = segment.chomp(extension) stripped.empty? ? segment : stripped end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="title_from_url-instance_method"> #title_from_url(url) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Turn the last meaningful path segment of a URL into a human-readable title: "<a href="https://blog.google/technology/ai/gemini-update-2024/&quot">https://blog.google/technology/ai/gemini-update-2024/&quot</a>; becomes "Gemini Update 2024". Trailing slashes, query strings and fragments are ignored, percent-escapes are decoded, and a page extension is dropped. Segments carrying no letters on their own (ids, /2024/05/ date parts) are skipped in favour of an earlier one, and a URL left with no wordy segment at all - a bare domain, an all-numeric path - falls back to its host. Never raises and never returns an empty string. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 136 137 138 139 140 141 142 143 144 145</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 136 def title_from_url(url) segments = url_path_segments(url) segments[-1] = strip_page_extension(segments[-1]) unless segments.empty? title = humanize_segment(segments.reverse.find { |segment| segment.match?(/[[:alpha:]]/) }) return title unless title.empty? host = url_host(url) host.empty? ? <a href="#FALLBACK_TITLE-constant" title="AlExtPosts::ExternalPostsGenerator::FALLBACK_TITLE (constant)">FALLBACK_TITLE</a> : host end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="url_host-instance_method"> #url_host(url) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Host of a URL, without a leading "www.". Empty when there is none. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 160 161 162 163 164 165 166 167 168</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 160 def url_host(url) host = begin URI.parse(url.to_s).host rescue URI::Error nil end host.to_s.sub(/\Awww\./i, '') end</pre> </td> </tr> </table> </div> <div class="method_details "> <h3 class="signature " id="url_path_segments-instance_method"> #url_path_segments(url) ⇒ <tt>Object</tt> </h3><div class="docstring"> <div class="discussion"> Path segments of a URL, percent-decoded and stripped of blanks. Falls back to trimming the query/fragment by hand for inputs URI cannot parse. </div> </div> <div class="tags"> </div><table class="source_code"> <tr> <td> <pre class="lines"> 149 150 151 152 153 154 155 156 157</pre> </td> <td> <pre class="code"># File 'lib/al_ext_posts.rb', line 149 def url_path_segments(url) path = begin URI.parse(url.to_s).path.to_s rescue URI::Error url.to_s.scrub('').split('#', 2).first.to_s.split('?', 2).first.to_s end path.split('/').map { |segment| decode_url_segment(segment) }.reject { |segment| segment.strip.empty? } end</pre> </td> </tr> </table> </div> </div> </div> <div id="footer"> Generated on Wed Jul 29 03:46:53 2026 by <a href="https://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a> 0.9.45 (ruby-4.0.2). </div> </div> </body> </html>

Class: AlExtPosts::ExternalPostsGenerator

Constant Summary collapse

Instance Method Summary collapse

Instance Method Details

#build_slug(source_name, url, title) ⇒ Object

#build_slug(source_name, url, title) ⇒ `Object`