Module: Linguist::BlobHelper

Included in:
Blob, FileBlob, LazyBlob
Defined in:
lib/linguist/blob_helper.rb

Overview

DEPRECATED Avoid mixing into Blob classes. Prefer functional interfaces like Linguist.detect over Blob#language. Functions are much easier to cache and compose.

Avoid adding additional bloat to this module.

BlobHelper is a mixin for Blobish classes that respond to "name", "data" and "size" such as Grit::Blob.

Constant Summary collapse

MEGABYTE = 
1024 * 1024
VendoredRegexp = 
Regexp.new(vendored_paths.join('|'))
DocumentationRegexp = 
Regexp.new(documentation_paths.join('|'))
DETECTABLE_TYPES = 
[:programming, :markup].freeze

Instance Method Summary collapse

Instance Method Details

#_mime_type ⇒ Object

Internal: Lookup mime type for filename.

Returns a MIME::Type



32
33
34
35
36
37
38
 
# File 'lib/linguist/blob_helper.rb', line 32

def _mime_type
  if defined? @_mime_type
    @_mime_type
  else
    @_mime_type = MiniMime.lookup_by_filename(name.to_s)
  end
end

#binary? ⇒ Boolean

Public: Is the blob binary?

Return true or false

Returns:

  • (Boolean) 


143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
# File 'lib/linguist/blob_helper.rb', line 143

def binary?
  # Large blobs aren't even loaded into memory
  if data.nil?
    true

  # Treat blank files as text
  elsif data == ""
    false

  # Charlock doesn't know what to think
  elsif encoding.nil?
    true

  # If Charlock says its binary
  else
    detect_encoding[:type] == :binary
  end
end

#binary_mime_type? ⇒ Boolean

Internal: Is the blob binary according to its mime type

Return true or false

Returns:

  • (Boolean) 


55
56
57
 
# File 'lib/linguist/blob_helper.rb', line 55

def binary_mime_type?
  _mime_type ? _mime_type.binary? : false
end

#content_type ⇒ Object

Public: Get the Content-Type header value

This value is used when serving raw blobs.

Examples

# => 'text/plain; charset=utf-8'
# => 'application/octet-stream'

Returns a content type String.



78
79
80
81
 
# File 'lib/linguist/blob_helper.rb', line 78

def content_type
  @content_type ||= (binary_mime_type? || binary?) ? mime_type :
    (encoding ? "text/plain; charset=#{encoding.downcase}" : "text/plain")
end

#csv? ⇒ Boolean

Public: Is this blob a CSV file?

Return true or false

Returns:

  • (Boolean) 


193
194
195
 
# File 'lib/linguist/blob_helper.rb', line 193

def csv?
  text? && extname.downcase == '.csv'
end

#detect_encoding ⇒ Object

Try to guess the encoding

Returns: a Hash, with :encoding, :confidence, :type this will return nil if an error occurred during detection or no valid encoding could be found



118
119
120
 
# File 'lib/linguist/blob_helper.rb', line 118

def detect_encoding
  @detect_encoding ||= CharlockHolmes::EncodingDetector.new.detect(data) if data
end

#disposition ⇒ Object

Public: Get the Content-Disposition header value

This value is used when serving raw blobs.

# => "attachment; filename=file.tar"
# => "inline"

Returns a content disposition String.



91
92
93
94
95
96
97
98
99
 
# File 'lib/linguist/blob_helper.rb', line 91

def disposition
  if text? || image?
    'inline'
  elsif name.nil?
    "attachment"
  else
    "attachment; filename=#{CGI.escape(name)}"
  end
end

#documentation? ⇒ Boolean

Public: Is the blob in a documentation directory?

Documentation files are ignored by language statistics.

See "documentation.yml" for a list of documentation conventions that match this pattern.

Return true or false

Returns:

  • (Boolean) 


263
264
265
 
# File 'lib/linguist/blob_helper.rb', line 263

def documentation?
  path =~ DocumentationRegexp ? true : false
end

#empty? ⇒ Boolean

Public: Is the blob empty?

Return true or false

Returns:

  • (Boolean) 


165
166
167
 
# File 'lib/linguist/blob_helper.rb', line 165

def empty?
  data.nil? || data == ""
end

#encoded_newlines_re ⇒ Object 



305
306
307
308
309
 
# File 'lib/linguist/blob_helper.rb', line 305

def encoded_newlines_re
  @encoded_newlines_re ||= Regexp.union(["\r\n", "\r", "\n"].
                                          map { |nl| nl.encode(ruby_encoding, "ASCII-8BIT").force_encoding(data.encoding) })

end

#encoding ⇒ Object 



101
102
103
104
105
 
# File 'lib/linguist/blob_helper.rb', line 101

def encoding
  if hash = detect_encoding
    hash[:encoding]
  end
end

#extname ⇒ Object

Public: Get the extname of the path

Examples

blob(name='foo.rb').extname
# => '.rb'

Returns a String



25
26
27
 
# File 'lib/linguist/blob_helper.rb', line 25

def extname
  File.extname(name.to_s)
end

#first_lines(n) ⇒ Object 



311
312
313
314
315
316
317
318
319
320
321
 
# File 'lib/linguist/blob_helper.rb', line 311

def first_lines(n)
  return lines[0...n] if defined? @lines
  return [] unless viewable? && data

  i, c = 0, 0
  while c < n && j = data.index(encoded_newlines_re, i)
    i = j + $&.length
    c += 1
  end
  data[0...i].split(encoded_newlines_re, -1)
end

#generated? ⇒ Boolean

Public: Is the blob a generated file?

Generated source code is suppressed in diffs and is ignored by language statistics.

May load Blob#data

Return true or false

Returns:

  • (Boolean) 


376
377
378
 
# File 'lib/linguist/blob_helper.rb', line 376

def generated?
  @_generated ||= Generated.generated?(path, lambda { data })
end

#high_ratio_of_long_lines? ⇒ Boolean

Internal: Does the blob have a ratio of long lines?

Return true or false

Returns:

  • (Boolean) 


223
224
225
226
 
# File 'lib/linguist/blob_helper.rb', line 223

def high_ratio_of_long_lines?
  return false if loc == 0
  size / loc > 5000
end

#image? ⇒ Boolean

Public: Is the blob a supported image format?

Return true or false

Returns:

  • (Boolean) 


179
180
181
 
# File 'lib/linguist/blob_helper.rb', line 179

def image?
  ['.png', '.jpg', '.jpeg', '.gif'].include?(extname.downcase)
end

#include_in_language_stats? ⇒ Boolean

Internal: Should this blob be included in repository language statistics?

Returns:

  • (Boolean) 


397
398
399
400
401
402
403
404
405
406
 
# File 'lib/linguist/blob_helper.rb', line 397

def include_in_language_stats?
  !vendored? &&
  !documentation? &&
  !generated? &&
  !(lfs_tracked? && lfs_pointer?) &&
  language && ( defined?(detectable?) && !detectable?.nil? ?
    detectable? :
    DETECTABLE_TYPES.include?(language.type)
  )
end

#language ⇒ Object

Public: Detects the Language of the blob.

May load Blob#data

Returns a Language or nil if none is detected



385
386
387
 
# File 'lib/linguist/blob_helper.rb', line 385

def language
  @language ||= Linguist.detect(self)
end

#large? ⇒ Boolean

Public: Is the blob too big to load?

Return true or false

Returns:

  • (Boolean) 


209
210
211
 
# File 'lib/linguist/blob_helper.rb', line 209

def large?
  size.to_i > MEGABYTE
end

#last_lines(n) ⇒ Object 



323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
 
# File 'lib/linguist/blob_helper.rb', line 323

def last_lines(n)
  if defined? @lines
    if n >= @lines.length
      @lines
    else
      lines[-n..-1]
    end
  end
  return [] unless viewable? && data

  no_eol = true
  i, c = data.length, 0
  k = i
  while c < n && j = data.rindex(encoded_newlines_re, i - 1)
    if c == 0 && j + $&.length == i
      no_eol = false
      n += 1
    end
    i = j
    k = j + $&.length
    c += 1
  end
  r = data[k..-1].split(encoded_newlines_re, -1)
  r.pop if !no_eol
  r
end

#lfs_pointer? ⇒ Boolean

Public: Is the blob content a Git LFS pointer file?

Return true or false

Returns:

  • (Boolean) 


125
126
127
128
129
130
131
 
# File 'lib/linguist/blob_helper.rb', line 125

def lfs_pointer?
  return false unless data

  data.start_with?("version https://git-lfs.github.com/spec/v1\n") &&
    data.include?("\noid sha256:") &&
    data.match?(/\nsize \d+\n?\z/)
end

#lfs_tracked? ⇒ Boolean

Public: Is the blob path tracked by Git LFS?

Return true or false

Returns:

  • (Boolean) 


136
137
138
 
# File 'lib/linguist/blob_helper.rb', line 136

def lfs_tracked?
  false
end

#likely_binary? ⇒ Boolean

Internal: Is the blob binary according to its mime type, overriding it if we have better data from the languages.yml database.

Return true or false

Returns:

  • (Boolean) 


64
65
66
 
# File 'lib/linguist/blob_helper.rb', line 64

def likely_binary?
  binary_mime_type? && !Language.find_by_filename(name)
end

#lines ⇒ Object

Public: Get each line of data

Requires Blob#data

Returns an Array of lines



272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
 
# File 'lib/linguist/blob_helper.rb', line 272

def lines
  @lines ||=
    if viewable? && data
      # `data` is usually encoded as ASCII-8BIT even when the content has
      # been detected as a different encoding. However, we are not allowed
      # to change the encoding of `data` because we've made the implicit
      # guarantee that each entry in `lines` is encoded the same way as
      # `data`.
      #
      # Instead, we re-encode each possible newline sequence as the
      # detected encoding, then force them back to the encoding of `data`
      # (usually a binary encoding like ASCII-8BIT). This means that the
      # byte sequence will match how newlines are likely encoded in the
      # file, but we don't have to change the encoding of `data` as far as
      # Ruby is concerned. This allows us to correctly parse out each line
      # without changing the encoding of `data`, and
      # also--importantly--without having to duplicate many (potentially
      # large) strings.
      begin
        # `data` is split after having its last `\n` removed by
        # chomp (if any). This prevents the creation of an empty
        # element after the final `\n` character on POSIX files.
        data.chomp.split(encoded_newlines_re, -1)
      rescue Encoding::ConverterNotFoundError
        # The data is not splittable in the detected encoding.  Assume it's
        # one big line.
        [data]
      end
    else
      []
    end
end

#loc ⇒ Object

Public: Get number of lines of code

Requires Blob#data

Returns Integer



355
356
357
 
# File 'lib/linguist/blob_helper.rb', line 355

def loc
  lines.size
end

#mime_type ⇒ Object

Public: Get the actual blob mime type

Examples

# => 'text/plain'
# => 'text/html'

Returns a mime type String.



48
49
50
 
# File 'lib/linguist/blob_helper.rb', line 48

def mime_type
  _mime_type ? _mime_type.content_type : 'text/plain'
end

#pdf? ⇒ Boolean

Public: Is the blob a PDF?

Return true or false

Returns:

  • (Boolean) 


200
201
202
 
# File 'lib/linguist/blob_helper.rb', line 200

def pdf?
  extname.downcase == '.pdf'
end

#ruby_encoding ⇒ Object 



107
108
109
110
111
 
# File 'lib/linguist/blob_helper.rb', line 107

def ruby_encoding
  if hash = detect_encoding
    hash[:ruby_encoding]
  end
end

#safe_to_colorize? ⇒ Boolean

Public: Is the blob safe to colorize?

Return true or false

Returns:

  • (Boolean) 


216
217
218
 
# File 'lib/linguist/blob_helper.rb', line 216

def safe_to_colorize?
  !large? && text? && !high_ratio_of_long_lines?
end

#sloc ⇒ Object

Public: Get number of source lines of code

Requires Blob#data

Returns Integer



364
365
366
 
# File 'lib/linguist/blob_helper.rb', line 364

def sloc
  lines.grep(/\S/).size
end

#solid? ⇒ Boolean

Public: Is the blob a supported 3D model format?

Return true or false

Returns:

  • (Boolean) 


186
187
188
 
# File 'lib/linguist/blob_helper.rb', line 186

def solid?
  extname.downcase == '.stl'
end

#text? ⇒ Boolean

Public: Is the blob text?

Return true or false

Returns:

  • (Boolean) 


172
173
174
 
# File 'lib/linguist/blob_helper.rb', line 172

def text?
  !binary?
end

#tm_scope ⇒ Object

Internal: Get the TextMate compatible scope for the blob



390
391
392
 
# File 'lib/linguist/blob_helper.rb', line 390

def tm_scope
  language && language.tm_scope
end

#vendored? ⇒ Boolean

Public: Is the blob in a vendored directory?

Vendored files are ignored by language statistics.

See "vendor.yml" for a list of vendored conventions that match this pattern.

Return true or false

Returns:

  • (Boolean) 


248
249
250
 
# File 'lib/linguist/blob_helper.rb', line 248

def vendored?
  path =~ VendoredRegexp ? true : false
end

#viewable? ⇒ Boolean

Public: Is the blob viewable?

Non-viewable blobs will just show a "View Raw" link

Return true or false

Returns:

  • (Boolean) 


233
234
235
 
# File 'lib/linguist/blob_helper.rb', line 233

def viewable?
  !large? && text?
end