Module: Ocran::ZipWriter

Defined in:
lib/ocran/zip_writer.rb

Overview

Minimal ZIP archive appender, used to inject an application into the ZIP store of a cosmopolitan Ruby APE (see ZipPayloadBuilder).

Why not shell out to the zip command: OCRAN packages applications on Windows build hosts too, where zip generally does not exist, and even on POSIX it is not guaranteed to be installed. Why not rubyzip: OCRAN has exactly one runtime dependency (fiddle) and adding a gem just to append a few hundred stored/deflated entries is not worth it. Zlib is part of the standard library, and the format below is the 1989-era subset (no ZIP64, no encryption, no data descriptors) that Cosmopolitan's zipos reads.

Appending, specifically: an APE already contains a ZIP archive (the interpreter's own standard library lives in it), and the existing entries must keep working. The layout of a ZIP file is

[local header + data]*  [central directory]  [end of central directory]

and the central directory records absolute offsets of the local headers. So the append is: cut the file at the start of the central directory, write the new local headers there, write the ORIGINAL central directory bytes unchanged (every offset it holds is still valid, because nothing before it moved), then the central directory records for the new entries, then a fresh end-of-central-directory record. This is what the zip command does when it appends to an archive with a non-ZIP prefix, and it leaves the executable part of the APE - which lives before all of this - byte-identical.

Defined Under Namespace

Classes: Entry

Constant Summary collapse

EOCD_SIGNATURE =

End of central directory record: signature plus 18 bytes of fixed fields; a trailing archive comment may follow.

"PK\x05\x06".b
EOCD_SIZE = 
22
MAX_EOCD_SEARCH =

A ZIP archive comment can be up to 0xffff bytes, so the record can start at most that far from the end of the file.

0xffff + EOCD_SIZE
ZIP64_EOCD_LOCATOR_SIGNATURE =

Markers of the ZIP64 format extensions. OCRAN never writes them; an input archive that uses them is rejected rather than corrupted.

"PK\x06\x07".b
CENTRAL_SIGNATURE = 
"PK\x01\x02".b
LOCAL_SIGNATURE = 
"PK\x03\x04".b
VERSION_MADE_BY =

"Made by" field: UNIX (3) in the high byte so the external file attributes below are read as UNIX permission bits, ZIP spec 2.0 in the low byte.

(3 << 8) | 20
VERSION_NEEDED =

Version needed to extract: 2.0 is what DEFLATE requires.

20
FLAG_UTF8 =

General purpose bit 11: file name is UTF-8.

0x0800
METHOD_STORED = 
0
METHOD_DEFLATED = 
8
MSDOS_DIR_ATTRIBUTE =

MS-DOS directory attribute, set in the low byte of the external file attributes for directory entries.

0x10
S_IFREG =

UNIX st_mode file type bits, stored in the high word of the external file attributes together with the permission bits. They are NOT optional: Cosmopolitan's zipos reports the external attributes as st_mode, and a member without S_IFREG is not a regular file - Ruby's own require/load refuse to open it (they check S_ISREG), and a directory without S_IFDIR cannot be traversed, so Dir.glob comes back empty even though File.read on the exact path works. This mismatch is what makes an otherwise valid archive unusable inside an APE.

0o100000
S_IFDIR = 
0o040000
DEFAULT_FILE_MODE = 
0o644
DEFAULT_DIRECTORY_MODE = 
0o755

Class Method Summary collapse

Class Method Details

.append(path, entries) ⇒ Object

Appends the given entries to the ZIP archive at the end of path, in place. Returns the number of bytes the file grew by.

Raises when the file has no readable central directory, when it uses ZIP64, or when an entry would shadow a name the archive already contains (a duplicate name is not a format error, but for the APE it would mean an application file silently overriding part of the interpreter's own standard library).



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
 
# File 'lib/ocran/zip_writer.rb', line 102

def append(path, entries)
  entries = entries.reject(&:nil?)
  return 0 if entries.empty?

  File.open(path, "r+b") do |io|
    eocd = read_eocd(io, path)
    central = read_central_directory(io, eocd)
    existing = central_directory_names(central)

    entries.each do |entry|
      if existing.include?(entry.name)
        raise "cannot add #{entry.name} to #{path}: the archive already contains an entry with that name"
      end
    end

    entries = with_parent_directories(entries, existing)

    before = io.size
    io.truncate(eocd[:cd_offset])
    io.seek(eocd[:cd_offset])

    records = entries.map { |entry| write_local(io, entry) }

    cd_offset = io.pos
    io.write(central)
    records.each { |record| io.write(central_record(record)) }
    cd_size = io.pos - cd_offset

    io.write(end_of_central_directory(eocd[:total_entries] + records.size, cd_size, cd_offset))
    io.size - before
  end
end

.central_directory_names(central) ⇒ Object

Names of the entries already in the archive, so an application file cannot silently shadow one of them.



198
199
200
201
202
203
204
205
206
207
 
# File 'lib/ocran/zip_writer.rb', line 198

def central_directory_names(central)
  names = {}
  pos = 0
  while central.byteslice(pos, 4) == CENTRAL_SIGNATURE
    name_length, extra_length, comment_length = central.byteslice(pos, 46).unpack("x28vvv")
    names[central.byteslice(pos + 46, name_length)] = true
    pos += 46 + name_length + extra_length + comment_length
  end
  names
end

.central_record(record) ⇒ Object 



257
258
259
260
261
262
263
264
 
# File 'lib/ocran/zip_writer.rb', line 257

def central_record(record)
  external = (record[:mode] << 16) | (record[:directory] ? MSDOS_DIR_ATTRIBUTE : 0)

  [CENTRAL_SIGNATURE, VERSION_MADE_BY, VERSION_NEEDED, record[:flags], record[:method],
   record[:dos_time], record[:dos_date], record[:crc], record[:compressed_size],
   record[:size], record[:name].bytesize, 0, 0, 0, 0, external, record[:offset]]
    .pack("a4vvvvvvVVVvvvvvVV") + record[:name]
end

.compress(content) ⇒ Object

DEFLATE unless it does not pay off. Raw deflate streams (negative window bits) are what the ZIP format stores - Zlib.deflate would add a zlib header that no unzipper expects.



245
246
247
248
249
250
251
252
253
254
255
 
# File 'lib/ocran/zip_writer.rb', line 245

def compress(content)
  return ["".b, METHOD_STORED] if content.empty?

  deflater = Zlib::Deflate.new(Zlib::BEST_COMPRESSION, -Zlib::MAX_WBITS)
  deflated = begin
    deflater.deflate(content, Zlib::FINISH)
  ensure
    deflater.close
  end
  deflated.bytesize < content.bytesize ? [deflated, METHOD_DEFLATED] : [content, METHOD_STORED]
end

.dos_timestamp(time) ⇒ Object

MS-DOS packed time and date. The format has two-second resolution and starts in 1980, so earlier timestamps are clamped.



280
281
282
283
284
285
 
# File 'lib/ocran/zip_writer.rb', line 280

def dos_timestamp(time)
  time = time.getlocal
  year = [time.year, 1980].max
  [(time.hour << 11) | (time.min << 5) | (time.sec / 2),
   ((year - 1980) << 9) | (time.month << 5) | time.day]
end

.end_of_central_directory(total_entries, cd_size, cd_offset) ⇒ Object 



266
267
268
269
270
271
272
273
274
275
276
 
# File 'lib/ocran/zip_writer.rb', line 266

def end_of_central_directory(total_entries, cd_size, cd_offset)
  if total_entries > 0xffff
    raise "too many ZIP entries (#{total_entries}); OCRAN does not write ZIP64 archives"
  end
  if cd_offset + cd_size > 0xffffffff
    raise "the packaged archive would exceed 4 GiB; OCRAN does not write ZIP64 archives"
  end

  [EOCD_SIGNATURE, 0, 0, total_entries, total_entries, cd_size, cd_offset, 0]
    .pack("a4vvvvVVv")
end

.read_central_directory(io, eocd) ⇒ Object 



185
186
187
188
189
190
191
192
193
194
 
# File 'lib/ocran/zip_writer.rb', line 185

def read_central_directory(io, eocd)
  io.seek(eocd[:cd_offset])
  return "".b if eocd[:cd_size].zero?

  central = io.read(eocd[:cd_size]).to_s.b
  unless central.bytesize == eocd[:cd_size] && central.start_with?(CENTRAL_SIGNATURE)
    raise "the ZIP central directory is truncated or malformed"
  end
  central
end

.read_eocd(io, path) ⇒ Object

Locates and decodes the end-of-central-directory record. The record is searched for from the end of the file because a ZIP archive is identified by its tail, which is what allows one to be appended to an executable in the first place.



158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
 
# File 'lib/ocran/zip_writer.rb', line 158

def read_eocd(io, path)
  size = io.size
  tail_size = [size, MAX_EOCD_SEARCH].min
  io.seek(size - tail_size)
  tail = io.read(tail_size)

  offset = tail.rindex(EOCD_SIGNATURE)
  unless offset
    raise "#{path} does not end in a ZIP archive (no end-of-central-directory record found); " \
          "it cannot be a cosmopolitan APE with an embedded ZIP store"
  end

  if tail.rindex(ZIP64_EOCD_LOCATOR_SIGNATURE)
    raise "#{path} uses the ZIP64 format extensions, which OCRAN cannot append to"
  end

  _signature, _disk, _cd_disk, _disk_entries, total_entries, cd_size, cd_offset, comment_length =
    tail.byteslice(offset, EOCD_SIZE).unpack("a4vvvvVVv")

  eocd_start = size - tail_size + offset
  unless eocd_start + EOCD_SIZE + comment_length == size
    raise "#{path} has trailing data after its ZIP archive; OCRAN cannot append to it"
  end

  { cd_offset: cd_offset, cd_size: cd_size, total_entries: total_entries }
end

.st_mode(entry) ⇒ Object

The UNIX st_mode an extractor (and zipos) should report for the entry: the permission bits plus the file type.



234
235
236
237
238
239
240
 
# File 'lib/ocran/zip_writer.rb', line 234

def st_mode(entry)
  if entry.directory?
    S_IFDIR | (entry.mode || DEFAULT_DIRECTORY_MODE)
  else
    S_IFREG | (entry.mode || DEFAULT_FILE_MODE)
  end
end

.with_parent_directories(entries, existing) ⇒ Object

Returns the entries with an explicit directory entry inserted before every member for each parent directory that neither the archive nor the entry list already provides. A ZIP archive does not require them, but zipos builds its directory listings from the members it can see, so without them Dir.glob and Dir.entries do not find the packed tree.



140
141
142
143
144
145
146
147
148
149
150
151
152
 
# File 'lib/ocran/zip_writer.rb', line 140

def with_parent_directories(entries, existing)
  seen = existing.dup
  entries.each { |entry| seen[entry.name] = true }

  entries.flat_map { |entry|
    parents = entry.name.split("/")[0...-1].inject([]) { |acc, part|
      acc << "#{acc.last}#{part}/"
    }
    missing = parents.reject { |name| seen[name] }
    missing.each { |name| seen[name] = true }
    missing.map { |name| Entry.new(name: name) } << entry
  }
end

.write_local(io, entry) ⇒ Object

Writes one local file header plus its data at the current position and returns the bookkeeping the central directory record needs.



211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
 
# File 'lib/ocran/zip_writer.rb', line 211

def write_local(io, entry)
  content = entry.content
  crc = Zlib.crc32(content)
  compressed, method = compress(content)

  name = entry.name.b
  flags = name.ascii_only? ? 0 : FLAG_UTF8
  dos_time, dos_date = dos_timestamp(entry.mtime || Time.now)
  offset = io.pos

  io.write([LOCAL_SIGNATURE, VERSION_NEEDED, flags, method, dos_time, dos_date,
            crc, compressed.bytesize, content.bytesize, name.bytesize, 0]
             .pack("a4vvvvvVVVvv"))
  io.write(name)
  io.write(compressed)

  { name: name, flags: flags, method: method, dos_time: dos_time, dos_date: dos_date,
    crc: crc, compressed_size: compressed.bytesize, size: content.bytesize,
    offset: offset, mode: st_mode(entry), directory: entry.directory? }
end