Module: Xolo::Server::Helpers::Maintenance

Defined in:
lib/xolo/server/helpers/maintenance.rb

Overview

Nightly maintenance tasks

  • accept TEd EAs
  • autorelease versions
  • clean up old versions

Also, alerts will be posted, and Emails will be sent to the admins who added versions that have been in pilot for more than some period of time.

Constant Summary collapse

MAINT_HOUR =

At what hour should the nightly maintenance run?

2
TIMED_MAINT_TRIGGER_ROUTE =

the route we POST to, to start the nightly process.

'/maint/maint-internal'
UNRELEASED_PILOTS_NOTIFICATION_DAY =

on which day of the month should we send the unreleased pilot notifications?

1
DFT_DEPRECATED_LIFETIME_DAYS =

Once a version becomes deprecated, it will be automatically deleted this many days later. If not set in the server config, this is the default value. use 0 or less to disable maintenance of deprecated versions

30
DFT_UNRELEASED_PILOTS_NOTIFICATION_DAYS =

If a pilot has not been released in this many days, notify someone about it weekly, asking to release it or delete it. If not set in the server config, this is the default value.

180
SERVER_LAUNCHD_PLIST =

when doing a full shutdown, we need to unload the launchd plist

Pathname.new '/Library/LaunchDaemons/com.pixar.xoloserver.plist'

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.included(includer) ⇒ Object

when this module is included



28
29
30
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 28

def self.included(includer)
  Xolo.verbose_include includer, self
end

.last_maint ⇒ Time

When was our last maint?

Returns:

  • (Time) —

    the time of the last maint, or the epoch if never



96
97
98
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 96

def self.last_maint
  @last_maint ||= Time.at(0)
end

.last_maint=(time) ⇒ Time

Set the time of the last maint

Parameters:

  • time (Time) —

    the time of the last maint

Returns:

  • (Time) —

    the time of the last maint



104
105
106
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 104

def self.last_maint=(time)
  @last_maint = time
end

.maint_mutex ⇒ Mutex

A mutex for the maint process

Returns:

  • (Mutex) —

    the mutex



68
69
70
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 68

def self.maint_mutex
  @maint_mutex ||= Mutex.new
end

.maint_timer_task ⇒ Concurrent::TimerTask

nightly maint is done by a Concurrent::TimerTask, which checks every hour to see if it should do anything.

It will only do the maint if the current time is in the MAINT_HOUR hour (02:00 - 02:59 if MAINT_HOUR = 2)

We trigger the maint by POSTing to TIMED_MAINT_TRIGGER_ROUTE, so that it runs in the context of a request, having access to Title and Version instantiation.

Returns:

  • (Concurrent::TimerTask) —

    the timed task to do log rotation



83
84
85
86
87
88
89
90
91
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 83

def self.maint_timer_task
  return @maint_timer_task if @maint_timer_task

  @maint_timer_task =
    Concurrent::TimerTask.new(execution_interval: 3600) { post_to_start_maint }

  Xolo::Server.logger.info 'Created Concurrent::TimerTask for nightly maintenance.'
  @maint_timer_task
end

.post_to_start_maint(force: false) ⇒ void

This method returns an undefined value.

post to the server to start the maint process This is done so that the maint can run in the context of a request, having access to Title and Version instantiation.

Parameters:

  • force (Boolean) (defaults to: false) —

    force the maint to run now



115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 115

def self.post_to_start_maint(force: false)
  if Xolo::Server.shutting_down?
    Xolo::Server.logger.info 'Not starting maintenance, server is shutting down'
    return
  end

  # only run the maintenance if it's between 2am and 3am
  # and the last one was more than 23 hrs ago
  last_maint_hrs_ago = (Time.now - last_maint) / 3600
  return unless force || (Time.now.hour == MAINT_HOUR && last_maint_hrs_ago > 23)

  uri = URI.parse "https://#{Xolo::Server::Helpers::Auth::IPV4_LOOPBACK}#{TIMED_MAINT_TRIGGER_ROUTE}"
  https = Net::HTTP.new(uri.host, uri.port)
  https.use_ssl = true
  # The server cert may be self-signed and/or doesn't
  # match the hostname, so we need to disable verification
  https.verify_mode = OpenSSL::SSL::VERIFY_NONE

  request = Net::HTTP::Post.new(uri.path)
  request['Authorization'] = Xolo::Server::Helpers::Auth.internal_auth_token_header

  response = https.request(request)
  Xolo::Server.logger.info "Maintenance request response: #{response.code} #{response.body}"
end

Instance Method Details

#accept_title_editor_eas ⇒ void

This method returns an undefined value.

look for any titles that need their Title Editor EA's accepted, and auto accept them if we need to



172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 172

def accept_title_editor_eas
  unless Xolo::Server.config.jamf_auto_accept_xolo_eas
    log_info 'Maint: The xolo server is not configured to auto-accept Title Editor EAs'
    return
  end

  log_info 'Maint: Looking for Title Editor EAs to auto-accept'

  # TODO: Be DRY with this stuff and similar in title_jamf_access.rb
  Xolo::Server::Title.all_titles.each do |title|
    title_obj = instantiate_title title
    next if title_obj.subscribed?
    next unless title_obj.jamf_patch_ea_awaiting_acceptance?

    log_info "Maint: Auto-accepting Title Editor EA for title '#{title}'"
    title_obj.accept_jamf_patch_ea_via_api
  rescue => e
    log_error "Maint: Error auto-accepting Title Editor EA for title '#{title}': #{e}"
  end # Xolo::Server::Title.all_titles.each

  log_info 'Maint: Done with Title Editor EAs to auto-accept'
end

#auto_release_versions ⇒ void

This method returns an undefined value.

Do any pending auto-releases



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 198

def auto_release_versions
  log_info 'Maint: Auto-releasing appropriate versions'

  today = Date.today

  # Loop thru the titles
  Xolo::Server::Title.all_titles.each do |title|
    title_obj = instantiate_title title

    # title must be subscribed and autopkg
    # (enforced in xadm)
    next unless title_obj.subscribed? && !title_obj.autopkg_recipe.pix_empty?

    # title must have a non-negative integer for auto_release_delay,
    # (number of days to wait before release)
    # could be zero. (enforced in xadm)
    # NOTE: it is stored as a string because it might be 'none'
    # TODO: store none as nil, so we don't have to do the pix_integer check??
    delay = title_obj.auto_release_delay.to_i
    next unless title_obj.auto_release_delay.pix_integer? && !delay.negative?

    # Any version with this creation date should be released now, skip if not
    # So if the auto_release_delay is 7, its today - 7 days.
    # if the auto_release_delay is 0, its today - 0 days, aka today
    creation_date_to_be_released_today = today - delay

    # Find the newest pilot version whose creation date is creation_date_to_be_released_today
    # That one should be released now.
    vobj_to_release = nil
    title_obj.version_objects.each do |vobj|
      next unless vobj.pilot?
      next unless vobj.creation_date.to_date == creation_date_to_be_released_today

      vobj_to_release = vobj
      break
    end

    # no versions to release today
    next unless vobj_to_release

    # release it
    log_debug "Maint: About to auto-release version '#{vobj_to_release.version}' of '#{title}' which came out #{vobj_to_release.creation_date}"

    # releasing a version is done via the title obj, since it affects the status
    # of all versions.
    title_obj.release vobj_to_release.version

    log_info "Maint: Auto-released version '#{vobj_to_release.version}' of '#{title}' which came out #{vobj_to_release.creation_date}", alert: true
  end # each title

  log_debug 'Maint: Done auto-releasing'
end

#cleanup_deprecated_version(version) ⇒ void

This method returns an undefined value.

Cleanup a deprecated version.

Parameters:



279
280
281
282
283
284
285
286
287
288
289
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 279

def cleanup_deprecated_version(version)
  # do nothing if the deprecated_lifetime_days is 0 or less
  return unless deprecated_lifetime_days.positive?

  # how many days has this version been deprecated?
  days_deprecated = (Time.now - version.deprecation_date) / 86_400
  return unless days_deprecated > deprecated_lifetime_days

  log_info "Cleanup: Deleting deprecated version '#{version.version}' of title '#{version.title}'"
  version.delete
end

#cleanup_skipped_version(version) ⇒ void

This method returns an undefined value.

Cleanup a skipped version.

Parameters:



295
296
297
298
299
300
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 295

def cleanup_skipped_version(version)
  return if Xolo::Server.config.keep_skipped_versions

  log_info "Cleanup: Deleting skipped version '#{version.version}' of title '#{version.title}'"
  version.delete
end

#cleanup_versions ⇒ void

This method returns an undefined value.

Cleanup versions.



254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 254

def cleanup_versions
  log_info 'Maint: cleaning up deprecated and skipped versions'

  Xolo::Server::Title.all_titles.each do |title|
    title_obj = instantiate_title title

    title_obj.version_objects.each do |version|
      if version.deprecated?
        cleanup_deprecated_version version
      elsif version.skipped?
        cleanup_skipped_version version
      end # case
    end # each version

    notify_admins_of_unreleased_pilots(title_obj)
  end # each title

  Xolo::Server::Helpers::Maintenance.last_maint = Time.now
  log_info 'Maint: versions cleanup complete'
end

#deprecated_lifetime_days ⇒ Integer

how many days can a version be deprecated?

Returns:

  • (Integer) —

    the number of days a version can be deprecated



335
336
337
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 335

def deprecated_lifetime_days
  @deprecated_lifetime_days ||= Xolo::Server.config.deprecated_lifetime_days || DFT_DEPRECATED_LIFETIME_DAYS
end

#notify_admins_of_unreleased_pilots(title_obj) ⇒ void

This method returns an undefined value.

Notify the admins about unreleased pilots if needed



305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 305

def notify_admins_of_unreleased_pilots(title_obj)
  return unless Time.now.day == UNRELEASED_PILOTS_NOTIFICATION_DAY
  return unless unreleased_pilots_notification_days.positive?
  return unless title_obj.latest_version

  latest_vers_obj = instantiate_version title: title_obj, version: title_obj.latest_version
  return unless latest_vers_obj.pilot?

  days_in_pilot = ((Time.now - latest_vers_obj.creation_date) / 86_400).to_i

  return unless days_in_pilot > unreleased_pilots_notification_days

  alert_msg = "Cleanup: Notifying #{title_obj.contact_email} about unreleased pilot '#{latest_vers}' of title '#{title_obj.title}', in pilot for #{days_in_pilot} days"

  log_info alert_msg
  send_alert alert_msg

  email_msg = <<~MSG
    The newest version '#{latest_vers_obj.version}' of title '#{title_obj.title}' has been in pilot for #{days_in_pilot} days, which makes it seem like it's not going to be released.

    To reduce clutter, please consider releasing it, deleting it, or deleting the whole title if it's no longer needed.

    If this is intentional, you can ignore this monthly message.
  MSG
  send_email to: title_obj.contact_email, subject: 'Unreleased Pilot Notification', msg: email_msg
end

#run_maint ⇒ void

This method returns an undefined value.

Perform maintenance tasks



143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 143

def run_maint
  if Xolo::Server.shutting_down?
    log_info 'Maint: Not starting maint, server is shutting down'
    return
  end

  mutex = Xolo::Server::Helpers::Maintenance.maint_mutex

  if mutex.locked?
    log_warn 'Maint: already running, skipping this run'
    return
  end
  mutex.lock
  log_info 'Maint: starting'

  # add new maintenance tasks/methods here
  accept_title_editor_eas
  auto_release_versions
  cleanup_versions

  log_info 'Maint: complete'
ensure
  mutex&.unlock
end

#shutdown_pkg_deletion_pool ⇒ void

This method returns an undefined value.

Shutdown the pkg deletion pool



436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 436

def shutdown_pkg_deletion_pool
  # Start the shutdown of the pkg_deletion_pool. Will finish anything
  # in the queue, but not accept any new tasks.
  pkg_pool = Xolo::Server::Version.pkg_deletion_pool
  pkg_pool.shutdown
  pkg_pool_shutdown_start = Time.now
  progress 'Shutting down pkg deletion pool', log: :info
  # returns true when shutdown is complete
  until pkg_pool.wait_for_termination(20)
    msg = "..Waiting for pkg deletion pool to finish, processing: #{pkg_pool.length}, in queue: #{pkg_pool.queue_length}"
    progress msg, log: :debug
    next unless Time.now - pkg_pool_shutdown_start > Xolo::Server::Constants::MAX_JAMF_WAIT_FOR_PKG_DELETION

    msg = 'ERROR: Timeout waiting for pkg deletion pool to finish, some pkgs may not be deleted'
    progress msg, log: :error
    pkg_pool.kill
    break
  end
  progress 'Pkg deletion queue is empty'
end

#shutdown_server(restart) ⇒ void

This method returns an undefined value.

Shutdown the server



349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 349

def shutdown_server(restart)
  # let all the routes know we are shutting down
  Xolo::Server.shutting_down = true

  progress "Server Shutdown by #{session[:admin]}", log: :info

  stop_maint_timer_task
  stop_log_rotation_timer_task
  shutdown_pkg_deletion_pool
  wait_for_object_locks
  wait_for_progress_streams

  # without unloading the launchd job, the server will restart automatically
  # when we tell it to quit
  if restart
    progress 'Restarting the server now', log: :info
    Xolo::Server::App.quit!
  else
    progress 'Shutting down the server now', log: :info
    unload_server_launchd
  end
end

#stop_log_rotation_timer_task ⇒ void

This method returns an undefined value.

Stop the log rotation timer task



391
392
393
394
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 391

def stop_log_rotation_timer_task
  progress 'Stopping the log rotation timer task', log: :info
  Xolo::Server::Log.log_rotation_timer_task.shutdown
end

#stop_maint_timer_task ⇒ void

This method returns an undefined value.

Stop the maintenance timer task



383
384
385
386
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 383

def stop_maint_timer_task
  progress 'Stopping the maint timer task', log: :info
  Xolo::Server::Helpers::Maintenance.maint_timer_task.shutdown
end

#unload_server_launchd ⇒ void

This method returns an undefined value.

full shutdown of the server by unloading the launchd plist



375
376
377
378
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 375

def unload_server_launchd
  log_info 'Unloading the server launchd plist'
  system "/bin/launchctl unload #{SERVER_LAUNCHD_PLIST}"
end

#unreleased_pilots_notification_days ⇒ Object

Notify the admins about unreleased pilots when the newest one is older than this many days.



341
342
343
344
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 341

def unreleased_pilots_notification_days
  @unreleased_pilots_notification_days ||=
    Xolo::Server.config.unreleased_pilots_notification_days || DFT_UNRELEASED_PILOTS_NOTIFICATION_DAYS
end

#wait_for_object_locks ⇒ void

This method returns an undefined value.

Wait for all object locks to be released



399
400
401
402
403
404
405
406
407
408
409
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 399

def wait_for_object_locks
  Xolo::Server.remove_expired_object_locks

  until Xolo::Server.object_locks.empty?
    progress 'Waiting for object locks to be released', log: :info
    log_debug "Object locks: #{Xolo::Server.object_locks.inspect}"
    sleep 5
    Xolo::Server.remove_expired_object_locks
  end
  progress 'All object locks released', log: :info
end

#wait_for_progress_streams ⇒ void

This method returns an undefined value.

Wait for all progress streams to finish



414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 414

def wait_for_progress_streams
  prefix = Xolo::Server::Helpers::ProgressStreaming::PROGRESS_THREAD_NAME_PREFIX
  prog_threads = Thread.list.select { |th| th.name.to_s.start_with? prefix }
  # remove our own thread from the list
  prog_threads.delete Thread.current
  prog_threads.delete @streaming_thread

  until prog_threads.empty?
    progress 'Waiting for progress streams to finish', log: :info
    log_debug "Progress stream threads: #{prog_threads.map(&:name)}}"
    sleep 5
    prog_threads = Thread.list.select { |th| th.name.to_s.start_with? prefix }
    # remove our own thread from the list
    prog_threads.delete Thread.current
    prog_threads.delete @streaming_thread
  end
  progress 'All progress streams finished', log: :info
end