BulkCsvParser

Generic bulk CSV import/update for any ActiveRecord model. You declare:

• which model to import into
• which attributes identify an existing record (find_by)
• which attributes should be created/updated (attributes)

...and the gem handles the rest: streaming large files without loading them fully into memory, batching, and (optionally) running the whole import in a background job.

Installation

Add to your Rails app's Gemfile:

gem "bulk_csv_parser", path: "../csv_gem" # or git: "..." once pushed, or from rubygems once published

bundle install
bin/rails generate bulk_csv_parser:install

This creates config/initializers/bulk_csv_parser.rb where you can set defaults (batch size, strategy, queue name, upload dir).

Basic usage (synchronous)

result = BulkCsvParser.import(
  file_path: params[:file].path,
  model: User,
  find_by: [:email],
  attributes: [:first_name, :last_name, :status]
)

result.success_count # => 980
result.failed_count  # => 20
result.errors        # => [{ row: {...}, error: "Email can't be blank" }, ...]

CSV headers are normalized automatically ("First Name" -> :first_name). If your CSV headers don't match your model's attribute names, map them:

BulkCsvParser.import(
  file_path: "/tmp/users.csv",
  model: User,
  find_by: [:email],
  attributes: [:first_name, :status],
  header_mapping: { "Email Address" => :email, "Full Name" => :first_name }
)

Need custom logic per row (parsing dates, computing derived fields, skipping rows)? Pass transform, which receives the normalized row hash and returns the hash to use (or nil to skip the row):

BulkCsvParser.import(
  file_path: "/tmp/users.csv",
  model: User,
  find_by: [:email],
  attributes: [:status, :activated_at],
  transform: ->(row) {
    row[:status] = row[:status].to_s.downcase
    row[:activated_at] = Date.parse(row[:activated_at]) rescue nil
    row
  }
)

Background processing (large files, and workers on a different machine)

Uploaded tempfiles disappear once the request ends, and in production your job queue is very often processed on a different machine than the one that received the upload (a separate Sidekiq/Resque/GoodJob fleet) with no access to that web server's local disk. Passing a bare file_path across that boundary silently breaks.

BulkCsvParser.enqueue solves this via BulkCsvParser::Storage: it uploads the file to whatever service Active Storage is configured with (config/storage.yml — S3, GCS, Azure) and passes a small signed reference to the job instead of a path. Any worker with the same storage credentials can fetch it back, regardless of which machine it runs on:

class ImportsController < ApplicationController
  def create
    BulkCsvParser.enqueue(
      file: params[:file], # anything responding to #path
      model: User,
      find_by: [:email],
      attributes: [:first_name, :last_name, :status],
      notify_email: current_user.email
    )

    redirect_to imports_path, notice: "Import queued"
  end
end

This requires Active Storage to be set up (bin/rails active_storage:install and a non-:local service configured for any multi-machine deployment — the local disk service has the exact same cross-machine problem the file upload itself does). config.storage defaults to :active_storage automatically whenever Active Storage is loaded.

If web and worker genuinely share a disk (single-machine deploys, or a shared NFS/EFS mount), you can opt into the simpler local-copy adapter:

BulkCsvParser.configure { |c| c.storage = :local } # config/initializers/bulk_csv_parser.rb

You can also enqueue the job directly if you already have a storage reference:

BulkCsvParser::ImportJob.perform_later(
  file_reference: some_active_storage_blob.signed_id,
  model: "User",
  find_by: [:email],
  attributes: [:first_name, :last_name],
  notify_email: "ops@example.com"
)

The job runs on whatever ActiveJob adapter your app uses (Sidekiq, Resque, GoodJob, etc.) on the bulk_csv_parser queue (configurable), and always deletes/purges the stored file when it finishes, even on failure.

Email notification of results

Pass notify_email: to BulkCsvParser.enqueue or ImportJob.perform_later and, once the import finishes, BulkCsvParser::ImportMailer sends an email with the success count, failure count, and the reason each failed row failed (capped at config.notify_error_limit, default 25, so a file with thousands of bad rows doesn't produce a giant email):

BulkCsvParser.enqueue(
  file: params[:file],
  model: User,
  find_by: [:email],
  attributes: [:first_name, :status],
  notify_email: "ops@example.com"
)

Set the sender address in the initializer:

BulkCsvParser.configure { |c| c.mailer_sender = "imports@yourapp.com" }

Strategies

• :save (default) — find_or_initialize_by + assign_attributes + save per row. Runs validations and callbacks. Safer, slower.
• :upsert — batches rows into a single upsert_all call per batch. Much faster for very large files, but skips validations/callbacks and requires a DB-level unique index covering the find_by columns. created_at is only set on insert and left alone on conflict, so re-importing an existing record won't clobber when it was originally created — only updated_at and the declared attributes change on update.

BulkCsvParser.import(
  file_path: "/tmp/huge_import.csv",
  model: Product,
  find_by: [:sku],
  attributes: [:name, :price, :stock],
  strategy: :upsert,
  batch_size: 5000
)

Large files

The processor uses CSV.foreach to stream row by row and only holds batch_size rows in memory at a time, so multi-GB files are safe to import. Tune batch_size (default 1000) to trade off memory usage vs. number of DB round-trips.

Configuration

BulkCsvParser.configure do |config|
  config.batch_size = 1000
  config.strategy    = :save
  config.queue_name  = :bulk_csv_parser
  config.upload_dir  = Rails.root.join("tmp", "bulk_csv_parser")
end

Handling row-level errors

BulkCsvParser.import(
  file_path: "/tmp/users.csv",
  model: User,
  find_by: [:email],
  attributes: [:status],
  on_row_error: ->(row, error) { Rails.logger.error("Row failed: #{row} - #{error.message}") }
)

License

MIT