UPD: 31/03/2024 I've opened a PR to add touch option to #update_columns and #update_column method

In modern web development, precise and efficient data management is crucial for making informed business decisions.

Data consistency, particularly in how database records are dated, plays a crucial role. Data engineers often rely on these dates not only for record-keeping but also as a method to download only the data that has changed, avoiding the need to process entire tables.

This guide addresses the challenge of mastering dating amid the inconsistencies found in Rails' handling of timestamps. It explores strategies for ensuring that timestamps are both accurate and dependable, offering valuable insights for developers looking to navigate these complexities. Whether you're new to the field or seeking solutions to specific dating challenges, this article provides practical tips and strategies to improve your data management practices.

TLDR

Not all ActiveRecord persistence methods affect timestamps or have a touch option. For methods like update_columns that don't automatically update timestamps, you can create a RuboCop custom cop, modify ActiveRecord directly, or use database triggers to keep updated_at always up-to-date.

Each method has its advantages and disadvantages, from how easy they are to manage to potential compatibility issues with future Rails updates or the risk of SQL operations skipping ActiveRecord methods.

If you're interested in this topic, join the Rails Discussion thread https://discuss.rubyonrails.org/t/proposal-add-touch-option-for-update-columns-update-column/85388

ActiveRecord timestamps configuration

ActiveRecord automatically timestamps create and update operations if the table has fields named created_at/created_on or updated_at/updated_on.

For turning off timestamping, add:

config.active_record.record_timestamps = false # true is default

Timestamps are in UTC by default, but you can use the local timezone by setting:

config.active_record.default_timezone = :local # is :utc by default

ActiveRecord keeps all the datetime and time columns timezone aware. By default, these values are stored in the database as UTC and converted back to the current Time.zone when pulled from the database.

This feature can be turned off completely by setting:

config.active_record.time_zone_aware_attributes = false # is true by default

ActiveRecord persistence methods and touching timestamps

There are a lot of persistence methods in ActiveRecord, but not all of them touch timestamps or have a touch option. You can find most of them in the table below.

As you can see, three methods don't update timestamps by default nor provide a touch option: update_column, update_columns, and update_all. Sometimes this may be a problem, i.e., there is some ETL processing that, instead of copying the whole table, looks into updated_at timestamps. So if someone uses update_columns because of performance reasons, it may lead to lost updates. However, there are a couple of methods to solve this problem.

Let's consider a pretty basic example.

class ApplicationController < ActionController::Base
  before_action :update_last_user_ip

  def update_last_user_ip
    ip = request.remote_ip
    return if current_user.last_ip != ip
    # we don't perform any callbacks or validations here
    # so use #update_columns
    current_user.update_columns(
        last_ip: request.remote_ip,
        updated_at: Time.current # but we still want to keep track of the last changes, so have to provide timestamp explicitly
    )
  end
end

Here, we aim to update the user's last seen IP in their record without triggering any validations or callbacks. The simplest method for this is using the #update_columns method. However, to ensure the timestamp remains current, we must explicitly include updated_at. What issues might arise from this approach?

Several, including:

• Remembering that #update_columns does not update timestamps, a behavior that is documented but might still catch you off guard.

• The need to explicitly set updated_at/updated_on.

• The absence of a touch option, unlike what you find in methods like #increment!.

• The record_timestamps setting does not affect timestamp behavior.

So, what can we do if we want to consistently update timestamps across the application? There are a few solutions.

Rubocop Cop

RuboCop lets you make your own custom cops. You need to make a new file for your custom cop, which we'll name UpdateColumnsCop. Put this file in a folder where RuboCop looks for custom cops. A usual spot for this is lib/rubocop/cop/.

Here's a simple setup for your custom cop:

# rubocop/cop/rails/update_columns_timestamps.rb
module RuboCop
  module Cop
    module Rails
      class UpdateColumnsCop < RuboCop::Cop::Base
        extend RuboCop::Cop::AutoCorrector

        MSG = "Ensure `updated_at` or `updated_on` is updated when using `update_columns`"

        def_node_matcher :update_columns?, <<-PATTERN
           (send _ {:update_columns} ...)
        PATTERN

        def on_send(node)
          return unless update_columns?(node)

          # Check if `updated_at` or `updated_on` is being updated
          updated_at_or_updated_on_updated = node.arguments.any? do |arg|
            arg.hash_type? && arg.pairs.any? do |pair|
              pair.key.value == :updated_at || pair.key.value == :updated_on
            end
          end

          return if updated_at_or_updated_on_updated

          add_offense(node, message: MSG) do |corrector|
            corrector.insert_after(node.loc.selector, ", updated_at: Time.current")
          end
        end
      end
    end
  end
end

To make RuboCop aware of your custom cop, you need to register it. Create a .rubocop.yml file in your project root if you don't already have one, and add the following configuration:

require:
 - ./rubocop/cop/rails/update_columns_timestamps.rb

Rails/UpdateColumnsCop:
 Enabled: true

It provides lint error in case of using update_columns without updated_at or update_on attribute:

app/controllers/application_controller.rb:10:5: C: [Correctable] Rails/UpdateColumnsCop: Ensure updated_at or updated_on is updated when using update_columns
    current_user.update_columns( ...

Pros:

• Identifies violations without changing behavior

• Can be modified or ignored like a standard RuboCop cop

Cons:

• It can't handle update_column because it doesn't offer an option for timestamps. This requires an additional rule that completely discourages the use of update_column in favor of update_columns.

• It doesn't address every situation. For instance, using raw SQL might still bypass updating the updated_at field.

Monkey patch ActiveRecord update_column, update_columns

This method, inspired by Tim McCarthy's gist (with Unathi Chonco as an original author), includes a few modifications for safer patching and extra features.

• Add an initializer for patches

•       # config/initializers/core_ext_require.rb
        # NOTE: Require all patches in lib/core_ext
        Dir[Rails.root.join("lib/core_ext/**/*.rb")].each { |f| require f }
  

• Add a patch for the Persistence module

    # lib/core_ext/active_record/persistence/update_columns_patch.rb
    module CoreExt
      module ActiveRecord
        module Persistence
          module UpdateColumnsPatch
            # https://github.com/rails/rails/blob/36c1591bcb5e0ee3084759c7f42a706fe5bb7ca7/activerecord/lib/active_record/persistence.rb#L931-L954
            def update_columns(attributes)
              touch = attributes.delete(:touch) { self.class.record_timestamps }
              if touch
                names = touch if touch != true
                names = Array.wrap(names)
                options = names.extract_options!
                touch_updates = self.class.touch_attributes_with_time(*names, **options)
                attributes.merge!(touch_updates) unless touch_updates.empty?
              end
              super(attributes)
            end
  
            # https://github.com/rails/rails/blob/36c1591bcb5e0ee3084759c7f42a706fe5bb7ca7/activerecord/lib/active_record/persistence.rb#L910-L913
            def update_column(name, value, touch: true)
              update_columns(name => value, :touch => touch)
            end
          end
        end
      end
    end
  
    ActiveRecord::Persistence.prepend(CoreExt::ActiveRecord::Persistence::UpdateColumnsPatch)
  

  This patch mimics the behavior of the #save method: it updates timestamps by default and introduces a touch: option to choose whether to skip the update. It also respects the record_timestamps setting, both globally and at the model level. With this change, we can simplify our example as follows:

    class ApplicationController < ActionController::Base
      before_action :update_last_user_ip
  
      def update_last_user_ip
        ip = request.remote_ip
        return if current_user.last_ip != ip
        # we don't perform any callbacks or validations here
        # so use #update_columns
        current_user.update_columns(last_ip: request.remote_ip)
      end
    end
  

  #update_columns does automatically update the updated_at field. However, if you need to avoid updating it for some reason, you can explicitly use the touch option:

    current_user.update_columns(last_ip: request.remote_ip, touch: false)
  

  Also, if attribute names are provided, they are updated together with the updated_at/updated_on attributes, similar to how #update_counters works.

    current_user.update_columns(
      last_ip: request.remote_ip,
      touch: :last_ip_updated_at
    )
  

Pros:

• An ad-hoc solution that's easy to manage and adjust.

• Behaves similarly to what we're used to with most methods in the Persistence module.

Cons:

• Involves monkey patching, which might break in future Rails updates.

• Doesn't address all scenarios, for example, using raw SQL might still bypass updating the updated_at field.

If you think these changes are worth including in Rails, please join the Rails Discussion and leave a comment: https://discuss.rubyonrails.org/t/proposal-add-touch-option-for-update-columns-update-column/85388

Database triggers

If you need to update timestamps on each insert or update, even with raw SQL, you should use database triggers. Database triggers are pieces of procedural code that run in response to specific events in a database. For updating timestamps, this could be an UPDATE SQL statement.

First, we'll create the trigger function. This function is triggered whenever an update operation happens on a table. It will automatically update the updated_at column to the current timestamp.

CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
    NEW.updated_at = NOW();
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

This function, update_updated_at_column, is a simple PL/pgSQL function that sets the updated_at column of the new row (NEW) to the current timestamp (NOW()).

Next, you need to create a trigger for each table you want to track updates on. Here's how you can create a trigger for a specific table, let's say your_table_name:

CREATE TRIGGER update_your_table_name_trigger
BEFORE UPDATE ON your_table_name
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();

This trigger, update_your_table_name_trigger, is set to execute before any update operation on your_table_name. It calls the update_updated_at_column function, which updates the updated_at column.

To handle both insert and update events for setting created_at and updated_at timestamps, you'll need to create two separate triggers for each event type. The first trigger will handle the insert event, setting both created_at and updated_at to the current timestamp. The second trigger will handle the update event, setting only the updated_at column to the current timestamp:

CREATE OR REPLACE FUNCTION update_created_updated_at_columns()
RETURNS TRIGGER AS $$
BEGIN
    IF TG_OP = 'INSERT' THEN
        NEW.created_at = NOW();
        NEW.updated_at = NOW();
    ELSIF TG_OP = 'UPDATE' THEN
        NEW.updated_at = NOW();
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

This function checks the operation type (TG_OP) to determine if it's an insert or update operation. For inserts, it sets both created_at and updated_at to the current timestamp. For updates, it only updates the updated_at column.

Now, create the triggers for both insert and update events:

-- Trigger for INSERT
CREATE TRIGGER insert_your_table_name_trigger
BEFORE INSERT ON your_table_name
FOR EACH ROW
EXECUTE FUNCTION update_created_updated_at_columns();

-- Trigger for UPDATE
CREATE TRIGGER update_your_table_name_trigger
BEFORE UPDATE ON your_table_name
FOR EACH ROW
EXECUTE FUNCTION update_created_updated_at_columns();

For better triggers management within Rails it's recommended to use tools like fx or hair_trigger.

Pros:

• Always up-to-date created_at/updated_at timestamps

Cons:

• Triggers are difficult to manage

• You need to add triggers for each new table where you want to keep the timestamps current

• Triggers make the app behavior less obvious and, sometimes you might not want to update timestamps, and that removes control from the app

Sources:

• https://api.rubyonrails.org/classes/ActiveRecord/Timestamp.html

• https://api.rubyonrails.org/classes/ActiveRecord/Persistence.html

• https://gist.github.com/timm-oh/9b702a15f61a5dd20d5814b607dc411d

• https://en.wikipedia.org/wiki/Database_trigger

• https://github.com/teoljungberg/fx

• https://github.com/rubocop/rubocop

If you're looking for a quick solution to handle annoying or expected errors with Sidekiq just try the sidekiq-rescue gem

class MyJob
  include Sidekiq::Job
  include Sidekiq::Rescue::Dsl

  sidekiq_rescue Faraday::ConnectionFailed, delay: 60, limit: 5

  def perform(*)
    # ...
  end
end

If you are interested in learning some hints and tricks about this topic - welcome to the article!

Handling errors in Sidekiq

Sidekiq is one of the most popular background processors for Ruby. It's very easy to integrate with Rails or use it as a standalone without any frameworks.

One of the major aspects of software development is error observability and error handling. It's highly recommended to use one of the error services like Rollbar, Sentry, Honeybadger, Bugsnag, etc.

Most of these services have integration with Sidekiq and it's very easy to set up and have all your errors in one place. Thus, you and your team will always be able to see bugs that have occurred.

But while you are fixing the next bug, Sidekiq does one important thing for you - keep a fallen job in a special queue and retry it.

There is a standard way to retry Sidekiq jobs that have been falling with errors - the automatic job retry mechanism

class RetryableJob
  include Sidekiq::Job
  sidekiq_options retry: 25

  def perform(...)
  end
end

It's a default setting, so specify it only if you want to customize the number of retries

class RetryableJob
  include Sidekiq::Job

  def perform(...)
  end
end

It uses a very clever strategy with an exponential backoff using the formula (retry_count ** 4) + 15 + (rand(10) * (retry_count + 1)) (i.e. 15, 16, 31, 96, 271, ... seconds + a random amount of time). It will perform 25 retries over approximately 20 days.

It works very well when a job encounters a bug and fails because a developer has plenty of time before the job stops retrying. After that time it will go to the "morgue" and will be there for 6 months, and then will be discarded. It is possible to retry the job from the UI or console after a bug has been fixed.

But in real-world applications, not all the exceptions are bugs. It can be also a network issue, 3rd party services downtime. Or your application is using distributed lock heavily and the resource is busy now and needs to be accessed sometime later.

Sidekiq retry mechanism works fine with all of these cases, but there are some downsides though - the error is still reported to the error service. Such behavior decreases the visibility of real bugs and some errors can be quite annoying.

So there are two types of errors that devs usually deal with:

• unexpected errors that we want to see as soon as possible in the error tracker because it's probably a bug

• expected errors that can occur from time to time and should be reported only when get out of control

As with the first Sidekiq default retry mechanism works perfectly the second one requires some effort to handle.

Handle expecting errors

Let's consider an example for a clear picture:

class HardJob
  include Sidekiq::Job

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  end
end

In the given example, HardJob fails with approximately a 1% probability. Let's assume that we don't control how often this error occurs. Naturally, we could ignore such errors, but this might lead to a situation where the error spirals out of control and we remain oblivious to it.

However, several techniques can help!

#1 Technique: Ignore expected errors

The first technique proposed by Mike Perham, the author of Sidekiq, is pretty clever and elegant though. The idea is to patch Exception class with ignore flag:

# config/initializers/exceptions.rb
class Exception
  attr_accessor :ignore_please
end

And fully relies on the Sidekiq retry mechanism:

class HardJob
  include Sidekiq::Job

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  rescue AnnoyingError => e
    # flag it to be ignored
    e.ignore_please = true
    # re-raise it so Sidekiq will retry
    raise e
  end
end

And ignore errors with a flag in the error service config:

# https://docs.rollbar.com/docs/ruby#section-ignoring-items
# config/initializers/rollbar.rb
handler = proc do |options|
  raise Rollbar::Ignore if options[:exception].ignore_please
end
Rollbar.configure do |config|
  config.before_process << handler
end

# https://docs.honeybadger.io/lib/ruby/getting-started/ignoring-errors/#ignore-programmatically
# config/initializers/honeybadger.rb
Honeybadger.configure do |config|
  config.before_notify do |notice|
    notice.halt! if notice.exception.ignore_please
  end
end

Pros:

• Easy to implement

• Works with almost all error trackers

Cons:

• There are no limits on ignore exceptions. Thus, if an error is raised several times in a row within one job, it will not be reported to the error tracker

• Not all trackers can filter errors dynamically, i.e. NewRelic can only filter by error class names, and implementing such a filter needs deep knowledge of tracker internals or request a feature from maintainers.

• requires custom code for a job - if there are a couple of errors it needs to alter each job #perform method

We can improve it a bit by using sidekiq_retries_exhausted within the job

class HardJob
  include Sidekiq::Job

  sidekiq_retries_exhausted do |job, ex|
    ex.ignore_please = false
    # ErrorTracker is a wrapper around you tracker
    ErrorTracker.notify(ex, "#{job['class']} #{job["jid"]} just died with error #{ex.message}.")
  end

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  rescue AnnoyingError => e
    # flag it to be ignored
    e.ignore_please = true
    # re-raise it so Sidekiq will retry
    raise e
  end
end

Or globally with death_handlers

# this goes in your initializer
Sidekiq.configure_server do |config|
  config.death_handlers << ->(job, ex) do
    if ex.ignore_please # checks that some error has been ignored before send it to avoid double notify
      ex.ignore_please = false
      ErrorTracker.notify(ex, "#{job['class']} #{job["jid"]} just died with error #{ex.message}.")
    end
  end
end

With this improvement, we can see that some jobs have been retried too many times, have gone to the dead queue, and there are probably some bugs worth checking.

#2 Technique: Retry once before erroring

The second technique, by Mike Coutermarsh, is more complex, but provides exactly one retry and can be illustrated with this code example

class HardJob
  include Sidekiq::Job

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  rescue AnnoyingError => e
    # here we are retrying the job only once without notifying error tracker
    # and then if error will appear again it will be reported
    retry_once_before_raising_error(e)
  end
end

You can go through Mike's article to see how it's implemented in detail but I will provide here some ideas to have a clear picture of the final solution.

First of all, it needs to introduce an attribute accessor to the job instance:

class ApplicationJob
  include Sidekiq::Job
  attr_writer :retry_count

  # Job instances doesn't have direct access to the job payload
  # so we have to implement accessor
  def retry_count
    @retry_count ||= 0
  end
end

Then introduce custom server middleware and add it to the sidekiq config. It needs to be done this way because we have to fetch the retry counter and forward this data to the job instance.

module SidekiqMiddleware
  class RetryCount
    def call(job_instance, job_payload, _queue, &block)
      if job_instance.respond_to?(:retry_count)
        # assign retry count to the job instance to have access there
        job_instance.retry_count = job_payload.fetch("retry_count", 0)
      end

      yield
    end
  end
end

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add(SidekiqMiddleware::RetryCount)
  end
end

Implement a #retry_once_before_raising_error method:

class ApplicationJob
  # omit previous code for clarity
  RetryError = Class.new(StandardError)

  def retry_once_before_raising_error(exception)
    if retry_count < 1
      raise RetryError, exception.message
    else
      raise exception
    end
  end
end

And add RetryError to the tracker's ignore list:

# https://docs.rollbar.com/docs/ruby#section-exception-level-filters
# config/initializers/rollbar.rb
Rollbar.configure do |config|
  config.exception_level_filters.merge!({
    'ApplicationJob::RetryError' => 'ignore',
  })
end

# https://docs.honeybadger.io/lib/ruby/getting-started/ignoring-errors/#ignore-by-class
# config/honeybadger.yml
 exceptions:
  ignore:
    - 'ApplicationJob::RetryError'

Pros:

• provides an ability to retry once some flacky error and report it if it occurs next time. When the situation goes out of control the error will occur in the error tracker and the developer will see it

• easier to reuse by adding only one ApplicationJob::RetryError the exception to the ignore list of tracker

Cons:

• more complex to implement due to custom server middleware and altered job class

• it retries only once. In some cases, it needs to retry one or two times

• needs to alter #perform method of each job with such an error

Let's alter the last technique to have an option to retry several times

class HardJob
  include Sidekiq::Job

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  rescue AnnoyingError => e
    # here we are retrying the job limit times without notifying error tracker
    retry_before_raising_error(e, limit: 3)
  end
end

The only thing we need to change is the #retry_before_raising_error method

class ApplicationJob
  # omit previous code for clarity
  RetryError = Class.new(StandardError)

  def retry_before_raising_error(exception, limit: 1)
    if retry_count < limit
      raise RetryError, exception.message
    else
      raise exception
    end
  end
end

Pros:

• all pros from technique #2

• can retry as many times as it needs to

Cons:

• more complex to implement due to custom server middleware and altered job class

• needs to alter #perform method of each job with such an error

Generally, it's good enough for most of the cases, but what if we need some customized delay for this specific error? Then we can use sidekiq_retry_in block!

class HardJob
  include Sidekiq::Job

  sidekiq_retry_in do |count, exception, _jobhash|
    case exception
    when AnnoyingError
      10 * (count + 1) # (10, 20, 30, 40, 50)
    end
  end

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(10) == 0 # simulate error that occurs time to time
  rescue AnnoyingError => e
    # here we are retrying the job limit times without notifying error tracker
    retry_before_raising_error(e, limit: 3)
  end
end

Pros:

• all pros from technique #2

• can retry as many times as it needs to

• can have a custom delay

Cons:

• the final solution is quite complex - a lot of moving parts

#4 Technique: Use the sidekiq-rescue gem

The easiest, in my opinion, way to solve the problem is to use sidekiq-rescue gem. It's a tiny plugin, with zero dependency (besides Sidekiq itself!) that provides handy DSL and is very easy to set up.

Install the gem to your project

bundle add sidekiq-rescue && bundle install

Add the middleware to your Sidekiq configuration:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Sidekiq::Rescue::ServerMiddleware
  end
end

Let's consider our example:

class HardJob
  include Sidekiq::Job

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  end
end

All we need it's to include Sidekiq::Rescue::DSL module and use sidekiq_rescue

class HardJob
  include Sidekiq::Job
  include Sidekiq::Rescue::Dsl

  sidekiq_rescue AnnoyingError

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  end
end

And that's all! You can configure the number of retries and the delay (in seconds) between retries:

class HardJob
  include Sidekiq::Job
  include Sidekiq::Rescue::Dsl

  sidekiq_rescue AnnoyingError, delay: 60, limit: 5

  def perform(...)
    # do some important stuff
    raise AnnoyingError if rand(99) == 0 # simulate error that occurs time to time
  end
end

The delay is not the exact time between retries; it's a minimum delay. The actual delay is calculated based on the retries counter and delay value. The formula is delay + retries * rand(10) seconds. Randomization is used to avoid retry storms.
The default values are:

• delay: 60 seconds

• limit: 5 retries

Delay and limit can be configured globally:

# config/initializers/sidekiq.rb
Sidekiq::Rescue.configure do |config|
  config.delay = 65
  config.limit = 10
end

You can also configure a job to have the delay to be a proc:

sidekiq_rescue ExpectedError, delay: ->(counter) { counter * 60 }

Under the hood, this gem uses Sidekiq's perform_at and doesn't rely on the standard retry mechanism. Thus you have independent retry strategies for both types of errors: unexpected and expected.

Pros:

• easy to use

• independent retry strategy from the default Sidekiq error handling

• can retry as many times as it needs to

• can have a custom delay, even with proc

Cons:

• one more gem in a Gemfile

This gem is still under active development, but it has good coverage and has been tested in production.

As an author, I will be very grateful for any response about issues and PR's

Conclusion

In conclusion, error handling in Sidekiq is a critical aspect of software development. While the default retry mechanism works well for unexpected errors, handling expected errors requires additional strategies.

Techniques such as ignoring errors, retrying once before erroring, or using the sidekiq-rescue gem can be employed to handle these errors more effectively. Each method has its pros and cons, and the choice depends on the specific needs of your application.

By mastering these techniques, developers can significantly improve error observability and handling in their Sidekiq jobs, leading to more robust and reliable applications.

Sources:

• https://www.mikeperham.com/2013/08/25/please-use-an-error-service/

• https://github.com/sidekiq/sidekiq/wiki/Error-Handling

• https://github.com/leandromoreira/redlock-rb

• https://www.mikeperham.com/2017/09/29/retries-and-exceptions/

• https://www.mikecoutermarsh.com/silencing-errors-from-noisy-sidekiq-jobs/

• https://github.com/moofkit/sidekiq-rescue