tl;dr

As late as rspec-rails v7 vanilla transactional tests with use_transactional_fixtures = true have an unfortunate bug where quitting out of an example (binding.pry + !!!) does not end the example's transaction. This will interfere with any cleanup of records written to DB outside transactions in after(:all), after(:suite), and even at_exit. Consider using DatabaseCleaner-powered transactions instead, those close correctly.

The Story

I've been working on lowering the time it takes to run UPB system's CI. There are many little things one can do (caching CI dependencies, configuring parallel runners to use parallel_runtime_rspec.log correctly etc.), but one of the more impactful techniques tends to be using TestProf to identify any hotspots and addressing those.

In this case I identified that specs tend to require lots of project records to spec things in them. Many systems have these "everything else depends on these" records, like User.

A possible solution for this situation is to use AnyFixture to "seed" some generic records outside transactions, usually at suite start, and reuse them in transactional specs. The nice thing about this setup is that any changes to these "canonical" records will be rolled back at end of transactional specs, but the downside is that specs can no longer rely on the database being completely empty - .to change { Thing.count }.from(0).to(1) will have to be repaced with relative asserts .to change { Thing.count }.by(1).

Anywho, I prepared a MR and CI passed just fine, and we merged the changes, but starting work on new feature I encountered weird spec failures regarding uniqueness constraint violations - some records were there at suite start!

This had me baffled because I had made sure AnyFixture was cleaning itself up, but in some cases apparently it did not.

Several long hours of hair-pulling later I realized that the issue occurs if I quit out of an example. This is easy to reproduce:

Set up after-suite hooks

config.after(:all) do
  binding.pry
end

config.after(:suite) do
  binding.pry
end

at_exit do
  binding.pry
end

And hook into en example, before block also works:

before { binding.pry }

Run the example, and exit from the pry session in before hook normally (with exit), and run ActiveRecord::Base.connection.transaction_open? in all after hooks. It will be false.

Now run the example again, but this time quit out of the example with !!!. All after hooks will report a transaction being open.
Any cleanup in after hooks will occur in this still-open example transaction and will never got committed, leaving a dirty DB.

The workaround I've found for now is to turn vanilla transactionality off, and use DatabaseCleaner-powered transactions instead, those close correctly:

config.use_transactional_fixtures = false

DatabaseCleaner.strategy = :transaction

config.around do |example|
  DatabaseCleaner.cleaning { example.run }
end