Kamal MustafaSome gotcha we got into while upgrading an old system build since Django-1.0, which then gradually upgraded to 1.4, 1.8 and finally now to 1.11 (the last 1.x version). For notes, database migrations framework only landed in Django-1.7. In this new migrations framework, you have to explicitly generate a migration files. Prior to that, the syncdb command will generate tables (which doesn't exists) on the fly based on your model definition. Subsequent changes to the table, like adding new column has to be handled manually through external means.
Up until 1.8, having explicit migrations files not yet mandatory so the migrate command basically exhibit same behavior as the old syncdb. But since we're updating to 1.11, we have to create the migrations files for all the apps in the system. This initial migrations basically involved creating the tables when we run migrate command and since all the migrations not yet tracked, it will try to apply it which obviously will fail because all the tables already exists.
For that reason, django provide 2 options:-
--fake¶
Tells Django to mark the migrations as having been applied or unapplied, but
without actually running the SQL to change your database schema.
This is intended for advanced users to manipulate the current migration state
directly if they’re manually applying changes; be warned that using --fake runs the
risk of putting the migration state table into a state where manual recovery will
be needed to make migrations run correctly.
--fake-initial¶
Allows Django to skip an app’s initial migration if all database tables with the
names of all models created by all CreateModel operations in that migration already
exist. This option is intended for use when first running migrations against a
database that preexisted the use of migrations. This option does not, however,
check for matching database schema beyond matching table names and so is only safe
to use if you are confident that your existing schema matches what is recorded in
your initial migration.
We decided to use --fake-initial for the the migrate. But we got error table already exists. It doesn't make sense at first, as we can read above it will detect if table already exists and then just "fake apply" the migrations. And we know for sure the table really exists, so why Django still want to create that table ?
I stepped through the migrations code in django/db/migrations/executor.py and put a pdb.set_trace() in this part:-
# Make sure all create model and add field operations are done
for operation in migration.operations:
import pdb;pdb.set_trace()
if isinstance(operation, migrations.CreateModel):
model = apps.get_model(migration.app_label, operation.name)
if model._meta.swapped:
Stepping through this and fortunately the error happened pretty early, I can see it correctly detected the table exists:-
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(337)detect_soft_applied()
-> if should_skip_detecting_model(migration, model):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(339)detect_soft_applied()
-> if model._meta.db_table not in existing_table_names:
(Pdb) l
334 # We have to fetch the model to test with from the
335 # main app cache, as it's not a direct dependency.
336 model = global_apps.get_model(model._meta.swapped)
337 if should_skip_detecting_model(migration, model):
338 continue
339 -> if model._meta.db_table not in existing_table_names:
340 return False, project_state
341 found_create_model_migration = True
342 elif isinstance(operation, migrations.AddField):
343 model = apps.get_model(migration.app_label, operation.model_name)
344 if model._meta.swapped:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(341)detect_soft_applied()
-> found_create_model_migration = True
Until it came to this iteration:-
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(330)detect_soft_applied()
-> import pdb;pdb.set_trace()
(Pdb) operation.name
'Image'
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(331)detect_soft_applied()
-> if isinstance(operation, migrations.CreateModel):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(332)detect_soft_applied()
-> model = apps.get_model(migration.app_label, operation.name)
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(333)detect_soft_applied()
-> if model._meta.swapped:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(337)detect_soft_applied()
-> if should_skip_detecting_model(migration, model):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(339)detect_soft_applied()
-> if model._meta.db_table not in existing_table_names:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(340)detect_soft_applied()
-> return False, project_state
So why it detected the table as not exists ? Inspecting the variable existing_table_names finally gave me a light. The table indeed does not exists !
(Pdb) model._meta.db_table not in existing_table_names
True
(Pdb) model._meta.db_table
'blog_image'
(Pdb) existing_table_names
['auth_group', 'auth_group_permissions', 'auth_permission', 'auth_user', 'auth_user_groups', 'auth_user_user_permissions', 'blog_article', 'django_admin_log', 'django_content_type', 'django_migrations', 'django_session']
So what happened was that it detecting the table as a group, so if there's one table does not exists, then the whole of that particular migrations will be applied, which mean also creating table that already exists. In our case, this happened because when dumping the production database to test this migrations, we exclude some tables with large data to keep the dump small. Which mean the resulting structure not really matching what described in the migrations file, which already being warned in the documentation above, sigh.
