Running Django Migrations with ECS
A few months ago, my coworkers and I decided that maintaining our massive monolith Django app for our app’s API was too much hassle. We had just brought in some more developers, and trying to help them understand what was happening in that codebase was more work than doing the extra coding ourselves. We decided something had to change.
It took us some time, but we slowly began migrating the individual pieces of that large Django projects into smaller pieces… migrating our users to AWS Cognito, allowing API Gateway to handle authentication, separating features A from B, etc. But once it came time to start testing a few of these in a staging environment with all the individual pieces in place, we came across a bug that I have been spending the last few weeks trying to solve…
How do we run migrations on the database???
This was such a simple problem to solve in our Elastic Beanstalk environment thanks to a few .ebextensions config files. However in ECS there is no solution as elegant. So I began testing some options and finding some answers online. The answers I thought of myself were proven to be suboptimal after only a few minutes on Google. The best answer I found was from Adam Stepinski (his article can found here). While this piece is great for explaining the architecture to use, it was very architectural for someone like me who was brand new to ECS at the time. I needed to see something a little more concrete for me to understand how to implement this process.
I’m not going to pretend like my solution is the most elegant or cost-effective or secure… but it works for me, so maybe it’ll work for you.
And a quick side note: I automated all of these AWS resources using CloudFormation. There’s a lot of moving parts here, and I knew I’d forget half the pieces over a weekend. So if it seems like a lot to manage to create one service, I agree.
ECR & the Dockerfiles
The first thing I did was create a separate Dockerfile in the root directory of our Django apps, where the only difference from the image that runs our services is the CMD at the end of the file. Instead of running CMD ["gunicorn", "--bind", "0.0.0.0:8888", "project.wsgi"] the Migration Dockerfile runs CMD ["python", "manage.py", "migrate"] . I also created a new ECR repository to hold the migration images.
I’m positive that this process can be accomplished much easier by using something like Docker Compose; but like I said, elegance was not my goal — only migrations.
ECS Task Definitions
Just how you would create a Task Definition for any service to run in an ECS Cluster, we’re also going to create a task definition for our migrations. The key thing to remember here is that this task won’t be running alongside your regular service tasks — it’s only here to run one command in your app and then shut down; so more than likely you can reduce the size of the container it runs in to save on costs (although again, it will run for such a short time that it probably won’t matter much).
CodePipeline
AWS CodePipeline (or whichever CI/CD tool you prefer) is key to this process. I use CodeBuild to build and push images to both the primary and migration ECR repositories, which I highly recommend no matter how you go about running the actual migrations.
The Build operation in your pipeline is where the magic is going to happen. After pushing the images to ECR, we’re going to run the AWS CLI run-task command to perform the migrations.
Here is the buildspec.yml file I use for migrating our services (keep in mind that I also use CodeBuild’s environment variables quite liberally as I have automated this entire process using CloudFormation)
version: 0.2phases:pre_build:commands:- echo Logging in to Amazon ECR...- aws --version- $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)# define repository variables- SERVICE_URI=service/$SERVICE_NAME- MIGRATION_URI=migration/$SERVICE_NAME- REPOSITORY_URI=$ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/$SERVICE_URI- MIGRATION_REPO_URI=$ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/$MIGRATION_URI- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)- IMAGE_TAG=build-$(echo $CODEBUILD_BUILD_ID | awk -F":" '{print $2}')build:commands:# Create the repositories- echo Creating repositories- aws ecr create-repository --repository-name $SERVICE_URI || true- aws ecr create-repository --repository-name $MIGRATION_URI || true# Building the base docker image- echo Building the Docker image...- docker build -t $REPOSITORY_URI:latest .- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
# Building the migration image- echo Building the Migration image...- docker build -f MigrationDockerfile -t $MIGRATION_REPO_URI:latest .- docker tag $MIGRATION_REPO_URI:latest $MIGRATION_REPO_URI:$IMAGE_TAGpost_build:commands:# push the service image- echo Pushing the Docker images...- docker push $REPOSITORY_URI:latest- docker push $REPOSITORY_URI:$IMAGE_TAG# push the migration image- echo Pushing the Migration image...- docker push $MIGRATION_REPO_URI:latest- docker push $MIGRATION_REPO_URI:$IMAGE_TAG# Run the migration task- aws ecs run-task --cluster $CLUSTER_NAME --task-definition $MIGRATION_TASK_NAME --count 1 --launch-type FARGATE --network-configuration awsvpcConfiguration={subnets=[$SUBNETS],securityGroups=[$SECURITY_GROUPS],assignPublicIp=ENABLED} --platform-version LATEST- printf '[{"name":$SERVICE_NAME,"imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > imageDetail.jsonartifacts:files:- imageDetail.json
Of course you should change the run-task command to suite your needs and configurations. One thing to note from my experience: I had to enable the public IP address for the task because otherwise I was unable to download the image from ECR. If someone has a solution for that, I’d love to hear about it!
Conclusion
As I said, I know this is likely not the best solution to the migration problem, but it has served me well so far. If you have any suggestions on how to improve this process, please leave a comment below!
