I've been wanting to set up continuous integration for my personal projects for a long time. It's one of those things that seems super cool and useful but a bit too nebulous for me to actually sit down and devote any time to it. This weekend I finally did it, and it was a lot harder than it should have been.

First, the documentation for setting up Bitbucket Pipelines with Ruby on Rails is either nonexistent or just kind of wrong. It seems not a lot of people have done it, or of they have, they just haven't written about it online. Anyway, this blog post is about what I learned while setting up Bitbucket Pipelines for one of my Ruby on Rails projects.

Example bitbucket-pipelines.yml for Ruby on Rails

Here's my bitbucket-pipelines.yml file for reference:

# bitbucket-pipelines.yml
# This is a sample build configuration for Ruby.
# Check our guides at https://confluence.atlassian.com/x/8r-5Mw for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: ruby:2.4.0

pipelines:
  default:
    - step:
        caches:
          - bundler
        script: # Modify the commands below to build your repository.
          - apt-get update && apt-get install -y awscli
          - aws s3 cp --region us-east-1 s3://pricer-pro-auto-scaling/pricer_pro/application.yml $BITBUCKET_CLONE_DIR/config/application.yml
          - bundle install --path vendor/bundle
          - RAILS_ENV=test bundle exec rails db:drop db:create db:migrate
          - rails test
        services: 
          - mysql

definitions:
  caches:
    bundler: vendor/bundle
  services: 
    mysql: 
      image: mysql 
      environment: 
        MYSQL_DATABASE: 'pricer_pro_test'
        MYSQL_ROOT_PASSWORD: 'root'

Getting Secrets into Bitbucket Pipelines with S3

You're not supposed to check your secrets into your repository, and Bitbucket Pipelines only has access to your repository. So how do you get your secrets into Bitbucket? Well, I used S3 because I already have my secrets on S3 for when I deploy my Ruby on Rails projects to my production servers. If you don't have your secrets on S3, sorry, I don't have any ideas for how you can securely get them into Pipelines.

So how did I do it? Well I used awscli to download my secrets from S3 into the right spot in my application's directory. In Pipelines, the environment variable $BITBUCKET_CLONE_DIR tells you where your repository is cloned into the docker image, so I used the line below to set that up:

aws s3 cp --region us-east-1 s3://pricer-pro-auto-scaling/pricer_pro/application.yml $BITBUCKET_CLONE_DIR/config/application.yml

The next problem that popped up with doing it this way is that awscli needs my AWS credentials in order to download anything from my private S3 bucket. You can provide your credentials to awscli with two environment variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Add those two environment variables in your repository's settings like this:

Adding secured environment variables to Bitbucket Pipelines is super easy once you find the right page.

Make sure that you check the "Secured" checkbox!

Using utf8mb4 Encoding with MySQL on Bitbucket Pipelines

The next problem I had was with using utfmb4 encoding on MySQL. I kept getting the error "ActiveRecord::StatementInvalid: Mysql2::Error: Incorrect string value..." when my rails tests ran. I thought this was weird because my migrations specified utf8mb4 so I could support emojis and other 4-byte Unicode characters.

It turns out that fixing this was somewhat easy. Out of the box, Bitbucket Pipelines creates your database table for you. The problem is that it chooses the encoding of that table for you, and I couldn't find any documentation about how to change the default encoding. To fix this, I just had to have rails drop, create, and migrate my test database itself:

RAILS_ENV=test bundle exec rails db:drop db:create db:migrate

Caching bundler Gems with Bitbucket Pipelines

Finally, Bitbucket Pipelines has a cool caching feature that's supposed to speed up things like running bundle install. Unfortunately, the documentation for getting Pipelines caching working with Ruby on Rails is a bit odd:

The caching documentation for bundler on Bitbucket Pipelines is a bit inconsistent.

The example YAML file runs bundle install but then they have a note that says you should run bundle install --path vendor/bundle. Which is it? It's the one with --path in it.

Here's the important stuff about caching in Bitbucket Pipelines:

• A cache is saved only when the cache doesn't already exist, the step that it is used in completes successfully, and the cache is smaller than 1 GB in size after being compressed.
• Caches are deleted automatically when they're one week old.
• You can delete caches manually by clicking the "Caches" button at the top left of the Pipelines page for one of your Bitbucket repositories.

For me, caching my bundled gems sped up my build time by about 2 minutes:

Two Bitbucket Pipelines builds of one of my Ruby on Rails projects at the same commit (2300561). Build #15 was uncached and took 7 minutes and 38 seconds to complete. Build #16 was cached and took 5 minutes and 35 seconds to complete.

Summary

So, I had three big road blocks to getting Bitbucket Pipelines working with Ruby on Rails: 1) getting my application secrets into the Pipelines build securely, 2) making MySQL use the correct character encoding, and 3) caching my bundled gems for faster build times.

I hope this information is useful for you. If it is, or if you have any questions, let me know in the comments below!

Photo by Max Smith