Whenever we have a choice to make for a CI system to use on a project, we pick GitHub Actions, mainly for convenience. Our code is already hosted on GitHub, and it doesn’t make sense to introduce other tools unnecessarily, so some time ago we started using GitHub Actions as our CI provider.
Over the years we’ve been constantly improving our development workflows thereby adding more complexity to our CI. There were many steps in our pipeline for various code checks, Elixir tests, etc., each increasing the time we needed to wait to make sure our code was good to go. So we’d wait 5 to 10 minutes or so just to find out the code wasn’t formatted properly, there was some compiler warning or something trivial as that. We knew there were better ways to set up our CI, but we felt separating the workflows into separate jobs was going to make for a harder-to-maintain code because GitHub Actions does not support full YAML syntax.
I came to Elixir from the Ruby on Rails community where YAML is a default for any kind of configuration, so I was excited to see GitHub Actions using YAML for the workflow definitions. I quickly came to realize it’s not the same YAML I was used to (You’ve changed, bro). Specifically, I couldn’t use anchors which provide the ability to write reusable code in .yml files.
Script
Our way of working around this is writing workflow definitions in Elixir and translating them to YAML, letting us benefit from the beautiful Elixir syntax in sharing variables, steps, and jobs between workflows while still, as a result, having workflow files in the YAML format GitHub Actions supports.
To convert the workflow definitions from Elixir to YAML, we wrote a CLI script that uses fast_yaml library with a small amount of code wrapping it up in an easy-to-use package. We used this script internally for years, but now we’ve decided to share it with the community.
I’ve had some trouble distributing the script. Usually, we’d execute .exs script to convert the workflow, so I wanted to build an escript, but the fast_yaml library contains NIFs that don’t work with it. I liked the way Phoenix is installed so I tried adopting that approach, creating a mix project containing a task, then archiving the project into a .ez file, only to find out that when it gets installed, it doesn’t contain any dependencies. This can be alleviated using burrito or bakeware, but they introduce more complexity, and I didn’t like the way error messages were displayed in the Terminal, so I ended up with a hex package that’s added to an Elixir project in a usual way. Ultimately, I didn’t plan to use the script outside of Elixir projects, so that’s a compromise I was willing to make. If at a later point I feel the need, I’ll distribute it some other way, which will deserve another blog post.
Usage
Anyway, here’s how you can use this mix task. Add the github_workflows_generator package as a dependency to your mix.exs file:
defp deps do
[
{:github_workflows_generator, "~> 0.1"}
]
end
You most likely don’t want to use it in runtime and environments other than dev, so you might find this more appropriate:
defp deps do
[
{:github_workflows_generator, "~> 0.1", only: :dev, runtime: false}
]
end
That will let you execute
mix github_workflows.generate
command that given a .github/github_workflows.ex file like this one:
defmodule GithubWorkflows do
def get do
%{
"main.yml" => main_workflow(),
"pr.yml" => pr_workflow()
}
end
defp main_workflow do
[
[
name: "Main",
on: [
push: [
branches: ["main"]
]
],
jobs: [
test: test_job(),
deploy: [
name: "Deploy",
needs: :test,
steps: [
checkout_step(),
[
name: "Deploy",
run: "make deploy"
]
]
]
]
]
]
end
defp pr_workflow do
[
[
name: "PR",
on: [
pull_request: [
branches: ["main"]
]
],
jobs: [
test: test_job()
]
]
]
end
defp test_job do
[
name: "Test",
steps: [
checkout_step(),
[
name: "Run tests",
run: "make test"
]
]
]
end
defp checkout_step do
[
name: "Checkout",
uses: "actions/checkout@v4"
]
end
end
creates multiple files in the .github/workflows directory.
main.yml
name: Main
on:
push:
branches:
- main
jobs:
test:
name: Test
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Run tests
run: make test
deploy:
name: Deploy
needs: test
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy
run: make deploy
pr.yml
name: PR
on:
pull_request:
branches:
- main
jobs:
test:
name: Test
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Run tests
run: make test
Path to the source file and the output directory can be customized. To see available options, run
mix help github_workflows.generate
You might also want to read the documentation or check out the source code.
The generator’s repo contains its own CI workflows (something something-ception) that show how useful the command is in complex scenarios.
defmodule GithubWorkflows do
@moduledoc false
def get do
%{
"ci.yml" => ci_workflow()
}
end
defp ci_workflow do
[
[
name: "CI",
on: [
pull_request: [],
push: [
branches: ["main"]
]
],
jobs: [
compile: compile_job(),
credo: credo_job(),
deps_audit: deps_audit_job(),
dialyzer: dialyzer_job(),
format: format_job(),
hex_audit: hex_audit_job(),
prettier: prettier_job(),
test: test_job(),
unused_deps: unused_deps_job()
]
]
]
end
defp compile_job do
elixir_job("Install deps and compile",
steps: [
[
name: "Install Elixir dependencies",
env: [MIX_ENV: "test"],
run: "mix deps.get"
],
[
name: "Compile",
env: [MIX_ENV: "test"],
run: "mix compile"
]
]
)
end
defp credo_job do
elixir_job("Credo",
needs: :compile,
steps: [
[
name: "Check code style",
env: [MIX_ENV: "test"],
run: "mix credo --strict"
]
]
)
end
# Removed for brevity
# ...
defp elixir_job(name, opts) do
needs = Keyword.get(opts, :needs)
steps = Keyword.get(opts, :steps, [])
job = [
name: name,
"runs-on": "${{ matrix.versions.runner-image }}",
strategy: [
"fail-fast": false,
matrix: [
versions: [
%{
elixir: "1.11",
otp: "21.3",
"runner-image": "ubuntu-20.04"
},
%{
elixir: "1.16",
otp: "26.2",
"runner-image": "ubuntu-latest"
}
]
]
],
steps:
[
checkout_step(),
[
name: "Set up Elixir",
uses: "erlef/setup-beam@v1",
with: [
"elixir-version": "${{ matrix.versions.elixir }}",
"otp-version": "${{ matrix.versions.otp }}"
]
],
[
uses: "actions/cache@v3",
with:
[
path: ~S"""
_build
deps
"""
] ++ cache_opts(prefix: "mix-${{ matrix.versions.runner-image }}")
]
] ++ steps
]
if needs do
Keyword.put(job, :needs, needs)
else
job
end
end
# Removed for brevity
# ...
end
That creates a YAML file I wouldn’t want to look at, much less maintain it, but enables us to have this CI pipeline
Our phx.tools project has an even better example with 3 different workflows.
Let’s step back to see how the script works.
The only rule that we enforce is that the source file must contain a GithubWorkflows module with a get/0 function that returns a map of workflows in which keys are filenames and values are workflow definitions.
defmodule GithubWorkflows do
def get do
%{
"ci.yml" => [[
name: "Main",
on: [
push: []
],
jobs: []
]]
}
end
end
Everything else is up to you.
When you look at the generated .yml files, they might not look exactly the same as if you wrote them by hand.
For example, if you were to add caching for Elixir dependencies as in actions/cache code samples, you’d want to have this YAML code:
- uses: actions/cache@v3
with:
path: |
deps
_build
key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}
restore-keys: |
${{ runner.os }}-mix-
with two paths passed without quotes.
I haven’t found a way to tell the YAML encoder to format it that way, so my workaround is to use a sigil that preserves the newline, so that
[
uses: "actions/cache@v3",
with:
[
path: ~S"""
_build
deps
"""
],
key: "${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}"
restore-keys: ~S"""
${{ runner.os }}-mix-
"""
]
gets converted to
- uses: actions/cache@v3
with:
path: "_build\ndeps"
key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}
restore-keys: ${{ runner.os }}-mix-
Elixir DevOps series
In our workflows, you may notice some new ideas not seen elsewhere, so be sure to look out for more posts on our blog in a new series where we’ll unpack our unique DevOps practices. If you have any questions, you can contact us at blog@optimum.ba and we’ll try to answer them in our future posts.
If our approach to software development resonates with you and you're ready to kickstart your project, drop us an email at projects@optimum.ba. Share your project requirements and budget, and we'll promptly conduct a review. We'll then schedule a call to dive deeper into your needs. Let's bring your vision to life!
This was the first post from our Elixir DevOps series.
