Building Docker images from private GitHub repos

Docker has a nice build context feature that lets you specify a git repository URL as the location to build an image from. So, for instance, you can roll your own Redis image like this: docker build -t myredis:5 https://github.com/docker-library/redis.git#master:5.0 But…

Configuring Airflow DAGs with YAML

The Apache Airflow UI is nice to look at, but it's a pretty clunky way to manage your pipeline configuration. One alternative is to store your DAG configuration in YAML and use it to set the default configuration in the Airflow database when the DAG is first run.…