Ruby Meditation 30


2020-10-24 presentations πŸ’Ž

On Saturday, October. 24, 2020 I presented at the Ruby Meditation conference.

slides


title: Developing with Docker author: gitlab.com/xlgmokha/developing-with-docker date: 2020-10-24 β€”


      -----------------
     (                 )
    ( A whale of a tale )
     (                 )
      -----------------
             \
              \
               \              ##         .
                \       ## ## ##        ==
                 \  ## ## ## ## ##    ===
                /"""""""""""""""""\___/ ===
               {                       /  ===-
                \______ O           __/
                  \    \         __/
                    \____\_______/


    Mo Khan | Software Developer | https://www.mokhan.ca/

$ history

Mo Khan Software Developer Calgary, AB, Canada.
 7  GitLab       --type=dev-tools
 6  Cisco        --type=security-product
 5  Uppercut     --type=agency
 4  ARC          --type=information-systems
 3  eCompliance  --type=startup
 2  ThoughtWorks --type=consulting
 1  MediaLogic   --type=agency
 0  DataShapers  --type=startup

License scanning at GitLab

       ***********************
       < Show me the licenses >
       ***********************
              \
               \
                \
                              ##        .
                        ## ## ##       ==
                    ## ## ## ##      ===
                /""""""""""""""""___/ ===
           ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
                \______ o          __/
                  \    \        __/
                    \____\______/

Name Version Package Manager License
@rails/actioncable 6.0.3-3 Yarn MIT
@rails/ujs 6.0.3-2 Yarn MIT
@yarnpkg/lockfile 1.1.0 Yarn BSD-2-Clause
addressable 2.7.0 Bundler Apache-2.0
bcrypt 3.1.12 Bundler MIT
cbor 0.5.9.6 Bundler Apache-2.0
device_detector 1.0.0 Bundler LGPL-3.0
devise 4.7.3 Bundler MIT
diffy 3.3.0 Bundler MIT
docutils 0.13.1 Pipenv BSD-2-Clause
elasticsearch 6.8.2 Bundler Apache-2.0
eventmachine 1.2.7 Bundler Ruby AND GPL-2.0
ffi 1.13.1 Bundler BSD-3-Clause
ffi-compiler 1.0.1 Bundler Apache-2.0
jmespath 1.4.0 Bundler Apache-2.0
kgio 2.11.3 Bundler LGPL-2.1+
launchy 2.4.3 Bundler ISC
msgpack 1.3.3 Bundler Apache-2.0
nokogumbo 2.0.2 Bundler Apache-2.0
rails 6.0.3.3 Bundler MIT
vue 2.6.12 Yarn MIT

Constraints

  • supports multiple versions of:
    • Dotnet Core
    • Golang
    • Java
    • NodeJS
    • PHP
    • Python
    • Ruby
  • supports multiple package managers:
    • Bundler
    • pip
    • Pipenv
    • Gradle
    • Maven
  • includes system packages for common libraries:
    • libpq-dev
    • libsqlite3-dev
  • works in limited connectivity environments
       **************************************
       < Must be deployed as a Docker image >
       **************************************
            \
             \
              \
                            ##        .
                      ## ## ##       ==
                  ## ## ## ##      ===
              /""""""""""""""""___/ ===
         ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
              \______ o          __/
                \    \        __/
                  \____\______/

Constraints - πŸ€”

We need to build a Docker image that can:

  • scan any codebase
  • install the version of tools needed
  • install the project dependencies
  • be small enough to fit on a CD-ROM
                      ---------------
                    < Are you joking? >
                      ---------------

          .---.
         /o   o\
      __(=  "  =)__
       //\'-=-'/\\
          )   (_
         /      `"=-._
        /       \     ``"=.
       /  /   \  \         `=..--.
   ___/  /     \  \___      _,  , `\
  `-----' `""""`'-----``"""`  \  \_/

How does this work?

  -------------
  |    git    |
  -------------     ----------------
  | main*     | --> | gitlab-runner |
  | feature-a |     ----------------
  | feature-b |            |
  -------------     launch container
                           |
                           V
                    ---------------
                    |             | <-----------------
                    | Docker Host | ---------        |
                    |             |         |        |
                    ---------------         |        |
                           |             download    |
                           V                |        |
                    ⭐⭐⭐⭐⭐⭐⭐⭐        V        |
                    ⭐  License   ⭐   ------------  |
                    ⭐  Scanner   ⭐   | registry | -|
                    ⭐⭐⭐⭐⭐⭐⭐⭐   ------------
                           |
                     publish report
                           |
                           V
                    ----------------
                    | gitlab-rails |
                    ----------------

v1.0

  1. Scan project for lock files (Gemfile.lock, Pipfile.lock etc)
  2. Install project tools (Ruby 2.7.2, Python 3.8.4)
  3. Install project dependencies (Rails, Django)
  4. Scan for licenses
  5. Export JSON report
                      ************
                      | APPLAUSE |
                      ************
Name Version Package Manager License
@rails/actioncable 6.0.3-3 Yarn MIT
@rails/ujs 6.0.3-2 Yarn MIT
@yarnpkg/lockfile 1.1.0 Yarn BSD-2-Clause
addressable 2.7.0 Bundler Apache-2.0
bcrypt 3.1.12 Bundler MIT
cbor 0.5.9.6 Bundler Apache-2.0
device_detector 1.0.0 Bundler LGPL-3.0
                ---------------------------
              < Yay! We can detect licenses >
                ---------------------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/

Why so slow?

 _____________________________________
< That's cool, but why is it so slow? >
 -------------------------------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

Overview

      ---------------
      |             | <-----------------
      | Docker Host | ---------        |
      |             |         |        |
      ---------------         |        |
             |             download    |
             V                |        |
      ⭐⭐⭐⭐⭐⭐⭐⭐        V        |
      ⭐  License   ⭐   ------------  |
      ⭐  Scanner   ⭐   | registry | -|
      ⭐⭐⭐⭐⭐⭐⭐⭐   ------------
             |
        publish report
             |
             V
      ----------------
      | gitlab-rails |
      ----------------

What happens if the Docker image is too big?

  • slow downloads
  • more disk space is required
  • more bandwidth is consumed
                      ---------------
                    < That's not good >
                      ---------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/

Overview

      ---------------
      |             | <------ 10GB -----
      | Docker Host | ---------        |
      |             |         |        |
      ---------------         |        |
             |             download    |
             V                |        |
      ⭐⭐⭐⭐⭐⭐⭐⭐        V        |
      ⭐  License   ⭐   ------------  |
      ⭐  Scanner   ⭐   | registry | -|
      ⭐⭐⭐⭐⭐⭐⭐⭐   ------------
            --------------------------------------
          < Yikes! It takes 6 minutes to download >
            --------------------------------------

        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/

Docker 101

  • Definitions
  • Build
  • Analyze
  • Optimize
         ------------------
        < What's a Docker? >
         ------------------
            \
             \
              \
                            ##        .
                      ## ## ##       ==
                  ## ## ## ##      ===
              /""""""""""""""""___/ ===
          ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
              \______ o          __/
                \    \        __/
                  \____\______/

Definitions

Ruby Docker
Class Image
Object Container
class Person
  attr_reader :name

  def initialize(name)
    @name = name
  end

  def hug(other)
    puts "#{name} πŸ€— #{other.name}"
  end
end

mo = Person.new("mo")
me = Person.new("me")

mo.hugs(me) # Ewww...

Definitions - Registry

Registry: stores images and makes them available to others

This include metadata about images and blobs for each layer in the image.

For example:

  • https://registry-1.docker.io
  • https://registry.gitlab.com

Architecture

                      ----------
                      | Client |
                      ----------
                      | build  |
                      | pull   |
                      | run    |
                      ----------
                          |
                          V
                    ---------------
                    | Docker Host |
                    ---------------
                    | Daemon      |
                    | Containers  |
                    | Images      |
                    --------------
                          |  A
                          V  |
                     ------------
                     | Registry |
                     ------------
                     | Images   |
                     ------------

https://docs.docker.com/get-started/overview/#docker-architecture

$ docker run -it alpine:latest cat /etc/os-release

docker run -it alpine:latest cat /etc/os-release
  1. check if β€œalpine:latest” is on docker host
  2. download β€œalpine:latest” from registry to docker host
  3. start a container using the β€œalpine:latest” image

Dockerfile

Defines how to build a Docker image.

path: examples/001/Dockerfile
relative: true
lang: docker

https://docs.docker.com/engine/reference/builder/

docker build -t developing-with-docker:latest examples/001/

docker build --network=host -t developing-with-docker:latest examples/001/

docker run developing-with-docker:latest

docker run developing-with-docker:latest

Analysis

A docker image is made up of multiple layers. Each layer is a snapshot of the filesystem stored as an archive.

      -----------------      |      -----------------
      (    bread      )      |      | FROM          |
      -----------------      |      -----------------
      <  brie cheese  >      |      | RUN           |
      {  cranberries  }      |      | COPY          |
      [ sliced turkey ]      |      | RUN           |
      -----------------      |      -----------------
      (    bread      )      |      | ENTRYPOINT    |
      -----------------      |      -----------------
                    -------------------
                  < Yum. Let's dive in! >
                    -------------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/

Analysis - dive

path: examples/001/Dockerfile
relative: true
lang: docker
$ dive developing-with-docker:latest

β”‚ Layers β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
Size    Command
5.6 MB   FROM 5a96ef02e9cab83
  16 MB  apk add ruby
  40 B   #(nop) COPY file:253d38e67af26b201caf4e271248576ba6a7da
  40 B   chmod +x /usr/local/bin/hello

https://github.com/wagoodman/dive

dive - layer details

┃ ● Current Layer Contents ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Permission     UID:GID       Size  Filetree
drwxr-xr-x         0:0     841 kB  β”œβ”€βŠ• bin
drwxr-xr-x         0:0        0 B  β”œβ”€β”€ dev
drwxr-xr-x         0:0     383 kB  β”œβ”€βŠ• etc
drwxr-xr-x         0:0        0 B  β”œβ”€β”€ home
drwxr-xr-x         0:0     3.9 MB  β”œβ”€βŠ• lib
drwxr-xr-x         0:0        0 B  β”œβ”€βŠ• media
drwxr-xr-x         0:0        0 B  β”œβ”€β”€ mnt
drwxr-xr-x         0:0        0 B  β”œβ”€β”€ opt
dr-xr-xr-x         0:0        0 B  β”œβ”€β”€ proc
drwx------         0:0        0 B  β”œβ”€β”€ root
drwxr-xr-x         0:0        0 B  β”œβ”€β”€ run
drwxr-xr-x         0:0     226 kB  β”œβ”€βŠ• sbin
drwxr-xr-x         0:0        0 B  β”œβ”€β”€ srv
drwxr-xr-x         0:0        0 B  β”œβ”€β”€ sys
drwxrwxrwx         0:0        0 B  β”œβ”€β”€ tmp
drwxr-xr-x         0:0      14 MB  β”œβ”€βŠ• usr
drwxr-xr-x         0:0     1.8 MB  β””β”€βŠ• var

License scanner - v1.0

 __________________________________
/ How many layers does the license \
\ scanner image have?              /
 ----------------------------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

docker pull - v1.0

0a01a72a686c: Pull complete
cc899a5544da: Pull complete
19197c550755: Pull complete
716d454e56b6: Pull complete
4a00dd5b28fc: Pull complete
e039d25729bf: Pull complete
930b12354a74: Pull complete
5c0b45be82b1: Pull complete
2174b0b17785: Pull complete
79e1fdae2dfc: Extracting [================>   ]    322MB/396.1MB
2c46852653ff: Download complete
40e535af8764: Download complete
0c1954047133: Download complete
080be37ae17e: Download complete
4179fc4ef96a: Download complete
87d7bc66884f: Download complete
581b5a43e64d: Download complete
a754f2766934: Download complete
c1e41c3670fe: Download complete
e782eb3070d9: Download complete
50d1fb4b55b2: Download complete
276696690bcf: Download complete
5eec42d5363b: Download complete
2296aa2193e9: Download complete
5fe4c102c0bc: Download complete
97390612da81: Downloading [===========>       ]  81.05MB/174MB
311b1e270e29: Downloading [=====>             ]  42.12MB/189.4MB
53dfbd975f60: Downloading [=>                 ]  25.36MB/843.4MB
3dd2acdebe0f: Waiting
d548f098494f: Waiting
da1cc42017ff: Waiting
cfc3cd025ca9: Waiting
69ea647e6c07: Waiting
1e27d5f85aa2: Waiting
94cf5e06627d: Waiting
30e1f788589d: Waiting
d9238ec317d1: Waiting
e17797fa5e82: Waiting
9003b36c1e4e: Waiting

Layer cake

...
50d1fb4b55b2: Download complete
276696690bcf: Download complete
5eec42d5363b: Download complete
2296aa2193e9: Download complete
5fe4c102c0bc: Download complete
97390612da81: Downloading [===========>       ]  81.05MB/174MB
311b1e270e29: Downloading [=====>             ]  42.12MB/189.4MB
53dfbd975f60: Downloading [=>                 ]  25.36MB/843.4MB
3dd2acdebe0f: Waiting
d548f098494f: Waiting
da1cc42017ff: Waiting
cfc3cd025ca9: Waiting
69ea647e6c07: Waiting
1e27d5f85aa2: Waiting
94cf5e06627d: Waiting
30e1f788589d: Waiting
d9238ec317d1: Waiting
e17797fa5e82: Waiting
9003b36c1e4e: Waiting
                _______________________
                < That's a lot of layers! >
                -----------------------
                    \
                    \
                      \
                                    ##        .
                              ## ## ##       ==
                          ## ## ## ##      ===
                      /""""""""""""""""___/ ===
                 ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
                      \______ o          __/
                        \    \        __/
                          \____\______/

Start with a minimal base image

REPOSITORY                   TAG                 SIZE
debian                       stable-slim         69.2MB
licensefinder/license_finder 6.0.0               6.21GB

                  ------------------------------
                < Let's try a smaller base image >
                  ------------------------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
                             `-`
FROM licensefinder/license_finder:6.0.0

to

FROM debian:stable-slim

Be picky

Warning: The content below may be considered offensive.

RUN apt-get update && apt-get install -y \
  build-essential \
  curl \
  git-core \
  sudo \
  unzip \
  wget
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
    apt-get -y install nodejs
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - && \
  echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list && \
  apt-get update && \
  apt-get install yarn
RUN apt-get install -y python rebar
RUN apt-get install -y python-pip && \
    pip install --upgrade pip==$PIP_INSTALL_VERSION
RUN apt-get install -y locales
RUN wget https://packages.erlang-solutions.com/erlang-solutions_${MIX_VERSION}_all.deb && \
    sudo dpkg -i erlang-solutions_${MIX_VERSION}_all.deb && \
    sudo rm -f erlang-solutions_${MIX_VERSION}_all.deb && \
    sudo apt-get update && \
    sudo apt-get install -y esl-erlang && \
    sudo apt-get install -y elixir
RUN apt-get install -y python-dev && \
  pip install --ignore-installed six --ignore-installed colorama --ignore-installed requests --ignore-installed chardet --ignore-installed urllib3 --upgrade setuptools && \
  pip install conan
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF &&\
  echo "deb https://download.mono-project.com/repo/ubuntu stable-xenial main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list &&\
  apt-get update &&\
  apt-get install -y mono-complete &&\
  curl -o /usr/local/bin/nuget.exe https://dist.nuget.org/win-x86-commandline/latest/nuget.exe &&\
  echo "alias nuget=\"mono /usr/local/bin/nuget.exe\"" >> ~/.bash_aliases
RUN wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb &&\
  sudo dpkg -i packages-microsoft-prod.deb &&\
  sudo apt-get update &&\
  sudo apt-get install -y dotnet-runtime-2.1

Group β€˜apt-get install’ together

COPY config/install.sh /opt/install.sh
RUN bash /opt/install.sh
#!/bin/bash
apt-get install -y --no-install-recommends \
  apt-transport-https \
  autoconf \
  automake \
  bsdmainutils \
  bzip2 \
  ca-certificates \
  cmake \
  curl \
  gnupg2 \
  make \
  pkg-config \
  re2c \
  rebar \
  zstd

note: avoid installing build tools if you can

Deflate

Compress files that are large during build time.

#!/bin/bash

function deflate() {
  local file=$1
  local dir=$2
  local zstd_command="/usr/bin/zstd -19 -T0"
  tar --use-compress-program "$zstd_command" -cf "$file" "$dir"
}

cd /opt
deflate /opt/asdf.tar.zst asdf

cd /usr/lib
deflate /usr/lib/gcc.tar.zst gcc
deflate /usr/lib/mono.tar.zst mono
deflate /usr/lib/rustlib.tar.zst rustlib

cd /usr/share
deflate /usr/share/dotnet.tar.zst dotnet

Inflate

Decompress files when the container is launched by hooking into the ENTRYPOINT.

ENTRYPOINT ["/run.sh"]
#!/bin/bash

function inflate() {
  local file=$1
  local to_dir=$2
  if [ -f "$file" ]; then
    tar --use-compress-program zstd -xf "$file" -C "$to_dir"
    rm "$file"
  fi
}

inflate /opt/asdf.tar.zst /opt
inflate /usr/lib/gcc.tar.zst /usr/lib
inflate /usr/lib/mono.tar.zst /usr/lib
inflate /usr/lib/rustlib.tar.zst /usr/lib
inflate /usr/share/dotnet.tar.zst /usr/share

sh "$@"

Build specialized packages

build do
  env = with_standard_compiler_flags(with_embedded_path)
  configure_command = [
    "--disable-debug-env",
    "--disable-dtrace",
    "--disable-install-capi",
    "--disable-install-doc",
    "--disable-install-rdoc",
    "--disable-jit-support",
    "--enable-shared",
    "--prefix=#{install_dir}",
    "--with-out-ext=coverage,dbm,readline,rdoc,win32,win32ole,...",
    "--without-gdbm",
    "--without-gmp",
    "--without-jemalloc",
    "--without-tk",
    "--without-valgrind"
  ]
  configure(*configure_command, env: env)
  make "-j #{workers}", env: env
  make "-j #{workers} install", env: env
end
# ls -lh /opt/toolcache/ruby-* | awk '{ print $5 " " $9 }'
5.3M /opt/toolcache/ruby-2.4.10-1_amd64.deb
5.3M /opt/toolcache/ruby-2.4.5-1_amd64.deb
5.3M /opt/toolcache/ruby-2.4.9-1_amd64.deb
5.4M /opt/toolcache/ruby-2.5.8-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.0-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.1-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.2-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.3-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.4-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.5-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.6-1_amd64.deb
5.7M /opt/toolcache/ruby-2.7.0-1_amd64.deb
5.7M /opt/toolcache/ruby-2.7.1-1_amd64.deb
5.7M /opt/toolcache/ruby-2.7.2-1_amd64.deb

Results

Image Tag Size
license-scanner 3.28.1 1.4GB
license-scanner 2.8.0 9.83GB
license-scanner 1.5.0 4.06GB
                  ------------
                < 9.83GB to 1.4GB. Better! >
                  ------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/

Summary

  • Keep each layer small
  • Grouping install steps together
  • Cleanup transient artifacts in each layer
  • Deflate files at build time
  • Inflate files at run time
                  ----------
                < Thank you >
                  ----------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/

gitlab.com/xlgmokha/developing-with-docker