Consul & Registrator & Rails Service all together in separate Docker container

1.Start a container running Consul:

Here we use this image: https://hub.docker.com/r/progrium/consul/

$ docker run -it -h node -p 8500:8500 -p 53:53/udp progrium/consul -server -bootstrap -advertise  $DOCKER_IP

 

How to check that Consul is running:

Method1:

In Docker-Machine, run

$ docker ps

Will see running Consul container:

3

We can also check the logs:

$ docker logs $CONTAINER_ID

3

Method2:

Browse to $DOCKER_IP:8500 there is a dashboard to see the services that are registered in Consul:

3

 

 

2.Start a container running Registrator

Here we use this image: https://hub.docker.com/r/gliderlabs/registrator/

Registrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online.

Registrator watches for new Docker containers and inspects them to determine what services they provide. For our purposes, a service is anything listening on a port. Any services Registrator finds on a container, they will be added to a service registry, such as Consul or etcd.

$ docker run -it -v /var/run/docker.sock:/tmp/docker.sock -h $DOCKER_IP gliderlabs/registrator consul://$DOCKER_IP:8500

 

How to check that Registrator is running?

In Docker-Machine, run

$ docker ps

Will see running Registrator container:

3

We can also check the logs:

$ docker logs $CONTAINER_ID

3

 

 

3. Start a web service and let Registrator automatically register it on Consul

In command line, into a directory that has proper rails app, Dockerfile and docker-compose.yml file in place.

$ docker-compose build
$ docker-compose run web bundle exec rake sample_data:populate
$ docker-compose run -p 3000 -e ‘SERVICE_NAME=romui-web’ web rails s

How to check that this service is registered?

Method1:

Use Consul API:

$ curl $DOCKER_IP/v1/catalog/services

3

We can check the service details using Consul HTTP API:

$ curl http://localhost:8500/v1/catalog/service/$SERVICE_NAME

3

Method2:

Check in Consul UI:

3

 

4. Find the IP address and port of the service from Consul UI, and start using the service.

3

Now we are ready to access the service from browser:

2

 

Docker/Gitlab CI – Environment Set Up Guide

1. Install Docker Engine

Use Ubuntu as example, follow the official installation guide before the “Optional Configurations” section:

https://docs.docker.com/engine/installation/linux/ubuntulinux/

In “Optional Configurations” section, only need to do “Create a docker group” and “Configure a DNS server for use by Docker”.

Create a docker group:

Why: Need to do this so that we can run docker without sudo.

Test that it works:

$docker ps

Configure a DNS server for use by Docker:

Why: Need to do this so that docker engine can resolve hostnames.

Test that it works:

$docker run -it ubuntu /bin/bash

$ping archive.ubuntu.com

An example error screenshot of not having this configured:

1
2. Install Docker Compose

Follow official guide: https://docs.docker.com/compose/install/
3. Install GitLab CI Runner and register multiple runners to use with Docker

Follow this guide: https://littletechblogger.wordpress.com/2016/03/28/install-and-set-up-gitlab-ci-runner-to-use-docker/
4. Set up job to clean non-tagged images and stopped containers

Use meltwater/docker-cleanup:  https://hub.docker.com/r/meltwater/docker-cleanup/

Install and set up multiple Gitlab CI Runners to use Docker

GitLab CI allows you to use Docker Engine to build and test docker-based projects.

We will follow the official guide: Using Docker Build

Here we choose the first method “Use shell executor”.

1.Install GitLab Runner:

Here we install gitlab-ci-multi-runner on linux. Follow this official guide until “Register the runner”: Install using official GitLab repositories

2. Register the runner:

Using Docker Build is trying to register the runner in non-interactive way. Somehow it doesn’t work. Here we follow the registration step in Install using official GitLab repositories, installing in interactive way.

1

As we can see from the above picture, we will need to provide coordinator URL and token. They can be found in your repository at “Settings” -> “Runners”:

1

And one thing different from the screenshot, For “Pleasse enter the executor: shell, docker, docker-ssh, ssh?” We need to type in shell.

 

After registering, we should be able to see the runner in “Settings” -> “Runners”:

1

3. Install docker, set up privileges.

Just continue following Using Docker Build  “Use shell executor” section.

4. Register multiple runners and make them run builds concurrently.

The runner config file is located at:

/etc/gitlab-runner/config.toml

1

The concurrent is default to 1. Here we use 3 concurrent runners, so we set it to 3.

5. Verify whether runners are running or not:

$sudo gitlab-ci-multi-runner verify

Docker – How to create a custom database from docker mysql image

Problem:

I use the official mysql base image from Docker Hub to create mysql database. But by default it only creates one database. I want to create another database.

Solution:

mysql image actually provides a way to do this. Reference the official doc: https://github.com/docker-library/docs/tree/master/mysql  Under “Initializing a fresh instance” section.

So we just need to create a .sql file with create database command, and mount this file as volume to docker-entrypoint-initdb.d folder inside mysql container.

This is my docker-compose.yml file:

version: 2′

services:
  db:
    image: mysql   # Pull mysql image from Docker Hub
    environment:   # Set up mysql database name and password
      MYSQL_DATABASE: development
      MYSQL_ROOT_PASSWORD: password
    ports:    # Set up ports exposed for other containers to connect to
      - "3306"
    volumes:  # Mount relative path source folder on host to absolute path destination folder on docker container
      - ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
  web:
    build: .  # Look in the current directory for a Dockerfile and build images using it
    depends_on:    # Link web container to db container
      - db
    environment:  # Set up environment variables
      MYSQL_DATABASE: development
      MYSQL_ROOT_PASSWORD: password
      RAILS_ENV: docker
    ports:
      - "3000:3000"  # Forwards the exposed port 3000 on the container to port 3000 on the host machine

 

We create init.sql file inside docker-entrypoint-initdb.d folder, and use “volumes” to mount this folder to docker-entrypoint-initdb.d inside mysql container, so that mysql service will have access to all the files in docker-entrypoint-initdb.d folder on host. The content of init.sql file:

CREATE DATABASE test;

After we run

$docker-compose up

It will boot up db and web service. The db service will be a mysql container, which have two databases: develop and test.

Docker – SSH to Docker Machine – remote host identification has changed!

Since I’ve been deleting a docker-machine and recreate constantly, and the docker-machine ip is dynamically assigned, I got the “warning: remote host identification has changed!” a lot.

Here is the solution:

http://stackoverflow.com/questions/20840012/ssh-remote-host-identification-has-change

 

And the following is the screenshot of my process:

1.could not connect, got error message

2.run solution commend

3.successfully connected

1

Docker – Overview

1. How does Docker work?
1.1 Build Docker Images that hold your applications.
1.2 Create Docker Containers from those Docker Images to run your applications.
1.3 Share Docker images via Docker Hub or your own registry.

 

2.Docker container lifecycle

– Conception
Build an image from a dockerfile
– Birth
Run (create + start) a container
– Reproduction
Run a new container from an image
– Sleep
Kill a running container
– Wake
Start a stopped container
– Death
RM (delete) a stopped container
-Exitinction
RMI (delete) a container image

 

3.Docker Commands (Used in Dockerfile)

Attention: all relative paths are relative to the Dockerfile

FROM    Specifies a source image for our new image that we plan to build. (mandatory)

MAINTAINER    Tells us who built/maintains the image.

RUN   Executes a command.

CMD   Default command to run when a container is launched.

EXPOSE   Lists the network ports to open when a container is launched from this image.

ADD   Adds files and contents from host into the image.

Adds files as owned by ‘root’ with permissions 0755.

COPY   Copies the files.

The source directory must be inside the context of build (one of directories/sub-directories of the current directory where Dockerfile is located).

VOLUME   Creates a data volume mount point at the specified path.

Data volumes can be shared and reused between containers.

Data volumes persist until all containers referencing them are destroyed.

ENV   Specifies environment variables that should be set in any container launched from the image.

ENTRYPOINT   The first command/script that gets executed.

Docker – Commands Reference

1.General Docker command

List info about the Docker Client:

$ docker version

Get docker command help:

—- List all commands:

$ docker –help

—- For a specific command:

$ docker <command> –help

— List networks:

$ docker network ls

 

2.Containers

— List containers:

—- List containers that are up:

$ docker ps

—- List the last started container:

$ docker ps -l

—- List all the containers:

$ docker ps -a

Start a container:

$ docker start <container_id>

— Stop containers:

—- Stop a container:

$ docker stop <container_id>

—- Stop all containers:

$ docker stop $(docker ps -a -q)

Restart a container:

$ docker restart <container_id>

— Remove a container:

$ docker rm <container_id>

— Delete all containers:

$ docker rm $(docker ps -a -q)

— Run a container(creates and starts the container):

—- Run a container:

$ docker run <image_name> <command>

For ex:

$ docker run ubuntu /bin/echo ‘Hello world’

—- Run a container in interactive mode:

$ docker run -t -i <image_name> <command>

This will launch a Bash shell inside our container.

—- Run a container in daemonized mode:

$ docker run -d <image_name> <command>

This will run the container and put it in the background, to daemonize it.

Commit (save) a container state

$ docker commit <container> <some_name>

— Get container info:

$ docker inspect <container_id>

Returns a JSON document with configuration and status info.

We can also narrow down the info we want to return by requesting a specific element.

For example, this gives the IP address of the container:

$ docker inspect –format ‘{{ .NetworkSettings.IPAddress }}’ <container_id>

Return container standard output:

$ docker logs <container_id>

Acts like tail -f:

$ docker logs -f <container_id>

Examine processes inside a container:

$ docker top <container_name>

— Show container size:

$ docker ps -s

 

3.Images

— Create an image:

—- Use Dockerfile:

$ docker build -t <username/image_name:tag> .

The -t flag is to identify the new docker image as belonging to <username/image_name:tag>.

. is to indicate use the Dockerfile in the current directory.

—- Use commit command:

$ docker commit -m “<message>” -a “<author>” <container_id> <docker_hub_username>/<image_name>:<tag>

For ex: $ docker commit -m “Added json gem” -a “Kate Smith” 0b2616b0e5a8 ouruser/sinatra:v2

— List images:

$ docker images

— Remove an image:

$ docker rmi <image_id>

— Delete all images:

$ docker rmi $(docker images -q)

Pull an image

$ docker pull <image_name>

Pulls an image from docker hub/specified host.

— Build an image

$ docker build -t <host/img_name:tag> .

Builds an image with specified hostname as prefix.

— Push an image

$ docker push <username/image_name>

Pushes the docker image to the specified host/docker hub.

 

4.docker-compose

docker-compose build:

Build the containers that the web app and db will live in, and link them up.

— docker-compose run:

Run one-off commands for services. For example, to see environment variables are available to the web service:

$ docker-compose run web env

— docker-compose up:

Build, link, and start services.

 

5.docker-machine

–Create a docker machine:

$ docker-machine create –driver virtualbox <docker_machine_name>

–List docker machines:

$ docker-machine ls

–Restart docker machine:

$ docker-machine restart <docker_machine_name>

–Stop docker machine:

$ docker-machine stop <docker_machine_name>

–Connect to a docker machine:

$docker-machine ssh <docker_machine_name>

 

6.Useful commands

Check iptables:

$ sudo iptables -t nat -L -n

— Find the PID of the process:

$ ps aux | grep <process_name>

— Kill a process:

$ kill <pid>