Container

In this article, I am going to change the SSH port from 22 to 52241 and create a new firewall rule in VPC. Please do your own research and make sure you understand the impact, especially on a production environment.

When you deploy a new docker container on GCE, you can SSH into the host machine ( VM ). This is simply because the IP address is pointing to the host machine. But what about the docker container that runs on it? In this case, you must use docker exec or the gcloud command-line tool in order to get access to the container, which can be pretty time consuming over time.

docker exec -ti {docker-container} bash

1	docker exec -ti {docker-container} bash

I am working on a project where I need direct access to my alpine docker container on port 22 or any other port. As you can see in the below output, no ports are exposed for my container.

CONTAINER ID        IMAGE                                                                COMMAND                  CREATED             STATUS              PORTS               NAMES
36a7b16b17dc        gcr.io/tobiasforkel/magento2:latest                                   "/usr/bin/supervisor…"   15 minutes ago      Up 15 minutes                           klt-tobiasforkel-2-kwkm
4597512dd812        gcr.io/stackdriver-agents/stackdriver-logging-agent:0.2-1.5.33-1-1   "/entrypoint.sh /usr…"   16 minutes ago      Up 16 minutes                           stackdriver-logging-agent

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

36a7b16b17dc gcr.io/tobiasforkel/magento2:latest "/usr/bin/supervisor…" 15 minutes ago Up 15 minutes klt-tobiasforkel-2-kwkm

4597512dd812 gcr.io/stackdriver-agents/stackdriver-logging-agent:0.2-1.5.33-1-1 "/entrypoint.sh /usr…" 16 minutes ago Up 16 minutes stackdriver-logging-agent

This is mainly because the docker container shares the network with the host. Source

Containerized VMs launch containers with the network set to host mode. A container shares the host network stack, and all interfaces from the host are available to the container.

This is very convenient because it allows you to access your docker container from pretty much any port. However, only a few ports are actually open and configured in the VPC Network > Firewall settings for security reasons. For the SSH connection I decided to open a custom port 52241. You can do this in your GCP Console under VPC Network > Firewall or with the following gcloud command-line tool.

gcloud compute firewall-rules create custom-allow-ssh --allow tcp:52241 --priority=65534 --description="Allow SSH connections on port 52241" --direction=INGRESS

1	gcloud compute firewall-rules create custom-allow-ssh --allow tcp:52241 --priority=65534 --description="Allow SSH connections on port 52241" --direction=INGRESS

gcloud compute firewall-rules create custom-allow-ssh --allow tcp:52241 --priority=65534 --description="Allow SSH connections on port 52241" --direction=INGRESS
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/tobiasforkel/global/firewalls/custom-allow-ssh].
Creating firewall...done.                                                                        
NAME              NETWORK  DIRECTION  PRIORITY  ALLOW      DENY  DISABLED
custom-allow-ssh  default  INGRESS    65534     tcp:52241        False

gcloud compute firewall-rules create custom-allow-ssh --allow tcp:52241 --priority=65534 --description="Allow SSH connections on port 52241" --direction=INGRESS

Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/tobiasforkel/global/firewalls/custom-allow-ssh].

Creating firewall...done.

NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED

custom-allow-ssh default INGRESS 65534 tcp:52241 False

Once completed, here is how the new firewall rule looks like in the console. If you scroll down, you will also see all affected instances.

Before you can SSH into your container on port 52241 you must change the port in the sshd configuration /etc/ssh/sshd_config within your container or container image. Not on the VM.

Port 52241
PermitRootLogin no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PrintMotd yes
AcceptEnv LANG LC_*

Port 52241

PermitRootLogin no

PermitEmptyPasswords no

ChallengeResponseAuthentication no

PrintMotd yes

AcceptEnv LANG LC_*

You may have to restart your ssh service before you can use the new port. Once this is done, you can SSH into your docker container.

ssh tobias@34.73.237.216 -p 52241
tobias@34.73.237.216's password: 
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

ssh tobias@34.73.237.216 -p 52241

tobias@34.73.237.216's password:

Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general

information about administrating Alpine systems.

See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

Today was a strange day. I’ve launched a new version of my docker image within an ECS cluster, which usually worked fine. However, today my container always got recreated every 5 minutes for no reason.

The new docker image that I have pushed to the repository had no significant changes, except a redirect from non-SSL ( 80 ) to SSL ( 443 ) for all loaded virtual hosts. So, when I had a look at the processes, everything looked fine. There was nothing that could put the container into an unhealthy status.

PID   USER     TIME  COMMAND
    1 root      0:02 {supervisord} /usr/bin/python2 /usr/bin/supervisord -c /etc/supervisord.conf
    8 root      0:00 nginx: master process /usr/sbin/nginx -g daemon off;
    9 root      0:00 {mysqld.sh} /bin/sh /scripts/mysqld.sh
   10 root      0:00 /usr/sbin/crond -f
   13 nginx     0:13 nginx: worker process
   35 root      0:00 {php-fpm7.1} php-fpm: master process (/etc/php/7.1/php-fpm.conf)
   90 root      0:00 {mysqld_safe} /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql
  288 mysql     0:08 /usr/bin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mariadb/plugin --user=mysql --log-error=/var/lib/mysql/f817894c1752.err --pid-file=f817894c1752.pid --socket=/var/run/my
 3620 www-data  0:06 {php-fpm7.1} php-fpm: pool www
 3621 www-data  0:05 {php-fpm7.1} php-fpm: pool www
 3668 www-data  0:02 {php-fpm7.1} php-fpm: pool www
 3795 root      0:00 bash
 3807 root      0:00 ps aux

PID USER TIME COMMAND

1 root 0:02 {supervisord} /usr/bin/python2 /usr/bin/supervisord -c /etc/supervisord.conf

8 root 0:00 nginx: master process /usr/sbin/nginx -g daemon off;

9 root 0:00 {mysqld.sh} /bin/sh /scripts/mysqld.sh

10 root 0:00 /usr/sbin/crond -f

13 nginx 0:13 nginx: worker process

35 root 0:00 {php-fpm7.1} php-fpm: master process (/etc/php/7.1/php-fpm.conf)

90 root 0:00 {mysqld_safe} /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql

288 mysql 0:08 /usr/bin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mariadb/plugin --user=mysql --log-error=/var/lib/mysql/f817894c1752.err --pid-file=f817894c1752.pid --socket=/var/run/my

3620 www-data 0:06 {php-fpm7.1} php-fpm: pool www

3621 www-data 0:05 {php-fpm7.1} php-fpm: pool www

3668 www-data 0:02 {php-fpm7.1} php-fpm: pool www

3795 root 0:00 bash

3807 root 0:00 ps aux

Even the log output of the docker container didn’t show any error. After a little bit of research, I ended up on the following AWS developer guide page https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service.html

When the service scheduler replaces a task during an update, the service first removes the task from the load balancer (if used) and waits for the connections to drain. Then, the equivalent of docker stop is issued to the containers running in the task. This results in a SIGTERM signal and a 30-second timeout, after which SIGKILL is sent and the containers are forcibly stopped. If the container handles the SIGTERM signal gracefully and exits within 30 seconds from receiving it, no SIGKILL signal is sent. The service scheduler starts and stops tasks as defined by your minimum healthy percent and maximum percent settings.

When the service scheduler replaces a task during an update, the service first removes the task from the load balancer (if used) and waits for the connections to drain. Then, the equivalent of docker stop is issued to the containers running in the task. This results in a SIGTERM signal and a 30-second timeout, after which SIGKILL is sent and the containers are forcibly stopped. If the container handles the SIGTERM signal gracefully and exits within 30 seconds from receiving it, no SIGKILL signal is sent. The service scheduler starts and stops tasks as defined by your minimum healthy percent and maximum percent settings.

It was not exactly what I was looking for, but it pretty much explained the problem I had. One of my colleagues said I should maybe have a look at the health checks of the load balancer.

Indeed, when I had a look at the settings in EC2 > Load Balancing > Target Groups > Health Checks the health check expected a success code 200 instead of 301 on port 80. Because of that, the container has been marked unhealthy every 5 minutes.

I hope this will point someone in the right direction.

Tag: Container

GCE – SSH Directly Into Your Docker Container

AWS – ECS Keeps Recreating Docker Container