このエントリはKubernetesアドベントカレンダー2014の15日目です。
そろそろソロプレイが辛くなってきました。
ログまわりについては @kazunori_279 さんが Google、FluentdをKubernetesとCompute Engineの標準ログコレクタに採用 - Qiita を公開されています。
このエントリではもっと軽い、使い方を紹介します。
Kubernetes + flannel + CentOS7 4台でクラスタ構築 - netmark.jp で作った環境を前提に紹介します。CentOS7 4台(Master x1, Minion x3)でguestbook exampleを動かしています。
kubectl でマスターの kube-apiserver に問合せることで書くコンテナのログを取得できます。
syntaxは kubectl log Pod名 コンテナ名 です。
コンテナ名は Pod 一覧ではわからないのですが、個別にgetを発行すると取得することができます。
[root@master01 ~]# kubectl get pods -l name=redis-master -s http://172.16.1.1:8080
NAME IMAGE(S) HOST LABELS STATUS
redis-master dockerfile/redis 172.16.2.1/ name=redis-master Running
[root@master01 ~]# kubectl get pod redis-master -o json -s http://172.16.1.1:8080 | jq '.desiredState.manifest.containers[].name'
"master"
[root@master01 ~]# kubectl log redis-master master -s http://172.16.1.1:8080
2014-12-15T14:14:37.695852207Z _._
2014-12-15T14:14:37.695852207Z _.-``__ ''-._
2014-12-15T14:14:37.695852207Z _.-`` `. `_. ''-._ Redis 2.8.18 (00000000/0) 64 bit
2014-12-15T14:14:37.695852207Z .-`` .-```. ```\/ _.,_ ''-._
2014-12-15T14:14:37.695852207Z ( ' , .-` | `, ) Running in stand alone mode
2014-12-15T14:14:37.695852207Z |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
2014-12-15T14:14:37.695852207Z | `-._ `._ / _.-' | PID: 1
2014-12-15T14:14:37.695852207Z `-._ `-._ `-./ _.-' _.-'
2014-12-15T14:14:37.695852207Z |`-._`-._ `-.__.-' _.-'_.-'|
2014-12-15T14:14:37.695852207Z | `-._`-._ _.-'_.-' | http://redis.io
2014-12-15T14:14:37.695852207Z `-._ `-._`-.__.-'_.-' _.-'
2014-12-15T14:14:37.695852207Z |`-._`-._ `-.__.-' _.-'_.-'|
2014-12-15T14:14:37.695852207Z | `-._`-._ _.-'_.-' |
2014-12-15T14:14:37.695852207Z `-._ `-._`-.__.-'_.-' _.-'
2014-12-15T14:14:37.695852207Z `-._ `-.__.-' _.-'
2014-12-15T14:14:37.695852207Z `-._ _.-'
2014-12-15T14:14:37.695852207Z `-.__.-'
2014-12-15T14:14:37.695852207Z
2014-12-15T14:14:37.696848426Z [1] 15 Dec 14:14:37.695 # Server started, Redis version 2.8.18
2014-12-15T14:14:37.696848426Z [1] 15 Dec 14:14:37.696 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2014-12-15T14:14:37.696893061Z [1] 15 Dec 14:14:37.696 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2014-12-15T14:14:37.696956882Z [1] 15 Dec 14:14:37.696 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2014-12-15T14:14:37.696982004Z [1] 15 Dec 14:14:37.696 * The server is now ready to accept connections on port 6379
2014-12-15T14:21:41.868762616Z [1] 15 Dec 14:21:41.840 * Slave 172.17.38.1:6379 asks for synchronization
2014-12-15T14:21:41.868762616Z [1] 15 Dec 14:21:41.868 * Full resync requested by slave 172.17.38.1:6379
2014-12-15T14:21:41.868762616Z [1] 15 Dec 14:21:41.868 * Starting BGSAVE for SYNC with target: disk
2014-12-15T14:21:41.885043073Z [1] 15 Dec 14:21:41.881 * Background saving started by pid 8
2014-12-15T14:21:41.887151156Z [1] 15 Dec 14:21:41.886 * Slave 172.17.58.0:6379 asks for synchronization
2014-12-15T14:21:41.887151156Z [1] 15 Dec 14:21:41.887 * Full resync requested by slave 172.17.58.0:6379
2014-12-15T14:21:41.887219952Z [1] 15 Dec 14:21:41.887 * Waiting for end of BGSAVE for SYNC
2014-12-15T14:21:41.907801988Z [8] 15 Dec 14:21:41.906 * DB saved on disk
2014-12-15T14:21:41.907801988Z [8] 15 Dec 14:21:41.907 * RDB: 0 MB of memory used by copy-on-write
2014-12-15T14:21:42.037288456Z [1] 15 Dec 14:21:42.036 * Background saving terminated with success
2014-12-15T14:21:42.037288456Z [1] 15 Dec 14:21:42.037 * Synchronization with slave 172.17.38.1:6379 succeeded
2014-12-15T14:21:42.037288456Z [1] 15 Dec 14:21:42.037 * Synchronization with slave 172.17.58.0:6379 succeeded
コンテナが実際にはどのサーバにあるかを意識せずに、Masterからログを閲覧できました！
すごい！
今回は時間が取れず裏側の仕組みまでは確認できていませんが、
各Minionでそれぞれのコンテナの標準出力を集めた /var/lib/docker/containers/コンテナID/コンテナID-json.log を収集しているようです。