share facebook facebook2 twitter menu hatena pocket slack

2014.08.25 MON

HAProxy でバックエンドサーバーのヘルスチェックログを記録する

川原 洋平

WRITTEN BY川原 洋平

どうも、HAProxy 教日本支部東京分科会のかっぱ (@inokara)です。あまりに万能すぎる HAProxy に翻弄される毎日を送っています。

健康チェックしてますか?

HAProxy でバックエンドサーバーのヘルスチェックを行いたい場合には default セクションに以下を追加します。

	log     global
	option  log-health-checks

あと、rsyslog 経由でログを出力したいので global セクションにログ出力の設定をしておきます。

log         127.0.0.1 local6

上記のように設定します。

その上で /etc/rsyslog.conf に以下を追記します。

local6.* -/var/log/haproxy/haproxy_healthcheck.log;Haproxy

ログファイル名は任意(適当に…)で。

どんなログが出るか…

バックエンドを停止すると…

バックエンドが停止するパターンとして…

  • Web サーバー停止
  • Web サーバーは生きているけど、ヘルスチェック用にコンテンツへアクセス出来ない(コンテンツ異常)

上記の二つを想定してヘルスチェックのログを見てみました。

尚、ヘルスチェックの設定は下記の通りです。

 check fall 2 rise 2 inter 1000
  • 2 回のチェックに失敗したら切り離し
  • 2 回のチェックに成功したら切り戻し
  • チェック間隔は 1000ms

Web サーバー停止

 Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 1/2 UP.
 Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 1/2 UP.
 Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 0/2 DOWN.
 Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
 Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 0/2 DOWN.
 Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Web サーバー復旧

 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN.
 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN.
 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP.
 Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.
 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP.
 Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.

ヘルスチェックコンテンツ異常

 Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 4ms, status: 1/2 UP.
 Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 5ms, status: 1/2 UP.
 Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 4ms, status: 0/2 DOWN.
 Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
 Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 4ms, status: 0/2 DOWN.
 Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
 

コンテンツ復旧

 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN.
 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN.
 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP.
 Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.
 Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP.
 Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.

ほうほう。このログを監視してアラート上げたりとか出来そうですな。

とりあえず

出来ればこのログは出力するようにしておきたいものです。

元記事はこちらです。
HAProxy でバックエンドサーバーのヘルスチェックログを記録する