python腳本監控-用python監控python腳本執行情況（日誌記錄）

需求：現有爬蟲程序（名為CNSubAllInd），需要使其一直保持在後台運行（如果執行完畢，立即重新啟動，繼續執行），並記錄其運行日誌。

利用python的logging模塊來記錄日誌，利用subprocess模塊來和系統交互執行命令，檢測到子程序結束運行之後，重新開啟子程序。

代碼如下keeprunning.py(CNSubAllInd就是需要保持在後台運行的程序)：

# -*- coding: UTF-8 -*-
#!DATE: 2018/10/9
#!@Author: yingying
#keeprunning.py
import os
import subprocess
# logging
# require python2.6.6 and later
import logging
from logging.handlers import RotatingFileHandler
## log settings: SHOULD BE CONFIGURED BY config
LOG_PATH_FILE = "D:workspacePyCharmProjectCompanyInfoSpidermy_service_mgr.log"
LOG_MODE = "a"
LOG_MAX_SIZE = 10 * 1024 * 1024 # 10M per file
LOG_MAX_FILES = 10 # 10 Files: my_service_mgr.log.1, printmy_service_mgrlog.2, ...
LOG_LEVEL = logging.DEBUG
LOG_FORMAT = "%(asctime)s %(levelname)-10s[%(filename)s:%(lineno)d(%(funcName)s)] %(message)s"
handler = RotatingFileHandler(LOG_PATH_FILE, LOG_MODE, LOG_MAX_SIZE, LOG_MAX_FILES)
formatter = logging.Formatter(LOG_FORMAT)
handler.setFormatter(formatter)
Logger = logging.getLogger()
Logger.setLevel(LOG_LEVEL)
Logger.addHandler(handler)
# color output
#
pid = os.getpid()
def print_error(s):
print "33[31m[%d: ERROR] %s33[31;m" % (pid, s)
def print_info(s):
print "33[32m[%d: INFO] %s33[32;m" % (pid, s)
def print_warning(s):
print "33[33m[%d: WARNING] %s33[33;m" % (pid, s)
def start_child_proc(command, merged):
try:
if command is None:
raise OSError, "Invalid command"
child = None
if merged is True:
# merge stdout and stderr
child = subprocess.Popen(command)
# child = subprocess.Popen(command,
# stderr=subprocess.STDOUT, # 表示子進程的標準錯誤也輸出到標準輸出
# stdout=subprocess.PIPE # 表示需要創建一個新的管道
# )
else:
# DO NOT merge stdout and stderr
child = subprocess.Popen(command)
# child = subprocess.Popen(command,
# stderr=subprocess.PIPE,
# stdout=subprocess.PIPE)
return child
except subprocess.CalledProcessError:
pass # handle errors in the called executable
except OSError:
raise OSError, "Failed to run command!"
def run_forever(command):
print_info("start child process with command: " + " ".join(command))
Logger.info("start child process with command: " + " ".join(command))
merged = False
child = start_child_proc(command, merged)
failover = 0
while True:
while child.poll() != None:
failover = failover + 1
print_warning("child process shutdown with return code: " + str(child.returncode))
Logger.critical("child process shutdown with return code: " + str(child.returncode))
print_warning("restart child process again, times=%d" % failover)
Logger.info("restart child process again, times=%d" % failover)
child = start_child_proc(command, merged)
# read child process stdout and log it
out, err = child.communicate()
returncode = child.returncode
if returncode != 0:
for errorline in err.slitlines():
Logger.info(errorline)
else:
Logger.info("execute child process failed")
Logger.exception("!!!should never run to this!!!")
if __name__ == "__main__":
run_forever(["scrapy", "crawl", "CNSubAllInd"])

在這裡感謝cheungmine提供的subprocess腳本寫一個python的服務監控程序。

windows中運行方式：在命令行中輸入start pythonw keeprunning.py命令，之後便會打開pythonw窗口如下：

注意：這個窗口是關不掉的，因為有keeprunning在後台運行，一旦檢測到爬蟲程序結束了，就會重新打開一個窗口（也即重新開啟程序）。想要關閉的話，只能在任務管理器中關閉pythonw.exe程序，便停止了監控，當前爬蟲程序執行完畢之後便結束爬蟲。

但是原作者提供的通過read來獲取執行輸出結果的方法（如下），我使用的時候會出現deadlock，每次就卡在read這裡不往下執行了。

while True:
while child.poll() != None:
failover = failover + 1
print_warning("child process shutdown with return code: " + str(child.returncode))
Logger.critical("child process shutdown with return code: " + str(child.returncode))

print_warning("restart child process again, times=%d" % failover)
Logger.info("restart child process again, times=%d" % failover)
child = start_child_proc(command, merged)

# deadlock!!!
ch = child.stdout.read(1)
if ch != "" and ch != "
":
line += ch
if ch == "
":
print_info(line)
line = ""

查了相關資料以及官方文檔之後，Python Popen().stdout.read() hang發現問題就出在這裡，按照官方文檔的解釋是之所以調用.stdout會卡死，是因為在讀完最後一行後管道空了。

為了防止出現這樣的情況應該使用communicate()來代替.stdout.read()，communicate的使用見官方文檔

喜歡這篇文章嗎？立刻分享出去讓更多人知道吧！