Consideration about python logging and file configuration(2)

tatsuro included in python

2021-10-22 772 words 4 minutes

Contents

Former article (Consideration about python logging and file configuration(1)) has concerns to be updated.

There is more space to improve the usage for logging library with module naming space with __name__.
setting configuration by config.fileConfig is older and limited in features (e.g. filter setting is not supported) (ref. document). Instead, it’s better to use config.dictConfig and load configuration file in .json or .yml format, a format that can be loaded as python' dictionary format.

Environment

MacOS Catalina 10.15.7
Python 3.9.7

5. Single logger for different modules, with utilizing module naming space

Directory structure

1
2
3
4
5
6
7


python_logging
├── sample5
│   ├── __init__.py
│   ├── logging5.conf
│   ├── sample5_called.py
│   └── sample5_main.py
└── setup.py

Above sample directory structure can be said that every python module (sample5_main.py and sample5_called.py) belongs to sample5 package.

In that case, __name__ in each module is:

sample5_main.py : __main__ (this is because sample5_main.py is called as main routine. If it’s not called as main routine, __name__ is sample5.sample5_main)
sample5_called.py : sample5.sample5_called

So, if a logger is defined by a name of sample5, a root package name, the modules below can succeed that setting.

Using __name__ would be good to use logger function (handler, filter etc.) with distinguishing modules by naming space for expansion in the future.

Tip

For adding a root package name (sample5) as a module, it’s necessary to add path to the parent directory that python recognizes. For doing that, see some ways to add python path

logging5.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


[loggers]
keys=root, sample5

[handlers]
keys=fileHandler, consoleHandler

[logger_root]
handlers=fileHandler, consoleHandler
level=DEBUG

[logger_sample5]
handlers=fileHandler, consoleHandler
level=DEBUG
qualname=sample5
propagate=0

[handler_fileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=logFormatter
args=('sample5/sample5.log', 'D')

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=logFormatter
args=(sys.stdout, )

[formatters]
keys=logFormatter

[formatter_logFormatter]
class=logging.Formatter
format=%(asctime)s %(thread)d %(levelname)s %(name)s %(funcName)s %(lineno)d %(message)s

sample5_main.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


from logging import config, getLogger

config.fileConfig('logging.conf')
logger = getLogger(__name__)

def main():
    logger.debug("this is logged by main()")

if __name__ == '__main__':
    main()

sample5_called.py

1
2
3
4
5
6


from logging import getLogger

logger = getLogger(__name__)

def sample5_func():
    logger.debug("this is logged by sample5_called")

run module from outside of sample5 directory.

1
2
3


$ python sample5/sample5_main.py 
2021-10-22 13:59:02,028 4456861120 DEBUG __main__ main 8 this is logged by main()
2021-10-22 13:59:02,028 4456861120 DEBUG sample5.sample5_called sample5_func 6 this is logged by sample5_called

Output in sample5/sample5.log is like:

1
2


2021-10-22 13:59:02,028 4456861120 DEBUG __main__ main 8 this is logged by main()
2021-10-22 13:59:02,028 4456861120 DEBUG sample5.sample5_called sample5_func 6 this is logged by sample5_called

6. logger with `config.dictConfig`

Directory structure

1
2
3
4
5
6
7


python_logging
├── sample6
│   ├── __init__.py
│   ├── logging_conf.json
│   ├── sample6_called.py
│   └── sample6_main.py
└── setup.py

logging_conf.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


{
    "version": 1,
    "disable_existing_loggers": false,
    "root": {
      "level": "DEBUG",
      "handlers": [
        "consoleHandler",
        "logFileHandler"
      ]
    },
    "sample6": {
        "level": "DEBUG",
        "handlers": [
          "consoleHandler",
          "logFileHandler"
        ],
        "propagate": false
    },
    "handlers": {
      "consoleHandler": {
        "class": "logging.StreamHandler",
        "level": "DEBUG",
        "formatter": "consoleFormatter",
        "stream": "ext://sys.stdout"
      },
      "logFileHandler": {
        "class": "logging.handlers.TimedRotatingFileHandler",
        "level": "DEBUG",
        "formatter": "logFileFormatter",
        "filename": "./sample6/sample6.log",
        "when" : "MIDNIGHT",
        "backupCount": 7,
        "encoding": "utf-8"
      }
    },
    "formatters": {
      "consoleFormatter": {
        "format": "%(asctime)s %(thread)d %(levelname)s %(name)s %(funcName)s %(lineno)d %(message)s"
      },
      "logFileFormatter": {
        "format": "%(asctime)s %(thread)d %(levelname)s %(name)s %(funcName)s %(lineno)d %(message)s"
      }
    }
}

Tip

"disable_existing_loggers": false should be set for seeing existing logger including third party libraries' one with useful information.
"handlers.logFileHandler.when": "MIDNIGHT" seems to be better than "D" for surely enabling to switch log file over the end of the day.

sample6_main.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


import json
from logging import config, getLogger
from sample6.sample6_called import sample6_func

config_dict = None
with open('sample6/logging_conf.json', 'r', encoding='utf-8') as f:
    config_dict = json.load(f)

config.dictConfig(config_dict)
logger = getLogger(__name__)

def main():
    logger.debug("this is logged by main()")
    sample6_func()

if __name__ == '__main__':
    main()

sample6_called.py

1
2
3
4
5
6


from logging import getLogger

logger = getLogger(__name__)

def sample6_func():
    logger.debug("this is logged by sample6_called")

The output in standard output and log file is the same as sample5.

Sources

Sample codes are uploaded in github

Contents

Consideration about python logging and file configuration(2)

Environment

5. Single logger for different modules, with utilizing module naming space

6. logger with config.dictConfig

Sources

References

6. logger with `config.dictConfig`