In the eternal hunt for elusive bugs, logging is an indispensable aid. By recording the events and messages that occur during the execution of your program, logging opens the door to unparalleled debugging and performance monitoring capabilities.
It all starts with Python’s built-in logging module. However, the true power of Python logging is unlocked not merely by using it, but by mastering it.
Keep reading to learn the fundamentals of logging in Python, get up to speed with best practices, and how one advanced error logging tool, Rollbar, can help you discover, predict, and resolve errors in real-time.
The basics of logging in Python: words you should know
Before we dive into the code, let's cover some basic concepts related to logging in Python:
- Logger: An object named a logger represents a named channel for message logging. You can add several loggers to your program, each with a unique name and set of configurations. The name of the logger you want to use is specified when you log a message.
- Handler: An entity that receives and processes logged messages is referred to as a handler. Each handler can have its own configuration, including a file name, a format string, and a level threshold. A logger may have one or more handlers. All of the handlers connected to the logger receive a message when it is logged.
- Formatter: A formatter is an object that transforms the logged messages into a string that can be read by humans. This can include placeholders for the message's content, the date and time, the logger's name, and other properties.
- Level: A level is a number that identifies the seriousness of a message that has been logged. In order of increasing severity, the Python logging module specifies five standard levels:
DEBUG,INFO,WARNING,ERROR, andCRITICAL. To regulate which messages are logged and which are disregarded, you can establish a level threshold for a logger or a handler.
How to configure logging in Python
To use the logging module in Python, you create a logger object and configure its handlers and formatters.
Example
In this example, a logger object named mylogger is created and its level is set to DEBUG. A file handler and a console handler are then created, and their levels are set to DEBUG and ERROR , respectively. Next, a formatter object to format the messages using a custom format string is created.
After that, the handlers are added to the logger using the addHandler method and, finally, five messages of different levels are logged using the logger object. Here’s the code:
import logging
# Create a logger
logger = logging.getLogger('mylogger')
logger.setLevel(logging.DEBUG)
# Create a file handler
fh = logging.FileHandler('mylog.log')
fh.setLevel(logging.DEBUG)
# Create a console handler
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
# Create a formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)
# Add handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)
# Log some messages
logger.debug('Debug message')
logger.info('Info message')
logger.warning('Warning message')
logger.error('Error message')
logger.critical('Critical message')
When you run the above code, you get the following output:
2023-04-26 12:33:37,204 - mylogger - ERROR - Error message
2023-04-26 12:33:37,207 - mylogger - CRITICAL - Critical message
And you should also see a file named mylog.log in the current directory containing the following messages:
2023-04-26 12:33:37,199 - mylogger - DEBUG - Debug message
2023-04-26 12:33:37,199 - mylogger - INFO - Info message
2023-04-26 12:33:37,204 - mylogger - WARNING - Warning message
2023-04-26 12:33:37,204 - mylogger - ERROR - Error message
2023-04-26 12:33:37,207 - mylogger - CRITICAL - Critical message
Python logging levels explained
The logging module defines five standard levels of logging, in increasing order of severity:
-
DEBUGFor the purpose of debugging, detailed information. Debug messages are usually used only during development and testing because they can make a lot of noise in the production settings.
-
INFOGeneral details regarding how the program is being run. Info messages are helpful for monitoring the general development of your application and providing updates on significant occasions or milestones.
-
WARNINGA sign that something unexpected has transpired or that an issue will arise soon (like "disk space low"). Although the software is still functioning as intended, there might be some problems that need fixing.
-
ERRORThe software has been unable to carry out some tasks because of a more significant issue. This can be the result of incorrect code, a missing file, or another problem that stops the program from running properly.
-
CRITICALA highly significant error that might prevent the application from continuing to operate. This can be the result of a fatal programming error, an irrecoverable system error, or another problem that needs to be fixed right away.
Choose one of these predefined levels to describe the level of severity when logging a message. These levels should be adequate for the majority of use cases, but if bespoke levels are required, you can also define more on your own.
Best practices for Python logging
When logging in Python, follow these best practices to ensure your logs are useful, manageable, and secure.
-
Use meaningful log messages
Use precise language to explain what happened and why when logging messages. Later on, this will make interpreting the logs easier.
-
Use structured logging
Use structured logging formats like JSON or XML rather than simply recording a text. This enhances your log messages with extra context, such as information about the user or request that initiated the event.
-
Configure loggers, handlers, and formatters
Your logs, handlers, and formatters should be configured using the
logging.basicConfig()method. That way you can make sure that your logs are written in the right place, with the right format, and with the right amount of information. -
Use different logging levels
Logging levels regulate the level of detail in your logs. Log more detailed debug messages when working on a project, but switch to shorter info or warning messages when it comes time to launching it.
-
Use logging handlers
To send your log messages to various locations, such a file, a database, or a third-party logging service, use logging handlers.
-
Use log rotation
Implement log rotation to prevent excessive file growth and disk space usage in your log files. Old log files are automatically archived, compressed, and deleted using this approach. You can either use a third-party tool like
logrotateor the Python language's built-inRotatingFileHandlerto construct your own solution. -
Test your logging
To make sure that your log messages are being stored accurately and at the proper levels, test your logging configuration. Unit tests or integration tests can be used for this.
-
Use loggers for modules and classes
Loggers can be used to divide logging statements into modules and classes. As a result, it’s simpler to determine which section of the code is malfunctioning.
-
Avoid logging sensitive information
Logging sensitive data, such as passwords or user information, should be done with caution. Alternatively, log metadata that can be utilized to connect the logs with different system events.
-
Adopt the
ISO-8601format for timestampsIt’s simpler to interpret and compare logs across various systems and time zones thanks to this format, which offers a standardized way to describe dates and times in log messages. Especially when dealing with huge amounts of data, using a consistent timestamp format makes sure that your logs are simple to read and interpret. The timestamp represented in
ISO-8601format appears as follows:2023-06-14T15:00-03:00Example logging in ISO-8601 format
import logging from datetime import datetime logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO) # Log a message with the current time in ISO-8601 format now = datetime.utcnow().replace(microsecond=0).isoformat() + 'Z' logging.info(f'My log message at {now}')Output
2022-08-02T10:35:02Z My log message at 2022-08-02T10:35:02ZThe
Zstands for “Zulu time” which is a term used in military and aviation contexts to refer to Coordinated Universal Time (UTC). By adding theZto the end of the datetime string, it ensures that the time zone information is included and that the datetime is unambiguously in UTC format.
By adhering to these best practices, your logs will be reliable, effective, and offer insightful information about your code.
Use Rollbar for real-time error tracking and debugging
Logging is a crucial component of software development, but the logging module in Python can only take you so far. Automate error tracking and triaging with Rollbar to turbocharge your logging process. Here’s what sets Rollbar apart:
- Real-Time Error Alerting and Analysis: Rollbar collects and analyzes errors as they happen. It provides real-time alerts through various channels like email and Slack to ensure that developers are immediately aware of any issues.
- Intelligent Grouping and Aggregation: Rollbar groups similar errors together, making it easier to identify and fix recurring issues.
- Rich Contextual Data: For each error, Rollbar provides detailed contextual information such as stack traces, request parameters, environment details, affected users, and more.
- Integration with Existing Workflow: Rollbar integrates seamlessly with many popular development tools, project management platforms, alerting systems, and data analysis tools.
- Telemetry and Timeline Data: Rollbar captures a timeline of events leading up to each error, including user actions, network requests, console logs, and more. This telemetry data can provide valuable insight into what caused an error.
- Regulation Compliance: Rollbar provides features that help with compliance to various regulations like GDPR, HIPAA, etc. For example, it allows developers to control and manage how they capture and store user data.
So what are you waiting for? Rollbar makes fixing Python errors easier than ever. Try it today!
