Lesson 4

Managing Logs with AWS SDK for Python

Introduction to Boto3 Logging

Welcome to a new adventure! As we continue our journey with Python and AWS using Boto3, we are diving into logging in Boto3 in this lesson. We'll explore Python's logging library, its interaction with Boto3, and how to leverage logs to better understand your API requests and responses. Let's get started!

Importance of Logging

Our applications often present puzzles that require careful analysis and investigation. Logs are like the bread crumbs that help us navigate these puzzles. They provide insights into the actions of our application, assisting us in understanding AWS service interaction, identifying potential issues, and tracking the overall health of our operations. This kind of visibility is essential for efficient debugging and management of AWS interactions.

Python's logging library and Boto3

Boto3 features the set_stream_logger() function, which allows it to integrate with Python's logging library. You can control the details of what gets logged with the name parameter in this function. There are several options:

  • General Boto3 Logging: Use the name 'boto3' to log all Boto3 activities.
1import boto3 2import logging 3 4logging.basicConfig(level=logging.INFO) 5boto3.set_stream_logger(name='boto3', level=logging.DEBUG)
  • Specific Service Logging: To focus on one service, use the name of that service's client. For instance, for S3 operations, use 'botocore.client.S3'.
1import boto3 2import logging 3 4logging.basicConfig(level=logging.INFO) 5boto3.set_stream_logger(name='botocore.client.S3', level=logging.DEBUG)
  • Botocore Requests: Use the name 'botocore' to gain detailed insights into request and response interactions.
1import boto3 2import logging 3 4logging.basicConfig(level=logging.INFO) 5boto3.set_stream_logger(name='botocore', level=logging.DEBUG)
Recap: Common Logging Levels in Python

Python's logging module features several logging levels, representing different severities of the events to be logged. These are:

  • DEBUG : Captures detailed information, typically of interest only when diagnosing problems.
  • INFO : Confirms that things are working as expected.
  • WARNING : Indicates that something unexpected happened, or there may be a problem in the future (e.g., 'disk space low').
  • ERROR : More serious problems that have prevented the program from performing a function.
  • CRITICAL : Very serious errors, which may prevent the program from running.

By default, Python logging library is set to WARNING. This means the system will ignore all logging messages that have lower severity than WARNING. This includes DEBUG and INFO messages.

If you want all messages to be logged, you can change the base level to DEBUG or INFO via logging.basicConfig(level=logging.DEBUG) or logging.basicConfig(level=logging.INFO) respectively. It’s a common practice to adjust this setting during development, and then change it back to WARNING or higher once the application is deployed.

Logging Best Practices and Considerations

Logging requires careful handling. Avoid logging sensitive information like access tokens, passwords, and personally identifiable information (PII), and ensure log access is properly restricted within your organization. When properly configured, logging can offer invaluable insights for development and troubleshooting, while securely managing information.


Well done! You've successfully completed the lesson on Boto3 logging! You now understand how to integrate Python’s logging library with Boto3 and how to selectively capture logs for specific AWS services or all Boto3 activities. As you proceed with your AWS journey, leverage these logging techniques to better debug your applications and deepen your understanding of AWS interactions. Happy logging!

Enjoy this lesson? Now it's time to practice with Cosmo!

Practice is how you turn knowledge into actual skills.