Unreleased

Prelude

Application Security Management (ASM) has added support for preventing attacks by blocking malicious IPs using one click within Datadog.

Note

One click activation for ASM is currently in beta.

Build and deploy Python 3.9 wheels for releases

Major changes to context management. See the upgrade section for the specifics. Note that only advanced users of the library should be affected by these changes. For the details please refer to the Context section of the docs: https://ddtrace.readthedocs.io/en/v0.50.0/advanced_usage.html

Several deprecations have been made to Context as we prepare to move active span management out of this class.

Dynamic instrumentation allows instrumenting a running service dynamically to extract runtime information that could be useful for, e.g., debugging purposes, or to add extra metrics without having to make code changes and re-deploy the service. See https://ddtrace.readthedocs.io/en/stable/configuration.html for more details.

Add support for Python 3.9

Initial library support has been added for Python 3.11.

Note

Continuous Profiler and Dynamic Instrumentation are not yet compatible and must be disabled in order to use the library with Python 3.11. Support for them will be added in a future release. To track the status, see the Support Python 3.11 issue on GitHub.

The Datadog APM Python team is happy to announce the release of v1.0.0 of ddtrace. This release introduces a formal versioning policy that simplifies the public interface and defines a release version policy for backwards compatible and incompatible changes to the public interface.

The v1.0.0 release is an important milestone for the library as it has grown substantially in scope. The first commit to the library was made on June 20, 2016. Nearly sixty minor releases later, the library now includes over sixty integrations for libraries. And the library has expanded from Tracing to support the Continuous Profiler and CI Visibility.

Important

Before upgrading to v1.0.0, we recommend users install ddtrace>=0.60.0,<1.0.0 and enable deprecation warnings. All removals to the library interface and environment variables were deprecated on 0.x branch. Consult Upgrade 0.x for recommendations on migrating from the 0.x release branch.

Note

The changes to environment variables apply only to the configuration of the ddtrace library and not the Datadog Agent.

Upgrading summary

Functionality changes

The default logging configuration functionality of ddtrace-run has changed to address conflicts with application logging configuration. See note on the new default behavior and note on deprecation for future removal.

Removed legacy environment variables

These environment variables have been removed. In all cases the same functionality is provided by other environment variables and replacements are provided as recommended actions for upgrading.

Variable

Replacement

Note

DATADOG_ prefix

DD_ prefix

📝

DATADOG_SERVICE_NAME

DD_SERVICE

📝

DD_LOGGING_RATE_LIMIT

DD_TRACE_LOGGING_RATE

📝

DD_TRACER_PARTIAL_FLUSH_ENABLED

DD_TRACE_PARTIAL_FLUSH_ENABLED

📝

DD_TRACER_PARTIAL_FLUSH_MIN_SPANS

DD_TRACE_PARTIAL_FLUSH_MIN_SPANS

📝

Removed legacy tracing interfaces

These methods and module attributes have been removed. Where the same functionality is provided by a different public method or module attribute, a recommended action is provided for upgrading. In a few limited cases, because the interface was no longer used or had been moved to the internal interface, it was removed and so no action is provided for upgrading.

Module

Method/Attribute

Note

ddtrace.context

Context.clone

📝

ddtrace.pin

Pin.app

📝

Pin.app_type

📝

ddtrace.sampler

Sampler.default_sampler

📝

ddtrace.span

Span.tracer

📝

Span.__init__(tracer=)

📝

Span.meta

📝

Span.metrics

📝

Span.set_meta

📝

Span.set_metas

📝

Span.pprint

📝

ddtrace.tracer

Tracer.debug_logging

📝

Tracer.get_call_context

📝

Tracer.tags

📝

Tracer.writer

📝

Tracer.__call__

📝

Tracer.global_excepthook

📝

Tracer.log

📝

Tracer.priority_sampler

📝

Tracer.sampler

📝

Tracer.set_service_info

📝

ddtrace.ext

SpanTypes

📝

ddtrace.helpers

get_correlation_ids

📝

ddtrace.settings

Config.HTTPServerConfig

📝

Removed legacy integration tracing

These tracing functions in integrations were no longer used for automatic instrumentation so have been removed. Any manual instrumentation code in an application will need to be replaced with ddtrace.patch_all() or ddtrace.patch() when upgrading.

Module

Function/Class

ddtrace.contrib.cassandra

get_traced_cassandra

📝

ddtrace.contrib.celery

patch_task

📝

ddtrace.contrib.celery

unpatch_task

📝

ddtrace.contrib.flask

middleware.TraceMiddleware

📝

ddtrace.contrib.mongoengine

trace_mongoengine

📝

ddtrace.contrib.mysql

get_traced_mysql_connection

📝

ddtrace.contrib.psycopg

connection_factory

📝

ddtrace.contrib.pymongo

patch.trace_mongo_client

📝

ddtrace.contrib.pymysql

tracers.get_traced_pymysql_connection

📝

ddtrace.contrib.requests

legacy

📝

ddtrace.contrib.redis

tracers.get_traced_redis

📝

ddtrace.contrib.redis

tracers.get_traced_redis_from

📝

ddtrace.contrib.sqlite3

connection_factory

📝

Removed deprecated modules

These modules have been removed. Many were moved to the internal interface as they were not intended to be used as part of the public interface. In these cases, no action is provided for upgrading. In a few cases, other modules are provided as alternatives to maintain functionality. See the notes for more information.

Module

Note

ddtrace.compat

📝

ddtrace.contrib.util

📝

ddtrace.encoding

📝

ddtrace.ext.errors

📝

ddtrace.ext.priority

📝

ddtrace.ext.system

📝

ddtrace.http

📝

ddtrace.monkey

📝

ddtrace.propagation.utils

📝

ddtrace.util

📝

ddtrace.utils

📝

New Features

  • Add tracing support for aiomysql>=0.1.0.

  • Add aredis support >= 1.1.0

  • Add automatic unix domain socket detection for Dogstatsd. The expected path for the socket is /var/run/datadog/dsd.socket which if exists, will be used instead of the previous UDP default, udp://localhost:8125/. To be used in conjunction with dogstatsd_socket in your datadog.yaml file, or the DD_DOGSTATSD_SOCKET environment variable set on the Datadog agent.

  • Add automatic unix domain socket detection for traces. The expected path for the socket is /var/run/datadog/apm.socket which if exists, will be used instead of the previous http default, http://localhost:8126/. To be used in conjunction with apm_config.receiver_socket in your datadog.yaml file, or the DD_APM_RECEIVER_SOCKET environment variable set on the Datadog agent.

  • internal: Add support for Datadog trace tag propagation

  • Add new DD_TRACE_AGENT_TIMEOUT_SECONDS to override the default connection timeout used when sending data to the trace agent. The default is 2.0 seconds.

  • Add DD_CALL_BASIC_CONFIG={true,false} environment variable to control whether ddtrace calls logging.basicConfig. By default when using ddtrace-run or running in debug mode logging.basicConfig is called to ensure there is always a root handler. This has compatibility issues for some logging configurations. DD_CALL_BASIC_CONFIG=false can be used to skip calling logging.basicConfig. The default value is true to maintain existing behavior.

  • django: added DD_DJANGO_INSTRUMENT_TEMPLATES=false to allow tracing of Django template rendering.

  • internal: Add sampling mechanism trace tag

  • Add support for grpc.aio.

  • Add tracing support for the httpx library. Supported versions >=0.14.0.

  • Add environment variables to write ddtrace logs to a file with DD_TRACE_LOG_FILE, DD_TRACE_LOG_FILE_LEVEL, and DD_TRACE_FILE_SIZE_BYTES

  • Adds pytest-bdd integration to show more details in CI Visibility product.

  • Add new DD_TRACE_SAMPLING_RULES environment variable to override default sampling rules. For Example:: DD_TRACE_SAMPLING_RULES='[{"sample_rate":0.5,"service":"my-service"}]'

  • Add support for snowflake-connector-python >= 2.0.0. Note that this integration is in beta and is not enabled by default. See the snowflake integration documentation for how to enable.

  • Add Span.get_tags and Span.get_metrics.

  • Add support for aiobotocore 1.x and 2.x.

  • botocore: add distributed tracing support for AWS EventBridge, AWS SNS & AWS Kinesis.

  • Add new environment variables to configure the internal trace writer.

    DD_TRACE_WRITER_MAX_BUFFER_SIZE, DD_TRACE_WRITER_INTERVAL_SECONDS, DD_TRACE_WRITER_MAX_PAYLOAD_SIZE_BYTES

  • Add new environment variables to enable/disable django database and cache instrumentation.

    DD_DJANGO_INSTRUMENT_DATABASES, DD_DJANGO_INSTRUMENT_CACHES

  • agent: support URL with a base path

  • aiohttp: add client integration. This integration traces requests made using the aiohttp client and includes support for distributed tracing. See the documentation for more information.

  • aiohttp_jinja2: move into new integration. Formerly the aiohttp_jinja2 instrumentation was enabled using the aiohttp integration. Use patch(aiohttp_jinja2=True) instead of patch(aiohttp=True). To support legacy behavior patch(aiohttp=True) will still enable aiohttp_jinja2.

  • Added support for aiopg~=0.16.0.

  • Add tracing support for the aioredis library. Version 1.3+ is fully supported.

  • ASGI: store the ASGI span in the scope. The span can be retrieved with the

    ddtrace.contrib.asgi.span_from_scope function.

  • ASM: add support for one click activation using Remote Configuration Management (RCM). Set DD_REMOTE_CONFIGURATION_ENABLED=true to enable this feature.

  • ASM: add user information to traces.

  • ASM: collect http client_ip.

  • ASM: configure the sensitive data obfuscator.

  • ASM: Detect attacks on Django body.

  • ASM: Detect attacks on Flask request cookies

  • ASM: Detect attacks on Django request cookies

  • ASM: Detect attacks on Pylons body.

  • ASM: ip address collection will be enabled if not explicitly disabled and appsec is enabled.

  • ASM: now http.client_ip and network.client.ip will only be collected if ASM is enabled.

  • ASM: propagate user id.

  • ASM: Detect attacks on Pylons HTTP query.

  • ASM: Detect attacks on Pylons request cookies

  • ASM: detect attacks on Pylons path parameters.

  • ASM: Report HTTP method on Pylons framework

  • ASM: Collect raw uri for Pylons framework.

  • ASM: Support In-App WAF metrics report.

  • tracing: HTTP query string tagged by default in http.url tag (sensitive query strings will be obfuscated).

  • asyncpg: add integration supporting v0.18.0 and above. See the docs for more information.

  • botocore: added distributed_tracing configuration setting which is enabled by default.

  • botocore: allow defining error status codes for specific API operations.

    See our botocore document for more information on how to enable this feature.

  • cherrypy: introduce TraceMiddleware for the CherryPy web framework.

  • AppSec: collect response headers

  • Automated context management should now work in all asynchronous frameworks that use contextvars.

  • tracer.start_span() now accepts an activate argument (default False) to allow manual context management.

  • tracer.current_trace_context() has been added to be used to access the trace context of the active trace.

  • Adds support for the Datadog Lambda Extension. The tracer will send traces to the extension by default if it is present.

  • Add support for space-separated DD_TAGS.

  • The ddtrace-run command now supports the following arguments:

    -h, –help -d, –debug enable debug mode (disabled by default) -i, –info print library info useful for debugging -p, –profiling enable profiling (disabled by default) -v, –version show program’s version number and exit

    It now also has friendlier error messages when used incorrectly.

  • Add functionality to call gevent.monkey.patch_all() with ddtrace-run by setting the environment variable DD_GEVENT_PATCH_ALL=true. This ensures that gevent patching is done as early as possible in the application.

  • ASM: Detect attacks on Flask body.

  • ASM: Detect attacks on path parameters

  • django: tag root spans as measured.

  • Add django 4.0 support.

  • Django: set usr.id tag by default if request.user is authenticated.

  • dogpile.cache: is now automatically instrumented by default.

  • elasticsearch: add support for version 7.

  • fastapi: add support for tracing fastapi.routing.serialize_response.

    This will give an insight into how much time is spent calling jsonable_encoder within a given request. This does not provide visibility into how long it takes for Response.render/json.dumps.

  • Collect user agent in normalized span tag http.useragent.

  • ASM: Detect attacks on XML body (for Django, Pylons and Flask).

  • Submit runtime metrics as distribution metrics instead of gauge metrics.

  • Adds support for Lambda profiling, which can be enabled by starting the profiler outside of the handler (on cold start).

  • aiohttp: store request and response headers.

  • bottle: store request and response headers.

  • ciapp: detect code owners of PyTest tests

  • fastapi: add integration.

  • Store request headers in Flask integration.

  • flask: store response headers.

  • botocore: inject distributed tracing data to ClientContext to trace lambda invocations.

  • botocore: inject tracing data to MessageAttributes.

  • molten: store request headers.

  • pyodbc integration. This enables the pyodbc library to trace queries.

  • pyramid: store request and response headers.

  • Introduce support for the DD_SERVICE_MAPPING environment variable to allow remapping service names on emitted spans.

  • starlette integration resource aggregation This aggregates endpoints to the starlette application resource that was accessed. It occurs by default but it is configurable through config.starlette[“aggregate_resources”].

  • urllib3: add urllib3 integration

  • Support flask-caching (>= 1.10.0) with the Flask-Cache tracer.

  • flask: store request and response headers when auto-instrumented.

  • The futures integration is now enabled by default.

  • A warning has been added to alert when gevent monkey patching is done after ddtrace has been imported.

  • httplib: distributed tracing is now enabled by default.

  • Added support for jinja2~=3.0.0.

  • Add support to reuse HTTP connections when sending trace payloads to the agent. This feature is disabled by default. Set DD_TRACE_WRITER_REUSE_CONNECTIONS=true to enable this feature.

  • The ddtrace.Tracer.get_log_correlation_context method has been added to replace ddtrace.helpers.get_correlation_ids. It now returns a dictionary which includes the current span’s trace and span ids, as well as the configured service, version, and environment names.

  • Update the –info command to be easier to read and provide more helpful information.

  • Add MariaDB integration.

  • MySQLdb: Added optional tracing for MySQLdb.connect, using the configuration option here.

  • The CI tagging for the pytest plugin now includes OS and Python Runtime metadata including system architecture, platform, version, and Python runtime name and version.

  • The ddtrace.profiling.auto module will warn users if gevent monkey patching is done after the profiler is auto-instrumented.

  • The profiler now tracks the running gevent Greenlet and store it as part of the CPU and wall time profiling information.

  • The profiler is now able to upload profiles to the Datadog Agent by using a Unix Domain Socket.

  • It is now possible to pass a url parameter to the Profiler to specify the Datadog agent location.

  • The profiler now supports Windows.

  • The profiler now supports profiling asyncio.Lock objects.

  • The profiler now reports asyncio tasks as part as the task field in profiles. This is enabled by default by replacing the default asyncio loop policy. CPU time, wall time and threading lock times are supported.

  • The memory profile collector can now entirely disabled with the DD_PROFILING_MEMORY_ENABLED environment variable.

  • The new memory profiler for Python is now enabled by default. This improves the profiler memory consumption and performance impact. It can still be disabled by setting DD_PROFILING_MEMALLOC=0 as an environment variable.

  • profiler: collect and export the class name for the wall time, CPU time and lock profiles, when available.

  • Profiling now supports tracing greenlets with gevent version prior to 1.3.

  • The profiler now automatically injects running greenlets as tasks into the main thread. They can be seen within the wall time profiles.

  • The heap profiler is now enabled by default.

  • The Python heap profiler can now be enabled by setting the DD_PROFILING_HEAP_ENABLED environment variable to 1.

  • Add support for DD_PROFILING_ENDPOINT_COLLECTION_ENABLED env variable to disable endpoint name collection in profiler.

  • The exception profiler now gathers and exports the traces and spans information.

  • The profiler now supports code provenance reporting. This can be enabled by using the enable_code_provenance=True argument to the profiler or by setting the environment variable DD_PROFILING_ENABLE_CODE_PROVENANCE to true.

  • The Profiler object can now be passed tags with the tags keyword argument.

  • The profiler now exports active tasks for CPU and wall time profiles.

  • The profiler now captures the traces information with the lock profiling.

  • The gevent tasks are now tracked by the threading lock events

  • The profiler now uses the tracer configuration is no configuration is provided.

  • The profiler now supports most operation mode from uWSGI without much configuration. It will automatically plug itself in post fork hooks when multiprocess mode is used.

  • psycopg2: add option to enable tracing psycopg2.connect method.

    See our psycopg documentation for more information.

  • pylons: now supports all the standard http tagging including query string, custom error codes, and request/response headers.

  • add DD_PYMONGO_SERVICE configuration

  • Only for CI Visibility (pytest integration): include pytest version as a tag in the test span.

  • pytest integration. This enables the pytest runner to trace test executions.

  • Only for CI Visibility (pytest integration): remove traces whose root span is not a test.

  • The pytest plugin now includes support for automatically tagging spans with parameters in parameterized tests.

  • The ddtrace pytest plugin can now call ddtrace.patch_all via the --ddtrace-patch-all option.

  • The pytest integration now uses the name of the repository being tested as the default test service name.

  • Added official support for Python 3.10

  • tracer: added support for Python 3.11.

  • ASM: Redact sensitive query strings if sent in http.url.

  • Add asyncio support of redis ≥ 4.2.0

  • redis: track the connection client_name.

  • rediscluster: add service name configuration with DD_REDISCLUSTER_SERVICE

  • requests: add global config support. This enables the requests service name to be configured with ddtrace.config.requests['service'] or the DD_REQUESTS_SERVICE environment variable.

  • The Profiler instances now restart automatically in child process when the main program is forked. This only works for Python ≥ 3.7.

  • Add rq integration.

  • snowflake: add snowflake query id tag to sql.query span

  • Span now accepts a on_finish argument used for specifying functions to call when a span finishes.

  • Add Span.set_tag_str() as an alternative to the overloaded functionality of Span.set_tag() when the value can be coerced to unicode text.

  • Add support for Flask 2

  • Add tracing support for sanic>=21.9.0.

  • Enable telemetry collection when tracing is enabled.

  • Tag traces with HTTP headers specified on the DD_TRACE_HEADER_TAGS environment variable. Value must be either comma or space separated. e.g. Host:http.host,User-Agent,http.user_agent or referer:http.referer Content-Type:http.content_type.

  • Added retry logic to the tracer to mitigate potential networking issues, like timeouts or dropped connections.

  • Only for CI Visibility (pytest integration): Extract stage and job name from environment data in Azure Pipelines.

  • Only for CI Visibility (pytest integration): It is now possible to specify any of the following git metadata through environment variables: * DD_GIT_REPOSITORY_URL: The url of the repository where the code is stored * DD_GIT_TAG: The tag of the commit, if it has one * DD_GIT_BRANCH: The branch where this commit belongs to * DD_GIT_COMMIT_SHA: The commit hash of the current code * DD_GIT_COMMIT_MESSAGE: Commit message * DD_GIT_COMMIT_AUTHOR_NAME: Commit author name * DD_GIT_COMMIT_AUTHOR_EMAIL: Commit author email * DD_GIT_COMMIT_AUTHOR_DATE: The commit author date (ISO 8601) * DD_GIT_COMMIT_COMMITTER_NAME: Commit committer name * DD_GIT_COMMIT_COMMITTER_EMAIL: Commit committer email * DD_GIT_COMMIT_COMMITTER_DATE: The commit committer date (ISO 8601)

  • wsgi: add tracing middleware.

  • Add yaaredis ≥ 2.0.0 support.

Upgrade Notes

  • The deprecated attribute ddtrace.Sampler.default_sampler is removed.

  • Spans started after Tracer.shutdown() has been called will no longer be sent to the Datadog Agent.

  • Support for aiohttp previous to 2.0 has been removed.

  • botocore: Update trace propagation format for directly invoked Lambda functions. This breaks compatibility with Lambda functions instrumented with datadog-lambda-python < v41 or datadog-lambda-js < v3.57.0. Please upgrade datadog-lambda-* in invoked lambda functions, or engage legacy compatibility mode in one of two ways: - ddtrace.config.botocore.invoke_with_legacy_context = True - DD_BOTOCORE_INVOKE_WITH_LEGACY_CONTEXT=true

  • The deprecated dogstatsd_host and dogstatsd_port arguments to tracer.configure() have been removed.

  • Context.get() has been removed and the functionality has been rolled into Context.close_span().

  • Tracer.record() has similarly been removed as it is no longer useful with Context.get() removed.

  • ddtrace.contrib.asyncio - AsyncioContextProvider can now return and activate None, Span or Context objects.

  • ddtrace.contrib.gevent - GeventContextProvider can now return and activate None, Span or Context objects.

  • ddtrace.contrib.tornado - TracerStackContext can now return and activate None, Span or Context objects.

  • ddtrace.context.Context no longer maintains the active/current span state.

    get_current_root_span() has been removed. Use tracer.current_root_span() instead. get_current_span() has been removed. Use tracer.current_span() instead. add_span() has been removed. To activate a span in an execution use tracer.context_provider.activate() instead. close_span() has been removed. To deactivate a span in an execution use tracer.context_provider.activate() instead.

  • ddtrace.provider.BaseContextProvider active() now returns None, Span or Context objects. activate() now accepts None, Span or Context objects.

  • ddtrace.span.Span - Span.context will now return a Context

  • ddtrace.tracer.Tracer tracer.get_call_context() will now return a one-off Context reference. This is to maintain backwards compatibility with the API but the functionality differs slightly. tracer.start_span() passing a span.context for child_of no longer adds the strong _parent reference to the new span.

  • Default value of DD_CALL_BASIC_CONFIG was updated from True to False. Call logging.basicConfig() to configure logging in your application.

  • aiohttp_jinja2: use patch(aiohttp_jinja2=True) instead of patch(aiohttp=True) for enabling/disabling the integration.

  • Replace DD_TRACER_PARTIAL_FLUSH_ENABLED with DD_TRACE_PARTIAL_FLUSH_ENABLED Replace DD_TRACER_PARTIAL_FLUSH_MIN_SPANS with DD_TRACE_PARTIAL_FLUSH_MIN_SPANS

  • The legacy Django configuration method (deprecated in 0.34) has been removed.

  • Support for gevent 1.0 has been removed and gevent >= 1.1 is required.

  • The aredis integration is now enabled by default.

  • elasticsearch: removed get_traced_transport method and ddtrace.contrib.elasticsearch.transport module.

  • flask: deprecated configuration option extra_error_codes has been removed.

  • ddtrace.settings.Config.HTTPServerConfig is removed.

  • Instead of using error constants from ddtrace.ext.errors. Use constants from ddtrace.constants module. For example: ddtrace.ext.errors.ERROR_MSG -> ddtrace.constants.ERROR_MSG

  • Instead of using priority constants from ddtrace.ext.priority. Use constants from ddtrace.constants module. For Example:: ddtrace.ext.priority.AUTO_KEEP -> ddtrace.constants.AUTO_KEEP

  • Instead of using system constants from ddtrace.ext.system. Use constants from ddtrace.constants module. For Example:: ddtrace.ext.system.PID -> ddtrace.constants.PID

  • Support for MySQL-python has been removed.

  • Support for deprecated DD_PROFILING_API_URL environment variable has been removed. Use DD_SITE instead.

  • Support for deprecated DD_PROFILING_API_KEY environment variable has been removed. Use DD_API_KEY instead.

  • Profiling support for agentless mode must now be explicitly enabled.

  • The profiler will only load tags from the DD_TAGS environment variable once at start.

  • The profiler asyncio_loop_policy attribute has been renamed to asyncio_loop_policy_class to accept a user-defined class. This guarantees the same asyncio loop policy class can be used process children.

  • The deprecated compatibility module ddtrace.profile has been removed.

  • The deprecated pyddprofile wrapper has been removed. Use ddtrace-run with DD_PROFILING_ENABLED=1 set instead.

  • The profiler now uses the tracer configuration is no configuration is provided.

  • The profiler now automatically sets up uWSGI compatibility in auto mode or with profile_children=True. Make sure that you don’t have custom code instrumenting the profiler in those cases.

  • Support for psycopg < 2.7 has been removed.

  • The ddtrace pytest plugin can now label spans from test cases marked xfail with the tag “pytest.result”

    and the reason for being marked xfail under the tag “pytest.xfail.reason”.

  • Python 3.11: Continuous Profiler and Dynamic Instrumentation must be disabled as they do not current support Python 3.11.

  • Removed ddtrace.ext.AppTypes and its usages in the tracer library.

  • cassandra: get_traced_cassandra is removed. Use ddtrace.patch(cassandra=True) or ddtrace.patch_all() instead.

  • celery: ddtrace.contrib.celery.patch_task is removed. Use ddtrace.patch(celery=True) or ddtrace.patch_all() instead.

  • celery: ddtrace.contrib.celery.unpatch_task is removed. Use ddtrace.contrib.celery.unpatch() instead.

  • ddrace.context.Context.clone is removed. This is no longer needed since the tracer now supports asynchronous frameworks out of the box.

  • ddtrace.constants.FILTERS_KEY is removed.

  • ddtrace.constants.NUMERIC_TAGS is removed.

  • ddtrace.constants.LOG_SPAN_KEY is removed.

  • The deprecated module ddtrace.contrib.util is removed.

  • The deprecated module ddtrace.compat is removed.

  • The deprecated module ddtrace.encoding is removed.

  • The deprecated modules ddtrace.http and ddtrace.http.headers are removed. Use ddtrace.contrib.trace_utils.set_http_meta to store request and response headers on a span.

  • Remove deprecated ddtrace.install_excepthook.

  • Remove deprecated ddtrace.uninstall_excepthook.

  • The deprecated module ddtrace.propagation.utils is removed.

  • The deprecated module ddtrace.utils and its submodules are removed:

    • ddtrace.utils.attr

    • ddtrace.utils.attrdict

    • ddtrace.utils.cache

    • ddtrace.utils.config

    • ddtrace.utils.deprecation

    • ddtrace.utils.formats

    • ddtrace.utils.http

    • ddtrace.utils.importlib

    • ddtrace.utils.time

    • ddtrace.utils.version

    • ddtrace.utils.wrappers

  • ddtrace.Tracer.sampler is removed.

  • ddtrace.Tracer.priority_sampler is removed.

  • ddtrace.Tracer.tags is removed. Use the environment variable DD_TAGS to set the global tags instead.

  • ddtrace.Tracer.log was removed.

  • The deprecated module ddtrace.ext.errors is removed. Use the ddtrace.constants module instead:

    from ddtrace.constants import ERROR_MSG
from ddtrace.constants import ERROR_STACK
from ddtrace.constants import ERROR_TYPE

  • The deprecated module ddtrace.ext.priority is removed. Use the ddtrace.constants module instead for setting sampling priority tags:

    from ddtrace.constants import USER_KEEP
from ddtrace.constants import USER_REJECT

  • The deprecated module ddtrace.ext.system is removed. Use ddtrace.constants.PID instead.

  • The legacy environment variables DD_SERVICE_NAME and DATADOG_SERVICE_NAME are removed. Use DD_SERVICE instead.

  • mongoengine: The deprecated method ddtrace.contrib.mongoengine.trace_mongoengine is removed. Use ddtrace.patch(mongoengine=True) or ddtrace.patch() instead.

  • mysql: The deprecated method ddtrace.contrib.mysql.get_traced_mysql_connection is removed. Use ddtrace.patch(mysql=True) or ddtrace.patch_all() instead.

  • Pin.app is removed.

  • Pin.app_type is removed.

  • psycopg: ddtrace.contrib.psycopg.connection_factory is removed. Use ddtrace.patch(psycopg=True) or ddtrace.patch_all() instead.

  • ddtrace.Span.pprint is removed.

  • Span.to_dict is removed.

  • Span.tracer is removed.

  • The deprecated tracer argument is removed from ddtrace.Span.__init__().

  • sqlite3: ddtrace.contrib.sqlite3.connection_factory is removed. Use ddtrace.patch(sqlite3=True) or ddtrace.patch_all() instead.

  • Remove deprecated attribute ddtrace.Tracer.debug_logging. Set the logging level for the ddtrace.tracer logger instead:

    import logging
log = logging.getLogger("ddtrace.tracer")
log.setLevel(logging.DEBUG)

  • ddtrace.Tracer.__call__ is removed.

  • ddtrace.Tracer.global_excepthook is removed.

  • ddtrace.Tracer.set_service_info is removed.

  • ddtrace.Tracer.writer is removed. To force flushing of buffered traces to the agent, use ddtrace.Tracer.flush() instead.

  • ddtrace.warnings.DDTraceDeprecationWarning is removed.

  • DD_TRACE_RAISE_DEPRECATIONWARNING environment variable is removed.

  • The environment variables prefixed with DATADOG_ are removed. Use environment variables prefixed with DD_ instead.

  • The environment variable DD_LOGGING_RATE_LIMIT is removed. Use DD_TRACE_LOGGING_RATE instead.

  • The environment variable DD_TRACER_PARTIAL_FLUSH_ENABLED is removed. Use DD_TRACE_PARTIAL_FLUSH_ENABLED instead.

  • The environment variable DD_TRACER_PARTIAL_FLUSH_MIN_SPANS is removed. Use DD_TRACE_PARTIAL_FLUSH_MIN_SPANS instead.

  • ddtrace.util is removed.

  • requests: spans will no longer inherit the service name from the parent.

  • Use Span.set_tag_str() instead of Span.set_tag() when the tag value is a text type as a performance optimizations in manual instrumentation.

  • The return value of Span.pprint() has been changed to a single line in the tracer debug logs rather than the previous custom multiline format.

  • ddtrace.ext.SpanTypes is no longer an Enum. Use SpanTypes.<TYPE> instead of SpanTypes.<TYPE>.value.

  • Spans are now processed per tracer instance. Formerly spans were stored per-Context which could be shared between tracer instances. Note that context management is not affected. Tracers will still share active spans.

  • Spans from asynchronous executions (asyncio, gevent, tornado) will now be processed and flushed together. Formerly the spans were handled per-task.

  • tracer.write() will no longer have filters applied to the spans passed to it.

  • The Tracer class properties DEFAULT_HOSTNAME, DEFAULT_PORT, DEFAULT_DOGSTATSD_PORT, DEFAULT_DOGSTATSD_URL, DEFAULT_AGENT_URL have been removed.

  • Tracer.write has been removed.

  • A ValueError will now be raised on tracer initialization if the Agent URL specified to the initializer or with the environment variable DD_TRACE_AGENT_URL is malformed.

  • The function ddtrace.utils.merge_dicts has been removed.

  • uWSGI is no longer supported with ddtrace-run due to a limitation of how tracer initialization occurs. See the updated instructions for enabling tracing in the library uWSGI documentation.

Deprecation Notes

  • Context.clone is deprecated. It will not be required in 0.50.

  • Context.add_span is deprecated and will be removed in 0.50.

  • Context.add_span is deprecated and will be removed in 0.50.

  • Context.close_span is deprecated and will be removed in 0.50.

  • Context.get_current_span is deprecated and will be removed in 0.50 please use Tracer.current_span instead.

  • Context.get_current_root_span is deprecated and will be removed in 0.50 please use Tracer.current_root_span instead.

  • Deprecate the configuration of the analytics through the generic dbapi2 configuration. This should now be configured via integration configurations, for example:

    # Before
export DD_TRACE_DBAPI2_ANALYTICS_ENABLED=1

# After
export DD_TRACE_SQLITE3_ANALYTICS_ENABLED=1

  • DD_CALL_BASIC_CONFIG is deprecated.

  • Deprecate DATADOG_TRACE_AGENT_HOSTNAME, DATADOG_TRACE_AGENT_PORT, DATADOG_PRIORITY_SAMPLING, DATADOG_PATCH_MODULES in favor of their DD equivalents.

    [Deprecated environment variable] | [Recommended environment variable]

    • For DATADOG_TRACE_AGENT_HOSTNAME, use DD_AGENT_HOST

    • For DATADOG_TRACE_AGENT_PORT use DD_AGENT_PORT

    • For DATADOG_PRIORITY_SAMPLING, follow ingestion controls

    • For DATADOG_PATCH_MODULES, use DD_PATCH_MODULES

  • ddtrace.compat has been deprecated and will be removed from the public API in ddtrace version 1.0.0.

  • ddtrace.constants.FILTERS_KEY is deprecated. Use settings={"FILTERS": ...} instead when calling tracer.configure.

  • ddtrace.constants.NUMERIC_TAGS is deprecated.

  • ddtrace.constants.LOG_SPAN_KEY is deprecated.

  • Deprecate the DATADOG_ENV environment variable in favor of DD_ENV. The use of DD_ENV should follow Unified Service Tagging recommendations.

  • ddtrace.sampler.DatadogSampler.default_sampler property is deprecated and will be removed in 1.0.

  • Deprecate ddtrace.config.dbapi2 as default for TracedCursor and TracedConnection as well as DD_DBAPI2_TRACE_FETCH_METHODS. Use IntegrationConfig and DD_<INTEGRATION>_TRACE_FETCH_METHODS specific to each dbapi-compliant library. For example:

    # Before
config.dbapi2.trace_fetch_methods = True

# After
config.psycopg2.trace_fetch_methods = True

  • The DD_TRACER_PARTIAL_FLUSH_ENABLED and DD_TRACER_PARTIAL_FLUSH_MIN_SPANS environment variables have been deprecated and will be removed in version 1.0 of the library.

  • The use of ddtrace.encoding has been deprecated and will be removed in version 1.0.0.

  • The ddtrace.http module has been deprecated and will be removed in version 1.0.0, with the ddtrace.http.headers module now merged into ddtrace.trace_utils.

  • HttpServerConfig is no longer part of the public API.

  • monkey.patch_module is deprecated.

  • monkey.get_patch_module is deprecated.

  • Pin.app is deprecated.

  • The pyddprofile wrapper is deprecated. Use DD_PROFILING_ENABLED=true ddtrace-run instead.

  • ddtrace.Span.pprint() is deprecated and will be removed in v1.0.

  • ddtrace.Span.tracer is deprecated.

  • ddtrace.Tracer.log is deprecated. Use ddtrace.tracer.log instead.

  • ddtrace.Tracer.sampler is deprecated.

  • ddtrace.Tracer.priority_sampler is deprecated.

  • ddtrace.Tracer.tags is deprecated. Use the environment variable DD_TAGS to set the global tags instead.

  • ddtrace.Tracer.writer is deprecated. To force flushing of buffered traces to the agent, use ddtrace.Tracer.flush() instead.

  • ddtrace.propagation.utils has been deprecated and will be removed in version 1.0.

  • The reuse of a tracer that has been shut down has been deprecated. A new tracer should be created for generating new traces.

  • The ddtrace.utils module and all of its submodules have been copied over into ddtrace.internal in an effort to internalize these modules. Their public counterparts will be removed entirely in version 1.0.0.

  • The contents of monkey.py have been moved into _monkey.py in an effort to internalize the module. Public methods have been imported back into monkey.py in order to retain compatibility, but monkey.py will be removed entirely in version 1.0.0.

  • The collect_metrics argument of the tracer.configure method has been deprecated. Runtime metrics should be enabled only via the DD_RUNTIME_METRICS_ENABLED environment variable.

  • Moved ddtrace.ext.errors constants into the ddtrace.constants module. ddtrace.ext.errors will be removed in v1.0. Shorthand error constant (MSG,TYPE,STACK) in ddtrace.ext.errors will be removed in v1.0. Function get_traceback() in ddtrace.ext.errors is now deprecated and will be removed v1.0.

  • Moved ddtrace.ext.priority constants into ddtrace.constants module.

  • Moved ddtrace.ext.system constants into ddtrace.constants module.

  • The profiler does not catch uncaught exception anymore.

  • Removed the collect_metrics argument from Tracer.configure. See the release notes for v0.49.0 for the migration instructions.

  • The deprecated dbapi2 configuration has been removed. The integration-specific configuration should be used instead. Look at the v0.48.0 release notes for migration instructions.

Bug Fixes

  • Fixes deprecation warning for asyncio.coroutine decorator.

  • Fix issue with attrs and contextlib2 version constraints for Python 2.7.

  • CGroup file parsing was fixed to correctly parse container UUID for PCF containers.

  • starlette: Add back removed aggregate_resources feature.

  • fastapi: Add back removed aggregate_resources feature.

  • Fix application crash on startup when using channels >= 3.0.

  • Fix broken builds for Python 2.7 on windows where <stdint.h> was not available. This change also ensures we build and publish cp27-win wheels.

  • aiohttp_jinja2: use app_key to look up templates.

  • ASGI: handle decoding errors when extracting headers for trace propagation.

  • ASM: (flask) avoid json decode error while parsing request body.

  • ASM: only report actor.ip on attack.

  • ASM: fix Python 2 error reading WAF rules.

  • ASM: reset wsgi input after reading.

  • internal: normalize header names in ASM

  • profiling: implement __aenter__ and __aexit__ methods on asyncio.Lock wrapper.

  • The Records parameter to Firehose endpoint calls is being excluded from the tags to avoid generating traces with a large payload.

  • pytest: report exception details directly instead of through a RuntimeWarning exception.

  • Tracer: fix configuring tracer with dogstatsd url.

  • The OpenTracing tracer.start_span method no longer activates spans.

  • Datadog active spans will no longer take precedence over OpenTracing active spans.

  • Corrected some typing annotations for PEP 484 compliance

  • core: always reset the current_span in the context.

  • dbapi: add support for connection context manager usage

  • django: Http404 exceptions will no longer be flagged as errors

  • Django: add support for version 3.1+ ASGI applications. A different codepath is taken for requests starting in Django 3.1 which led to the top level span not being generated for requests. The fix introduces automatic installation of the ASGI middleware to trace Django requests.

  • Resolves an issue in Django tracing where, if query_string is not present in request.META, a KeyError is raised, causing the request to 500

  • Fixes an issue where all Django function middleware will share the same resource name.

  • django: add safe guards for building http.url span tag.

  • django: check view before instrumenting MRO.

  • dogpile.cache: handle both kwargs and args in the wrapper functions (using

    only kwargs would result in an IndexError.

  • Allow the elasticsearch service name to be overridden using the integration config or the DD_SERVICE_MAPPING environment variable.

  • elasticsearch: patch versioned elasticsearch modules (elasticsearch1, …, elasticsearch7).

  • core: use loose types when encoding.

  • aiobotocore: set span error for 5xx status codes.

  • aiohttp: fix issue causing ddtrace.contrib.aiohttp_jinja2.patch module to be imported instead of the patch() function.

  • aiomysql: fix AttributeError: __aenter__ when using cursors as context managers.

  • Fix parenting of Redis command spans when using aioredis 1.3. Redis spans should now be correctly attributed as child of any active parent spans.

  • Fixes incompatibility of wrapped aioredis pipelines in async with statements.

  • aioredis: added exception handling for CancelledError in the aioredis integration.

  • Fixes issue with aioredis when empty pool is not available and execute returns a coroutine instead of a future. When patch tries to add callback for the span using add_done_callback function it crashes because this function is only for futures.

  • Escape non-Unicode bytes when decoding aioredis args. This fixes a UnicodeDecodeError that can be thrown from the aioredis integration when interacting with binary-encoded data, as is done in channels-redis.

  • ASM: fix a body read problem on some corner case where don’t passing the content length makes wsgi.input.read() blocks.

  • Fix internal import of deprecated ddtrace.utils module.

  • CI Visibility: fixed AppVeyor integration not extracting the full commit message.

  • asgi, starlette, fastapi: Exclude background tasks duration from web request spans.

  • asgi: set the http.url tag using the hostname in the request header before defaulting to the hostname of the asgi server.

  • flask: add support for flask>=2.2.0

  • Fixes parsing of botocore env variables to ensure they are parsed as booleans.

  • botocore: Do not assume that ResponseMeta exists in the results.

  • botocore: omit SecretBinary and SecretString from span metadata for calls to Secrets Manager.

  • CGroup file parsing was fixed to correctly parse container ID with preceding characters.

  • tracing: make ddtrace.context.Context serializable which fixes distributed tracing across processes.

  • django: avoid SynchronousOnlyOperation when failing to retrieve user information.

  • Add iterable methods on TracedCursor. Previously these were not present and would cause iterable usage of cursors in DB API integrations to fail.

  • dbapi: avoid type error with potential non-compliance in db libraries when setting tag for row count.

  • tracing: fix handling of unicode _dd.origin tag for Python 2

  • Fix parsing of the DD_TAGS environment variable value to include support for values with colons (e.g. URLs). Also fixed the parsing of invalid tags that begin with a space (e.g. DD_TAGS=" key:val" will now produce a tag with label key, instead of `` key``, and value val).

  • Fixed the environment variable used for log file size bytes to be DD_TRACE_LOG_FILE_SIZE_BYTES as documented.

  • tracing: fix issue with ddtrace-run having the wrong priority order of tracer host/port/url env variable configuration.

  • The shutdown task is re-registered when a tracer is reused after it has been shut down.

  • Fix memory leak caused when the tracer is disabled.

  • django,redis: fix unicode decode error when using unicode cache key on Python 2.7

  • django: add legacy resource format of {handler}.

  • django: fix a bug where multiple database backends would not be instrumented.

  • django: fix a bug when postgres query is composable sql object.

  • django: handle erroneous middleware gracefully.

  • Fixes an issue with the Django integration where if the urlconf changes at any point during the handling of the request then the resource name will only be <METHOD> 404. This fix moves resource name resolution to the end of the request.

  • Fixes an issue where a manually set django.request span resource would get overwritten by the integration.

  • Fixed the handling of the Django template name tag causing type errors.

  • elasticsearch: set span error for 5xx status codes.

  • tracing/internal: fix encoding of propagated internal tags.

  • The thread safety of the custom buffered encoder was fixed in order to eliminate a potential cause of decoding errors of trace payloads (missing trace data) in the agent.

  • Fix handling of Python exceptions during trace encoding. The tracer will no longer silently fail to encode invalid span data and instead log an exception.

  • The tracer now captures the task ID from the cgroups file for Fargate >= 1.4.0 and reports it to the agent as the Datadog-Container-ID tag.

  • Fixes error with tagging non-string Flask view args.

  • werkzeug.exceptions.NotFound 404 errors are no longer raised and logged as a server error in the Flask integration.

  • Fixed the support for Celery workers that fork sub-processes with Python 3.6 and earlier versions.

  • Fix error when calling concurrent.futures.ThreadPoolExecutor.submit with fn keyword argument.

  • Ensure gevent is automatically patched.

  • grpc: handle None values for span tags.

  • grpc: ensure grpc.intercept_channel is unpatched properly

  • grpc: handle no package in fully qualified method

  • grpc: fix wrapper for streaming response to support libraries that call an internal method directly.

  • grpc: Add done callback in streaming response to avoid unfinished spans if a StopIteration is never raised, as is found in the Google Cloud libraries.

  • grpc: handle IPv6 addresses and no port in target.

  • jinja2: fix handling of template names which are not strings.

  • Fix JSON encoding error when a bytes string is used for span metadata.

  • Set correct service in logs correlation attributes when a span override the service.

  • Deprecate the DD_LOGGING_RATE_LIMIT variable in favor of the standard DD_TRACE_LOGGING_RATE for configuring the logging rate limit.

  • Fix DD_LOGS_INJECTION incompatibility when using a logging.StrFormatStyle (logging.Formatter(fmt, style="{")) log formatter.

  • Fix a bug when tracing a mako DefTemplate or any Template that does not have a filename property.

  • Fixes an issue when trying to manually start the runtime metrics worker:

    AttributeError: module 'ddtrace.internal.runtime' has no attribute 'runtime_metrics'

  • Fixed a bug that prevented the right integration name to be used when trying to patch a module on import that is already loaded.

  • Fixes type hinting for **patch_modules parameter for patch/patch_all functions.

  • mypy: Avoid parsing redis asyncio files when type checking Python 2

  • tracing: fix nested web frameworks re-extracting and activating HTTP context propagation headers.

  • opentracing: don’t raise an exception when distributed tracing headers are not present when attempting to extract.

  • Fix the reporting of the allocated memory and the number of allocations in the profiler.

  • Fixed an issue with gevent worker processes that caused them to crash and stop.

  • Profiler raises a typing error when Span.resource is unicode on Python 2.7.

  • Fix issue building ddtrace from source on macOS 12.

  • Fix issue building ddtrace for the Pyston Python implementation by not building the _memalloc extension anymore when using Pyston.

  • Fixed support for pytest-bdd 6.

  • Fixes an issue when using the pytest plugin with doctest which raises an AttributeError on DoctestItem.

  • Fixed JSON encoding errors in the pytest plugin for parameterized tests with dictionary parameters with tuple keys. The pytest plugin now always JSON encodes the string representations of test parameters.

  • Fixed JSON encoding errors in the pytest plugin for parameterized tests with complex Python object parameters. The pytest plugin now defaults to encoding the string representations of non-JSON serializable test parameters.

  • Fixes cases where a pytest test parameter object string representation includes the id() of the object, causing the test fingerprint to constantly change across executions.

  • Fixes cases in which the test.status tag of a test span from pytest would be missing because pytest_runtest_makereport hook is not run, like when pytest has an internal error.

  • Fixes a bug in the pytest plugin where xfail test cases in a test file with a module-wide skip raises attribute errors and are marked as xfail rather than skipped.

  • Ensure that worker threads that run on start-up are recreated at the right time after fork on Python < 3.7.

  • fastapi/starlette: when using sub-apps, formerly a call to /sub-app/hello/{name} would give a resource name of /sub-app. Now the full path /sub-app/hello/{name} is used for the resource name.

  • Fixes an issue with enabling the runtime worker introduced in v0.49.0 where no runtime metrics were sent to the agent.

  • sampler: removed bug causing sample_rate of 0 to be reset to 1, and raise ValueError instead of logging.

  • sanic: update instrumentation to support version 21.

  • sanic: Don’t send non-500s error traces.

  • Fixed the handling of sanic endpoint paths with non-string arguments.

  • sanic: use path parameter names instead of parameter values for the resource.

  • Fixed the optional argument of Span.finish to Optional[float] instead of Optional[int].

  • django, DRF, ASGI: fix span type for web request spans.

  • requests: fix split-by-domain service name when multiple @ signs are present in the url

  • sqlite3: fix error when using connection.backup method.

  • starlette: Add back removed ddtrace.contrib.starlette.get_resource and ddtrace.contrib.starlette.span_modifier.

  • fastapi: Add back removed ddtrace.contrib.fastapi.span_modifier.

  • starlette: unpatch calls correctly.

  • Ensure tornado spans are marked as an error if the response status code is 500 <= x < 600.

  • internal: fix exception raised for invalid values of DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH.

  • Fixes span id tagging in lock profiling.

  • Fixes exceptions raised when logging during tracer initialization when DD_LOGS_INJECTION is enabled.

  • The ddtrace.utils.wrappers.unwrap function now raises an error if trying to unwrap a non-wrapped object.

  • Fixes import path to not reference __init__. This could otherwise be a problem for mypy.

  • Configure a writer thread in a child process after forking based on writer configuration from its parent process.

  • Fix urllib3 patching not properly activating the integration.

  • Fix Pyramid caller_package level issue which resulted in crashes when starting Pyramid applications. Level now left at default (2).

  • flask: fix memory leak of sampled out traces.

  • flask: fix a bug when the query string would contain non-Unicode characters

  • flask_caching: fix redis tagging after the v2.0 release.

  • flask: fix resource naming of request span when errors occur in middleware.

  • Patch pynamodb on import to prevent patching conflicts with gevent.

  • gRPC client spans are now marked as measured by default.

  • Fixes issue of unfinished spans when response is not a grpc.Future but has the same interface, as is the case with the base future class in google-api-core.

  • Fix issue when httpx service name is bytes.

  • Change dependency from backport_ipaddress` to ``ipaddress. Only install ipaddress for Python < 3.7.

  • Fixes build issues on older MacOS versions by updating libddwaf to 1.0.18

  • tracer.get_log_correlation_context(): use active context in addition to

    active span. Formerly just the span was used and this would break cross execution log correlation as a context object is used for the propagation.

  • gevent: disable gevent after fork hook which could result in a performance regression.

  • Fix pymongo 3.12.0+ spans not being generated.

  • profiling: internal use of RLock needs to ensure original threading locks are used rather than gevent threading lock. Because of an indirection in the initialization of the original RLock, we end up getting an underlying gevent lock. We work around this behavior with gevent by creating a patched RLock for use internally.

  • opentracer: don’t override default tracing config for the ENABLED,

    AGENT_HOSTNAME, AGENT_HTTPS or AGENT_PORT settings.

  • opentracer: update set_tag and set_operation_name to return a

    reference to the span to match the OpenTracing spec.

  • Performance of the Celery integration has been improved.

  • pin protobuf to version >=3,<4 due to incompatibility with version 4.21.

  • Pin protobuf version to <3.18 for Python <=3.5 due to support being dropped.

  • Pin setup_requires dependency setuptools_scm[toml]>=4,<6.1 to avoid breaking changes.

  • Fix a bug in the heap profiler that could be triggered if more than 2^16 memory items were freed during heap data collection.

  • Fix a possible bug in the heap memory profiler that could trigger an overflow when too many allocations were being tracked.

  • profiler: Remove lock for data structure linking threads to spans to avoid deadlocks with the trade-off of correctness of spans linked to threads by stack profiler at a given point in time.

  • The CPU profiler now reports the main thread CPU usage even when asyncio tasks are running.

  • Fixes a performance issue with the profiler when used in an asyncio application.

  • The profiler won’t deadlock on fork when gevent monkey patch is enabled.

  • Fix CPU time and wall time profiling not ignoring the profiler tasks with gevent.

  • Fixes wrong numbers of memory allocation being reported in the memory profiler.

  • Fix a formatting issue on error exporting profiles.

  • The profiler now copy all user-provided attributes on fork.

  • Fix UDS upload for profiling not using the correct path.

  • In certain circumstances, the profiles generated in a uWSGI application could have been empty. This is now fixed and the profiler records correctly the generated events.

  • Fix an issue in the heap profiler where it would iterate on the wrong heap allocation tracker.

  • Fixed an issue in profiling exporting profiles twice when forking.

  • Make sure that correct endpoint name collected for profiling.

  • profiling: restart automatically on all Python versions.

  • profiling: fixes an issue with Gunicorn child processes not storing profiling events.

  • profiling: fix a possible deadlock due to spans being activated unexpectedly.

  • The default agent timeout for profiling has been restored from 2 to 10 seconds to avoid too many profiles from being dropped.

  • The profiler now updates the trace resource when capturing span information with the stack and lock collectors. That means that if the trace resource changes after the profiling events are created, the profiler samples will also be updated. This avoids having trace resource being empty when profiling, e.g., WSGI middleware.

  • Fix issue with missing traces when using pymemcache.client.hash.HashClient.

  • pymongo: fix write_command being patched with the wrong method signature.

  • Added missing pymongo integration configuration, which allows overriding the service name for all the emitted spans.

  • Pymongo instrumentation raises an AttributeError when tracer.enabled == False

  • Fix double patching of pymongo client topology.

  • Set the correct package name in the Pyramid instrumentation. This should fix an issue where the incorrect package name was being used which would crash the application when trying to do relative imports within Pyramid (e.g. when including routes from a relative path).

  • pytest: Add note for disabling ddtrace plugin as workaround for side-effects

  • pytest: fix unsafe access to xfail reason.

  • redis: create default Pin on asyncio client. Not having a Pin was resulting in no traces being produced for the async redis client.

  • flask: add support for flask v2.3. Remove deprecated usages of flask._app_ctx_stack and flask._request_ctx_stack.

  • core: fix race condition in TracerTagCollector.

  • Fix runtime-id and system.pid tags not being set on distributed traces.

  • tornado: handle when the current span is None in log_exception().

  • requests: fix TracedSession when patches are not applied.

  • Set required header to indicate top level span computation is done in the client to the Datadog agent. This fixes an issue where spans were erroneously being marked as top level when partial flushing or in certain asynchronous applications.

    The impact of this bug is the unintended computation of stats for non-top level spans.

  • Only for CI Visibility (pytest integration): Fix extraction of branch in GitLab CI.

  • Only for CI Visibility (pytest integration): Fix calculation of pipeline URL for GitHub Actions.

  • A workaround is provided for the problem with uWSGI worker processes failing to respawn. This can occur when using ddtrace-run for automatic instrumentation and configuration or manual instrumentation and configuration without the necessary uWSGI options. The problem is caused by how the tracer can end up starting threads in the master process before uWSGI forks to initialize the workers processes. To avoid this, we have provided updated instructions for enabling tracing in the library uWSGI documentation.

  • Fix a possible NoneType error in the WSGI middleware start_response method.

  • wsgi: ignore GeneratorExit Exception in wsgi.response spans

  • wsgi: ensures resource and http tags are always set on wsgi.request spans.

  • wsgi: when using more than one nested wsgi traced middleware in the same app ensure wsgi spans have the correct parenting.

Other Notes

  • Added runtime metrics status and sampling rules to start-up logs.

  • tracing: add http.route tag to root span for Flask framework.

  • The botocore integration excludes AWS endpoint call parameters that have a name ending with Body from the set of span tags.

  • Added environment variable DD_BOTTLE_DISTRIBUTED_TRACING to enable distributed tracing for bottle.

  • The pytest plugin now includes git metadata tags including author name and email as well as commit message from CI provider environments.

  • The attrs library has been unvendored and is now required as a normal Python dependency with a minimum version requirement of 19.2.0.

  • The six library has been removed from vendor and the system-wide version is being used. It requires version 1.12.0 or later.

  • tracing/internal: disable Datadog internal tag propagation

  • Documentation on how to use Gunicorn with the gevent worker class has been added.

  • Added environment variable DD_FALCON_DISTRIBUTED_TRACING to enable distributed tracing for falcon.

  • When extracting context information from HTTP headers, a new context is created when the trace ID is either 0 or not available within the headers.

  • perf: don’t encode default parent_id value.

  • The logic behind the header extraction for distributed tracing has been improved.

  • profiling: add support for protobuf >=4.0.

  • profiler: don’t initialize the AsyncioLockCollector unless asyncio is

    available. This prevents noisy logs messages from being emitted in Python 2.

  • The default connection timeout for the profiling agent has now been reduced from 10 to 2 seconds to match the tracer behavior.

  • The profiler won’t be ignoring its own resource usage anymore and will report it in the profiles.

  • The default maximum CPU time used for the stack profiler (CPU time, wall time and exceptions profiling) has been decreased from 2% to 1%.

  • The tracemalloc memory profiler, which was disabled by default, has been removed.

  • The default number of events kept by the profiler has been reduced to decreased CPU and memory overhead.

  • Added environment variable DD_PYLONS_DISTRIBUTED_TRACING to enable distributed tracing for pylons.

  • Update pymemcache test suite to test latest versions.

  • Added config.pyramid.distributed_tracing setting to integration config for pyramid.

  • The ddtrace.payload submodule has been removed.

  • Start-up logs are now disabled by default. To enable start-up logs use DD_TRACE_STARTUP_LOGS=true or DD_TRACE_DEBUG=true.

  • Added environment variable DD_TORNADO_DISTRIBUTED_TRACING to enable distributed tracing for tornado.

  • docs: Added troubleshooting section for missing error details in the root span of a trace.

  • Query strings are stripped out from URLs by default when setting URL metadata on a span. This change affects all integrations that store HTTP metadata, like aiohttp, falcon, requests, urllib3.