OTEL_SDK_DISABLED=true combined with LogExporting causes: RecursionError: maximum recursion depth exceeded while calling a Python object

[jabbera]

Describe your environment

OS: Ubuntu 22.05
Python version: 3.10.12
SDK version: 1.28.0
API version: 1.28.0

What happened?

Trying to run the log exporter example code here: https://github.com/open-telemetry/opentelemetry-python/blob/06809f4e6a7084f7c79a1fd9e974fad9641a6b86/docs/examples/logs/example.py

with the environment variable: OTEL_SDK_DISABLED=true

raises a recursion error.

It looks like the act of trying to log the warning: "SDK is disabled." is causing a logger to be created, which is in then causing the warning: "SDK is disabled." to be logged, which is causing a logger to be created, repeat forever and ever.

Steps to Reproduce

pip install opentelemetry-sdk==1.28.0

Run the following code

import os

os.environ["OTEL_SDK_DISABLED"] = "true"

import logging
from unittest.mock import Mock
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk.trace import TracerProvider

logger_provider = LoggerProvider()
set_logger_provider(logger_provider)

logger_provider.add_log_record_processor(Mock())
handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)

# Attach OTLP handler to root logger
logging.getLogger().addHandler(handler)
# Trace context correlation
tracer = TracerProvider().get_tracer(__name__)

Expected Result

No exception

Actual Result

Get the following exception:

  File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/jabbera/temp/otel/.venv/lib/python3.10/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 578, in emit
    logger = get_logger(record.name, logger_provider=self._logger_provider)
  File "/home/jabbera/temp/otel/.venv/lib/python3.10/site-packages/opentelemetry/_logs/_internal/__init__.py", line 287, in get_logger
    return logger_provider.get_logger(
  File "/home/jabbera/temp/otel/.venv/lib/python3.10/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 695, in get_logger
    _logger.warning("SDK is disabled.")
  File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/jabbera/temp/otel/.venv/lib/python3.10/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 578, in emit
    logger = get_logger(record.name, logger_provider=self._logger_provider)
  File "/home/jabbera/temp/otel/.venv/lib/python3.10/site-packages/opentelemetry/_logs/_internal/__init__.py", line 287, in get_logger
    return logger_provider.get_logger(
  File "/home/jabbera/temp/otel/.venv/lib/python3.10/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 695, in get_logger
    _logger.warning("SDK is disabled.")
  File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1622, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/lib/python3.10/logging/__init__.py", line 1591, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/lib/python3.10/logging/__init__.py", line 288, in __init__
    ct = time.time()
RecursionError: maximum recursion depth exceeded while calling a Python object

Additional context

No response

Would you like to implement a fix?

None

[jabbera]

This appears to have been caused by: #4208

[jabbera]

I shortened the repro above and removed the dependency on opentelemetry-exporter-otlp. Original code copied here for reference:

import os

os.environ["OTEL_SDK_DISABLED"] = "true"

import logging

from opentelemetry import trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
OTLPLogExporter,
)
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
BatchSpanProcessor,
ConsoleSpanExporter,
)

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
BatchSpanProcessor(ConsoleSpanExporter())
)

logger_provider = LoggerProvider(
resource=Resource.create(
{
"service.name": "shoppingcart",
"service.instance.id": "instance-12",
}
),
)
set_logger_provider(logger_provider)

exporter = OTLPLogExporter(insecure=True)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)

Attach OTLP handler to root logger

logging.getLogger().addHandler(handler)

Trace context correlation

tracer = trace.get_tracer(name)

[aabmass]

Thanks for the bug report! We're hoping to fix it with a patch or reverting the culprit and make a patch release ASAP.

More broadly, similar recursion issues have popped up a few times already. I think we need to carefully look at uses of stdlib logging and rework to make recursion impossible.

[jabbera]

Sounds good. We wrap otel configuration in a custom lib so I worked around it.

[xrmx]

@jabbera Could you please give it a try to #4259?