FirecREST logging¶
FirecREST runs on the Uvicorn web server from which it inherits the Python3 logging configuration.
By default, the FirecREST logger is disabled and the standard Uvicorn logging is applied. To enable it, and accept customized types of loggers and formatters, a configuration YAML file must be mounted into the firecrest-v2 container and the environment variable UVICORN_LOG_CONFIG shall be set pointing to it.
To enable the log in FirecREST configuration, set to true the logger.enable_tracing_log configuration variable.
Logging config file¶
Sysadmins can use the following file as a starting point to configure the FirecREST deployment:
Logging config file example
version: 1
disable_existing_loggers: False
formatters:
default:
# "()": uvicorn.logging.DefaultFormatter
format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
access:
# "()": uvicorn.logging.AccessFormatter
format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
jsonformatter:
format: '%(asctime)s %(levelname)s %(name)s %(message)s'
class: pythonjsonlogger.jsonlogger.JsonFormatter
handlers:
default:
formatter: default
class: logging.StreamHandler
stream: ext://sys.stderr
access:
formatter: access
class: logging.StreamHandler
stream: ext://sys.stdout
json_handler:
formatter: jsonformatter
class: logging.StreamHandler
stream: ext://sys.stdout
loggers:
uvicorn.error:
level: INFO
handlers:
- default
propagate: no
uvicorn.access:
level: INFO
handlers:
- access
propagate: no
f7t_v2_tracing_log:
level: INFO
handlers:
- json_handler
propagate: no
root:
level: ERROR
handlers:
- default
propagate: no
Tracing logger¶
The format of the logs is defined in the configuration file above. As a standard FirecREST uses JSON formatted messages.
The f7t_v2_tracing_log currently supports the following messages:
Format for HTTP backend¶
When the backend service that FirecREST is interfacing (for instance, SLURM RESTd abstraction), the format includes in the backend field the queried url and the HTTP response_status of the HTTP response.
FirecREST Log format for HTTP backend
{
"asctime": "2025-04-15 16:51:57,598",
"levelname": "INFO",
"name": "f7t_v2_tracing_log",
"message": "",
"username": "fireuser",
"system_name": "cluster-slurm-api",
"endpoint": "/status/cluster-slurm-api/partitions",
"resource": "status",
"status_code": 200,
"user_agent": "PostmanRuntime/7.43.2",
"backend": {
"url": "http://192.168.240.2:6820/slurm/v0.0.42/partitions",
"response_status": "200"
}
}
Log output examples¶
Sample output for JSON formatter (pythonjsonlogger.jsonlogger.JsonFormatter)
firecrest-1 | {"asctime": "2025-04-15 16:46:09,900", "levelname": "INFO", "name": "f7t_v2_tracing_log", "message": "", "username": "fireuser", "endpoint": "/status/systems", "resource": "status", "status_code": 200, "user_agent": "PostmanRuntime/7.43.2"}
firecrest-1 | {"asctime": "2025-04-15 16:47:06,075", "levelname": "INFO", "name": "f7t_v2_tracing_log", "message": "", "username": "fireuser", "system_name": "cluster", "endpoint": "/status/cluster/partitions", "resource": "status", "status_code": 404, "user_agent": "PostmanRuntime/7.43.2"}
firecrest-1 | {"asctime": "2025-04-15 16:51:20,278", "levelname": "INFO", "name": "f7t_v2_tracing_log", "message": "", "username": "fireuser", "system_name": "cluster", "endpoint": "/status/cluster/userinfo", "resource": "status", "status_code": 404, "user_agent": "PostmanRuntime/7.43.2"}
firecrest-1 | {"asctime": "2025-04-15 16:51:57,598", "levelname": "INFO", "name": "f7t_v2_tracing_log", "message": "", "username": "fireuser", "system_name": "cluster-slurm-api", "endpoint": "/status/cluster-slurm-api/partitions", "resource": "status", "status_code": 200, "user_agent": "PostmanRuntime/7.43.2", "backend": {"url": "http://192.168.240.2:6820/slurm/v0.0.42/partitions", "response_status": "200"}}
firecrest-1 | {"asctime": "2025-04-15 16:52:19,457", "levelname": "INFO", "name": "f7t_v2_tracing_log", "message": "", "username": "fireuser", "system_name": "cluster-slurm-cli", "endpoint": "/status/cluster-slurm-cli/partitions", "resource": "status", "status_code": 404, "user_agent": "PostmanRuntime/7.43.2"}
firecrest-1 | {"asctime": "2025-04-15 16:52:25,506", "levelname": "INFO", "name": "f7t_v2_tracing_log", "message": "", "username": "fireuser", "system_name": "cluster-slurm-ssh", "endpoint": "/status/cluster-slurm-ssh/partitions", "resource": "status", "status_code": 200, "user_agent": "PostmanRuntime/7.43.2", "backend": {"command": "scontrol -a show -o partitions", "exit_status": "0"}}
Format for SSH backend¶
On the other hand, if FirecREST is executing a command via the SSH Client, then the backend field provides the actual command to be executed, and the POSIX exit_status of the command.
FirecREST Log format for SSH backend
{
"asctime": "2025-04-15 16:52:25,506",
"levelname": "INFO",
"name": "f7t_v2_tracing_log",
"message": "",
"username": "fireuser",
"system_name": "cluster-slurm-ssh",
"endpoint": "/status/cluster-slurm-ssh/partitions",
"resource": "status",
"status_code": 200,
"user_agent": "PostmanRuntime/7.43.2",
"backend": {
"command": "scontrol -a show -o partitions",
"exit_status": "0"
}
}
Log output examples¶
Sample output for Default formatter
2025-04-15 16:57:20,908 - f7t_v2_tracing_log - INFO - {'username': 'fireuser', 'system_name': 'cluster-slurm-ssh', 'endpoint': '/status/cluster-slurm-ssh/partitions', 'resource': 'status', 'status_code': 200, 'user_agent': 'PostmanRuntime/7.43.2', 'backend': {'command': 'scontrol -a show -o partitions', 'exit_status': '0'}}
2025-04-15 16:59:46,478 - f7t_v2_tracing_log - INFO - {'username': 'fireuser', 'system_name': 'cluster-slurm-ss', 'endpoint': '/status/cluster-slurm-ss/partitions', 'resource': 'status', 'status_code': 404, 'user_agent': 'PostmanRuntime/7.43.2'}
2025-04-15 16:59:57,159 - f7t_v2_tracing_log - INFO - {'username': 'fireuser', 'system_name': 'cluster-slurm-ssh', 'endpoint': '/status/cluster-slurm-ssh/nodes', 'resource': 'status', 'status_code': 200, 'user_agent': 'PostmanRuntime/7.43.2', 'backend': {'command': "sinfo -N --noheader --format='%z|%c|%O|%e|%f|%N|%o|%n|%T|%R|%w|%v|%m|%C'", 'exit_status': '0'}}
Response's header tracing logger¶
When tracing logs are enabled, you can record a list of specific headers from incoming HTTP requests by configuring the loggable_request_headers field in the logger section of the YAML file.
Configure the Response header tracing logger
A valid header key in the input field tells the logger to extract that header from the incoming HTTP request and inserts it in the JSON log format using the value of the output field as key.
If loggable_request_headers is omitted, header tracing is disabled. By default, FirecREST does not log any headers other than those explicitly listed.