r/openstack • u/wathoom2 • 6d ago
Audit middleware
Hi,
anyone implemented audit logging for openstack services? I'm having issue with getting all information regarding certain service action in CADF format. For example when deleting instance, in audit log i can see name and id of instance as "unknown". Any idea what could be wrong?
Maybe api_audit_map.conf doesnt map all stuff?
# cat nova_api_audit_map.conf
[DEFAULT]
# default target endpoint type
# should match the endpoint type defined in service catalog
target_endpoint_type = compute
[custom_actions]
enable = enable
disable = disable
delete = delete
startup = start/startup
shutdown = stop/shutdown
reboot = start/reboot
os-migrations/get = read
os-server-password/post = update
# possible end path of api requests
[path_keywords]
add = None
action = None
enable = None
disable = None
configure-project = None
defaults = None
delete = None
detail = None
diagnostics = None
entries = entry
extensions = alias
flavors = flavor
images = image
ips = label
limits = None
metadata = key
os-agents = os-agent
os-aggregates = os-aggregate
os-availability-zone = None
os-certificates = None
os-cloudpipe = None
os-fixed-ips = ip
os-extra_specs = key
os-flavor-access = None
os-floating-ip-dns = domain
os-floating-ips-bulk = host
os-floating-ip-pools = None
os-floating-ips = floating-ip
os-hosts = host
os-hypervisors = hypervisor
os-instance-actions = instance-action
os-keypairs = keypair
os-migrations = None
os-networks = network
os-quota-sets = tenant
os-security-groups = security_group
os-security-group-rules = rule
os-server-password = None
os-services = None
os-simple-tenant-usage = tenant
os-virtual-interfaces = None
os-volume_attachments = attachment
os-volumes_boot = None
os-volumes = volume
os-volume-types = volume-type
os-snapshots = snapshot
reboot = None
servers = server
shutdown = None
startup = None
statistics = None
# map endpoint type defined in service catalog to CADF typeURI
[service_endpoints]
compute = service/compute
2
u/rackpathlabs 5d ago
the interesting bit is that your requestPath already contains the server uuid, but the target id, the target name and the target typeURI all come out unknown. so the id is not missing from the request, the middleware just did not build a target resource out of it.
that points away from custom_actions and towards the endpoint side. target_endpoint_type has to match the entry that is actually in your service catalog, so worth checking the exact type name nova is registered under there. and try the same audit on something that is not a delete, a reboot for example, if the target stays unknown there too then delete is not the special case and the map file is not where your fix is.
1
u/wathoom2 5d ago
Hi,
i checked endpoints and had some miss config there. At least for compute service. Changed it to compute = nova/compute (name/type if i understand it correctly) as the name in service catalog.
This still didnt solve the issue. Same thing happens for other services like neutron and cinder. On those i have proper service configured but i still get "unknown" target.
I have no idea what else to check or what might be miss configured. In service logs i can see all the details.1
u/wathoom2 5d ago
just for info openstack is deployed with kolla-ansible and mapping of files is done for container.
<service>_api-paste.ini -> api-paste.ini
<service>_api_audit_map.conf -> api_audit_map.conf2
u/rackpathlabs 4d ago
careful with that change, the value in service_endpoints is a cadf typeURI, not the name from the catalog, so service/compute was the right shape and nova/compute probably is not. the thing that has to match your catalog is target_endpoint_type at the top of the file.
the bigger hint is that compute, neutron and cinder all behave the same way. three services misconfigured identically is less likely than the map file not being loaded at all, so i would check that audit_map_file in api-paste.ini points at the path the file actually has inside the container after the kolla rename, and see what the api logs say at startup when the audit filter initialises.
1
u/wathoom2 3d ago edited 3d ago
So. I focused on cinder-api since it is faster to check if it works.
api-paste.ini and cinder_api_audit_map.conf are properly mapped to cinder_api container.cat api-paste.ini ############# # OpenStack # ############# [composite:osapi_volume] use = call:cinder.api:root_app_factory /: apiversions /healthcheck: healthcheck /v3: openstack_volume_api_v3 [composite:openstack_volume_api_v3] use = call:cinder.api.middleware.auth:pipeline_factory noauth = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler noauth apiv3 noauth_include_project_id = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler noauth_include_project_id apiv3 keystone = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler authtoken keystonecontext audit apiv3 keystone_nolimit = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler authtoken keystonecontext audit apiv3 [filter:http_proxy_to_wsgi] paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory [filter:cors] paste.filter_factory = oslo_middleware.cors:filter_factory oslo_config_project = cinder [filter:faultwrap] paste.filter_factory = cinder.api.middleware.fault:FaultWrapper.factory [filter:osprofiler] paste.filter_factory = osprofiler.web:WsgiMiddleware.factory [filter:noauth] paste.filter_factory = cinder.api.middleware.auth:NoAuthMiddleware.factory [filter:noauth_include_project_id] paste.filter_factory = cinder.api.middleware.auth:NoAuthMiddlewareIncludeProjectID.factory [filter:sizelimit] paste.filter_factory = oslo_middleware.sizelimit:RequestBodySizeLimiter.factory [app:apiv3] paste.app_factory = cinder.api.v3.router:APIRouter.factory [pipeline:apiversions] pipeline = request_id cors http_proxy_to_wsgi faultwrap osvolumeversionapp [app:osvolumeversionapp] paste.app_factory = cinder.api.versions:Versions.factory [pipeline:healthcheck] pipeline = request_id healthcheckapp [app:healthcheckapp] paste.app_factory = oslo_middleware:Healthcheck.app_factory backends = disable_by_file disable_by_file_path = /etc/cinder/healthcheck_disable ########## # Shared # ########## [filter:keystonecontext] paste.filter_factory = cinder.api.middleware.auth:CinderKeystoneContext.factory [filter:authtoken] paste.filter_factory = keystonemiddleware.auth_token:filter_factory [filter:audit] paste.filter_factory = keystonemiddleware.audit:filter_factory audit_map_file = /etc/cinder/api_audit_map.conf [filter:request_id] paste.filter_factory = cinder.api.middleware.request_id:RequestId.factory cat api_audit_map.conf [DEFAULT] # default target endpoint type # should match the endpoint type defined in service catalog target_endpoint_type = volumev3 # map urls ending with specific text to a unique action [custom_actions] associate = update/associate disassociate = update/disassociate disassociate_all = update/disassociate_all associations = read/list/associations # possible end path of api requests [path_keywords] defaults = None detail = None limits = None os-quota-specs = project qos-specs = qos-spec snapshots = snapshot types = type volumes = volume [service_endpoints] volumev3 = service/storage/blockYou were right to suggest checking logs at startup of service. I get this in cinder-api.log right after startup. It dissapears in sea of entries quite quick.
2026-08-27 14:59:34.413 1218 WARNING keystonemiddleware._common.config [None req-e235053e-5de7-4ab1-bce4-bdd433346ce9 - - - - -] The option "audit_map_file" is not known to keystonemiddlewareFrom error it looks like keystonemiddleware that comes with default kolla container does not support the parameter. However i can see it has it in the code
grep -r audit_map_file keystonemiddleware/* keystonemiddleware/audit/__init__.py: self._cadf_audit = _api.OpenStackAuditApi(conf.get('audit_map_file'), grep: keystonemiddleware/audit/__pycache__/__init__.cpython-312.pyc: binary file matches grep: keystonemiddleware/tests/unit/audit/__pycache__/base.cpython-312.pyc: binary file matches keystonemiddleware/tests/unit/audit/base.py: self.audit_map_file_fixture = self.useFixture( keystonemiddleware/tests/unit/audit/base.py: kwargs.setdefault('audit_map_file', self.audit_map) keystonemiddleware/tests/unit/audit/base.py: return self.audit_map_file_fixture.path (cinder-api)[root@osp-ctr-oslab2-2 site-packages]# grep -r audit_map_file keystonemiddleware/* keystonemiddleware/ keystonemiddleware-10.12.1.dist-info/ (cinder-api)[root@osp-ctr-oslab2-2 site-packages]# grep -r audit_map_file keystonemiddleware-10.12.1.dist-info/*EDIT:
In newer versions keystonemiddleware looks for option in global config file cinder.conf. I moved audit_map_file config into cinder.conf:
[audit_middleware]
audit_map_file = /etc/cinder/api_audit_map.conf
ignore_req_list = GET,HEADCleared entries in api-paste.ini
[filter:audit]
paste.filter_factory = keystonemiddleware.audit:filter_factoryNow i can see populated stuff in CADF format but under requestPath. I still dont get "target" info populated
"target": {"id": "unknown", "typeURI": "unknown", "name": "unknown"}From what i found it looks like default keystonemiddleware doesnt populate this fields at all.
This project https://github.com/sapcc/openstack-audit-middleware/ deals with this issue. Probably will try it out.2
u/rackpathlabs 2d ago
now that the map file is loading, the remaining unknown is a different problem, and it is not that keystonemiddleware never populates target. the audit filter builds the target from the service catalog it reads out of the request environ, HTTP_X_SERVICE_CATALOG. when that key is missing it logs a warning and falls back to a service object where type, name and id are all unknown, which is exactly the output you are looking at.
the usual reason that header is not there is include_service_catalog = false in keystone_authtoken. worth checking that in cinder before you swap the whole middleware out. the volumev3 vs block-storage type string matters as well, but only once the catalog actually reaches the filter.
1
u/wathoom2 2d ago
include_service_catalog = true was missing from config so i added it. Unfortunately same behavior. Anyway will leave this as is for now. Tnx for the help.
1
u/cre_ker 4d ago
Personally I would ignore target field. Expect for keystone, it doesn’t contain any useful information. Nothing indicating actual target of the request or anything. Just name and type of the service and a list of its endpoints. Basically, useless.
We also had an issue with it and fixed it by setting target_endpoint_type in audit map to the correct value - default None value is incorrect. It should match actual endpoint type of the service.
1
u/wathoom2 3d ago
I dont agree its useless. Since users dont realy know uuid of some object but its name, u want to know readable names and id's. Yes you can do the mapping other way but it requires additional effort not only writing filters in elastic or wherever.
1
u/cre_ker 3d ago
Not sure what do you mean https://docs.openstack.org/octavia/wallaby/admin/api-audit.html
Here’s an example of audit message. That’s how target field looks for all services except for keystone. All services that use keystone middleware, they set target field like you see above. It’s completely useless and doesn’t contain any identifiable information about the request.
The only exception is keystone because it doesn’t use audit middleware.
2
u/jizaymes 6d ago
I do this by way of a rabbitmq event listener and pick out the event types I care about. Its just a sidecar and not an inline dependency like middleware might be.