Max level shown:
12025-04-30T06:26:13.499ZDEBGtest_disk_metrics (init_with_steps): Running step start_crdb
22025-04-30T06:26:13.500ZDEBGtest_disk_metrics: Starting CRDB
32025-04-30T06:26:13.562ZINFOtest_disk_metrics: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpVwiJwJ
42025-04-30T06:26:13.562ZINFOtest_disk_metrics: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmpVwiJwJ/data)
52025-04-30T06:26:13.580ZINFOtest_disk_metrics: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpVwiJwJ/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpVwiJwJ/listen-url
62025-04-30T06:26:13.580ZINFOtest_disk_metrics: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=3 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:26:14.913ZINFOtest_disk_metrics: cockroach pid: 21965
82025-04-30T06:26:14.913ZINFOtest_disk_metrics: cockroach listen URL: postgresql://root@[::1]:34295/omicron?sslmode=disable
92025-04-30T06:26:14.913ZDEBGtest_disk_metrics (init_with_steps): Running step start_clickhouse
102025-04-30T06:26:14.913ZDEBGtest_disk_metrics: Starting Clickhouse
112025-04-30T06:26:15.861ZDEBGtest_disk_metrics (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:26:15.861ZDEBGtest_disk_metrics: Starting Management Gateway
132025-04-30T06:26:15.983ZDEBGtest_disk_metrics (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:26:15.983ZDEBGtest_disk_metrics: Starting Dendrite for switch0
152025-04-30T06:26:16.069ZDEBGtest_disk_metrics (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:26:16.072ZDEBGtest_disk_metrics: Starting mgd for switch0
172025-04-30T06:26:16.187ZDEBGtest_disk_metrics: mgd port is 54200
182025-04-30T06:26:16.187ZDEBGtest_disk_metrics (init_with_steps): Running step record_switch_dns
192025-04-30T06:26:16.187ZDEBGtest_disk_metrics: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:26:16.187ZDEBGtest_disk_metrics (init_with_steps): Running step start_internal_dns
212025-04-30T06:26:16.187ZINFOtest_disk_metrics (store): opening sled database path = /var/tmp/omicron_tmp/.tmpBEyEYv
222025-04-30T06:26:16.289ZINFOtest_disk_metrics (store): pruning trees for generations newer than 0
232025-04-30T06:26:16.289ZINFOtest_disk_metrics (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:26:16.290ZINFOtest_disk_metrics (dns): DNS server bound to address kind = dns local_address = [::1]:60986
252025-04-30T06:26:16.290ZDEBGtest_disk_metrics (http): registered endpoint kind = dns local_addr = [::1]:57001 method = GET path = /config versions = all
262025-04-30T06:26:16.291ZDEBGtest_disk_metrics (http): registered endpoint kind = dns local_addr = [::1]:57001 method = PUT path = /config versions = all
272025-04-30T06:26:16.291ZINFOtest_disk_metrics (http): listening kind = dns local_addr = [::1]:57001
282025-04-30T06:26:16.292ZDEBGtest_disk_metrics (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:57001
292025-04-30T06:26:16.293ZDEBGtest_disk_metrics (init_with_steps): Running step start_external_dns
302025-04-30T06:26:16.293ZINFOtest_disk_metrics (store): opening sled database path = /var/tmp/omicron_tmp/.tmpmoLVk1
312025-04-30T06:26:16.350ZINFOtest_disk_metrics (store): pruning trees for generations newer than 0
322025-04-30T06:26:16.350ZINFOtest_disk_metrics (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:26:16.350ZINFOtest_disk_metrics (dns): DNS server bound to address kind = dns local_address = [::1]:41159
342025-04-30T06:26:16.350ZDEBGtest_disk_metrics (http): registered endpoint kind = dns local_addr = [::1]:37651 method = GET path = /config versions = all
352025-04-30T06:26:16.350ZDEBGtest_disk_metrics (http): registered endpoint kind = dns local_addr = [::1]:37651 method = PUT path = /config versions = all
362025-04-30T06:26:16.350ZINFOtest_disk_metrics (http): listening kind = dns local_addr = [::1]:37651
372025-04-30T06:26:16.351ZDEBGtest_disk_metrics (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:37651
382025-04-30T06:26:16.351ZDEBGtest_disk_metrics (init_with_steps): Running step start_nexus_internal
392025-04-30T06:26:16.351ZDEBGtest_disk_metrics: Starting Nexus (internal API)
402025-04-30T06:26:16.352ZINFOtest_disk_metrics: setting up nexus server
412025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Action
422025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Database
462025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Fleet
482025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Inventory
492025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:26:16.355ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Project
582025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Disk
592025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Instance
642025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = IpPool
652025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Vpc
672025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Image
752025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Certificate
802025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Rack
852025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SshKey
872025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Silo
882025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:26:16.356ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Sled
942025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = Service
1012025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:26:16.357ZINFOtest_disk_metrics (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:26:16.428ZINFOtest_disk_metrics (ServerContext): Setting up resolver using DNS address: [::1]:60986
1042025-04-30T06:26:16.428ZINFOtest_disk_metrics (DnsResolver): new DNS resolver addresses = [[::1]:60986]
1052025-04-30T06:26:16.429ZINFOtest_disk_metrics (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:34295/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 34295,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:26:16.430ZDEBGtest_disk_metrics (ServerContext): registered USDT probes
1072025-04-30T06:26:16.530ZINFOtest_disk_metrics (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:26:16.636ZDEBGtest_disk_metrics (clickhouse-client): registered USDT probes id = dbd4bb3a-6b34-4895-b2c3-d9f7b770d8e5
1092025-04-30T06:26:16.714ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /bgtasks versions = all
1102025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /demo-saga versions = all
1202025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:26:16.773ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /metrics/producers versions = all
1442025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /mgs-updates versions = all
1452025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /sagas versions = all
1522025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:26:16.774ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /sleds/add versions = all
1572025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = GET path = /v1/ping versions = all
1612025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:39551 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:26:16.775ZINFOtest_disk_metrics (dropshot_internal): listening local_addr = 127.0.0.1:39551
1642025-04-30T06:26:16.775ZDEBGtest_disk_metrics (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:39551
1652025-04-30T06:26:16.775ZINFOtest_disk_metrics (SEC): SEC running sec_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
1662025-04-30T06:26:16.777ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:26:16.778ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:26:16.782ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:26:16.782ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in users
1702025-04-30T06:26:16.784ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:26:16.784ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:26:16.784ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:26:16.785ZINFOtest_disk_metrics (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:26:16.811ZTRACtest_disk_metrics (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: a6 10 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:54843 req_id = 81d3dd3b-0bb3-45f0-93b3-74ca62497a5e
1752025-04-30T06:26:16.811ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42512,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54843 req_id = 81d3dd3b-0bb3-45f0-93b3-74ca62497a5e
1762025-04-30T06:26:16.814ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42512, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54843 req_id = 81d3dd3b-0bb3-45f0-93b3-74ca62497a5e
1772025-04-30T06:26:16.925ZINFOtest_disk_metrics (DataLoader): created 5 built-in users
1782025-04-30T06:26:16.925ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:26:16.926ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:26:16.928ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:26:16.945ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in roles
1822025-04-30T06:26:16.945ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:26:16.946ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:26:16.946ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:26:16.999ZINFOtest_disk_metrics (DataLoader): created 10 built-in roles
1862025-04-30T06:26:16.999ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:26:17.001ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:26:17.004ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:26:17.004ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:26:17.004ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:26:17.004ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:26:17.004ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:26:17.234ZINFOtest_disk_metrics (DataLoader): created 4 built-in role assignments
1942025-04-30T06:26:17.234ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:26:17.234ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:26:17.235ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:26:17.235ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in silos
1982025-04-30T06:26:17.236ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:26:17.236ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:26:17.236ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:26:17.380ZINFOtest_disk_metrics (DataLoader): created 2 built-in silos
2022025-04-30T06:26:17.380ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:26:17.380ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:26:17.380ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:26:17.392ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:26:17.392ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:26:17.393ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:26:17.393ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in projects
2092025-04-30T06:26:17.394ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:26:17.394ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:26:17.394ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:26:17.398ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:26:17.398ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:26:17.399ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:26:17.399ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:26:17.399ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:26:17.440ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:26:17.440ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:26:17.440ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:26:17.441ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:26:17.446ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:26:17.449ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:26:17.449ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:26:17.449ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:26:17.449ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:26:17.449ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:26:17.450ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:26:17.452ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:26:17.452ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:26:17.452ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:26:17.452ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:26:17.457ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:26:17.461ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:26:17.461ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:26:17.461ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:26:17.461ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:26:17.497ZINFOtest_disk_metrics (DataLoader): created built-in services project
2382025-04-30T06:26:17.497ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:26:17.497ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:26:17.499ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:26:17.499ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:26:17.499ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:26:17.499ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:26:17.500ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:26:17.503ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:26:17.503ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:26:17.504ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:26:17.507ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:26:17.507ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:26:17.507ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:26:17.507ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:26:17.508ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:26:17.510ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:26:17.510ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:26:17.510ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:26:17.511ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:26:17.514ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:26:17.514ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:26:17.514ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:26:17.515ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:26:17.517ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:26:17.523ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:26:17.523ZTRACtest_disk_metrics (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:26:17.523ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:26:17.523ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:26:17.523ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:26:17.523ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:26:17.524ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:26:17.526ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:26:17.526ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:26:17.526ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:26:17.527ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:26:17.529ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:26:17.529ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:26:17.529ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:26:17.529ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:26:17.531ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:26:17.536ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:26:17.536ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:26:17.536ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:26:17.537ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:26:17.556ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:26:17.556ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:26:17.556ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:26:17.556ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:26:17.557ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:26:17.559ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:26:17.559ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:26:17.559ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:26:17.559ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:26:17.562ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:26:17.562ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:26:17.562ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:26:17.562ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:26:17.565ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:26:17.571ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:26:17.571ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:26:17.571ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:26:17.572ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:26:17.580ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:26:17.580ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:26:17.580ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:26:17.587ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:26:17.587ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:26:17.587ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:26:17.604ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:26:17.604ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:26:17.604ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:26:17.604ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:26:17.605ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:26:17.624ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:26:17.624ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:26:17.624ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:26:17.625ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:26:17.649ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:26:17.649ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:26:17.650ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:26:17.652ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:26:17.656ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:26:17.662ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:26:17.668ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:26:17.669ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:26:17.670ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:26:17.720ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:26:17.721ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:26:17.721ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:26:17.721ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:26:17.722ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:26:17.730ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:26:17.733ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:26:17.733ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:26:17.740ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:26:17.742ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:26:17.742ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:26:17.742ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:26:17.743ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:26:17.772ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:26:17.782ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:26:17.782ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:26:17.782ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:26:17.782ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:26:17.854ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:26:17.854ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:26:17.854ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:26:17.855ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:26:17.859ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:26:17.875ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:26:17.877ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:26:17.877ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:26:17.878ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:26:17.884ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:26:17.884ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:26:17.885ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:26:17.885ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:26:17.893ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:26:17.915ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:26:17.915ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:26:17.915ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:26:17.915ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:26:17.922ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:26:17.929ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:26:17.929ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:26:17.930ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:26:17.941ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:26:17.941ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:26:17.942ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:26:17.945ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:26:17.945ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:26:17.945ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:26:17.954ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:26:17.954ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:26:17.954ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:26:17.954ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:26:17.955ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:26:17.961ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:26:17.961ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:26:17.961ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:26:17.962ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:26:17.975ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:26:17.977ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:26:17.977ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:26:17.977ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:26:17.987ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:26:17.996ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:26:17.996ZTRACtest_disk_metrics (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:26:17.996ZTRACtest_disk_metrics (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:26:17.996ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:26:17.996ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:26:17.996ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:26:17.997ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:26:17.999ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:26:17.999ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:26:17.999ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:26:18.001ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:26:18.008ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:26:18.008ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:26:18.008ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:26:18.009ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:26:18.015ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:26:18.022ZDEBGtest_disk_metrics (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:26:18.022ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:26:18.022ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:26:18.023ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:26:18.034ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:26:18.034ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:26:18.034ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:26:18.035ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:26:18.035ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:26:18.039ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:26:18.039ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:26:18.039ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:26:18.041ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:26:18.066ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:26:18.067ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:26:18.067ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:26:18.068ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:26:18.071ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:26:18.077ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:26:18.077ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:26:18.077ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:26:18.077ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:26:18.185ZINFOtest_disk_metrics (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:26:18.185ZDEBGtest_disk_metrics (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:26:18.185ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:26:18.185ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:26:18.185ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:26:18.192ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:26:18.192ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:26:18.193ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:26:18.196ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:26:18.196ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:26:18.196ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:26:18.199ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:26:18.199ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:26:18.199ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:26:18.199ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:26:18.200ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:26:18.213ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:26:18.213ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:26:18.213ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:26:18.213ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:26:18.217ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:26:18.217ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:26:18.217ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:26:18.218ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:26:18.220ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:26:18.226ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:26:18.226ZTRACtest_disk_metrics (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:26:18.226ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:26:18.226ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:26:18.226ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:26:18.232ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:26:18.232ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:26:18.232ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:26:18.232ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:26:18.234ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:26:18.237ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:26:18.237ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:26:18.237ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:26:18.238ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:26:18.246ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:26:18.246ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:26:18.246ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:26:18.247ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:26:18.249ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:26:18.257ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:26:18.269ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:26:18.270ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:26:18.270ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:26:18.270ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:26:18.271ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:26:18.276ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:26:18.276ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:26:18.276ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:26:18.277ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:26:18.282ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:26:18.282ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:26:18.282ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:26:18.284ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:26:18.287ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:26:18.297ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:26:18.297ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:26:18.297ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:26:18.297ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:26:18.348ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:26:18.348ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:26:18.350ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:26:18.354ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:26:18.354ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:26:18.354ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:26:18.354ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:26:18.354ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:26:18.356ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:26:18.356ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:26:18.356ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:26:18.356ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:26:18.358ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:26:18.358ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:26:18.358ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:26:18.359ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:26:18.361ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:26:18.366ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:26:18.370ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:26:18.370ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:26:18.371ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:26:18.371ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:26:18.371ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:26:18.373ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:26:18.373ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:26:18.373ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:26:18.374ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:26:18.376ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:26:18.376ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:26:18.376ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:26:18.377ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:26:18.379ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:26:18.393ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:26:18.393ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:26:18.393ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:26:18.393ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:26:18.400ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:26:18.400ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:26:18.401ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:26:18.404ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:26:18.404ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:26:18.404ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:26:18.404ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:26:18.405ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:26:18.409ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:26:18.409ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:26:18.409ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:26:18.410ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:26:18.412ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:26:18.412ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:26:18.412ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:26:18.414ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:26:18.416ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:26:18.421ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:26:18.426ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:26:18.426ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:26:18.426ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:26:18.426ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:26:18.427ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:26:18.430ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:26:18.430ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:26:18.430ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:26:18.430ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:26:18.433ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:26:18.433ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:26:18.433ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:26:18.434ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:26:18.436ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:26:18.444ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:26:18.444ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:26:18.445ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:26:18.445ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:26:18.458ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:26:18.458ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:26:18.459ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:26:18.463ZINFOtest_disk_metrics (DataLoader): created built-in services vpc subnets
5612025-04-30T06:26:18.463ZINFOtest_disk_metrics (DataLoader): created built-in services vpc
5622025-04-30T06:26:18.463ZTRACtest_disk_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:26:18.463ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:26:18.468ZDEBGtest_disk_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:26:18.468ZDEBGtest_disk_metrics (DataLoader): attempting to create silo users
5662025-04-30T06:26:18.468ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:26:18.468ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:26:18.469ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:26:18.476ZINFOtest_disk_metrics (DataLoader): created 2 silo users
5702025-04-30T06:26:18.476ZDEBGtest_disk_metrics (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:26:18.476ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:26:18.477ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:26:18.477ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:26:18.481ZINFOtest_disk_metrics (DataLoader): created 2 silo user role assignments
5752025-04-30T06:26:18.481ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:26:18.481ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:26:18.483ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:26:18.491ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:26:18.491ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:26:18.492ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:26:18.504ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:26:18.504ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:26:18.504ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:26:18.504ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:26:18.505ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:26:18.508ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:26:18.513ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:26:18.513ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:26:18.513ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:26:18.514ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:26:18.520ZTRACtest_disk_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:26:18.520ZTRACtest_disk_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:26:18.520ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:26:18.520ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:26:18.521ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:26:18.523ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:26:18.525ZDEBGtest_disk_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:26:18.527ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:26:18.528ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:26:18.529ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:26:18.629ZTRACtest_disk_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:26:18.629ZDEBGtest_disk_metrics (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:26:18.631ZDEBGtest_disk_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:26:18.639ZINFOtest_disk_metrics (nexus): populate complete
6052025-04-30T06:26:18.639ZINFOtest_disk_metrics (nexus): activating background tasks
6062025-04-30T06:26:18.651ZDEBGtest_disk_metrics (init_with_steps): Running step start_sled1
6072025-04-30T06:26:18.651ZINFOtest_disk_metrics (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:26:18.730ZINFOtest_disk_metrics (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:26:18.731ZINFOtest_disk_metrics (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:26:18.732ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:26:18.733ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:26:18.733ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:26:18.733ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:26:18.734ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:26:18.734ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:26:18.735ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:26:18.735ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:26:18.735ZWARNtest_disk_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:26:18.735ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 313.544\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:26:18.736ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:26:18.736ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:26:18.736ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:26:18.736ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:26:18.737ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:26:18.737ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:26:18.737ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:26:18.738ZINFOtest_disk_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpSz5UZI/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6282025-04-30T06:26:18.738ZDEBGtest_disk_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpSz5UZI/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6292025-04-30T06:26:18.738ZINFOtest_disk_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpTLbgOh/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6302025-04-30T06:26:18.738ZDEBGtest_disk_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpTLbgOh/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6312025-04-30T06:26:18.791ZDEBGtest_disk_metrics (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:58149 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6322025-04-30T06:26:18.791ZINFOtest_disk_metrics (dropshot (Repo Depot)): listening local_addr = [::1]:58149 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6332025-04-30T06:26:18.792ZDEBGtest_disk_metrics (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:58149 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6342025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6352025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6362025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6372025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6382025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6392025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6402025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6412025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6422025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6432025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6442025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6452025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6462025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6472025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6482025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6492025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6502025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6512025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6522025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6532025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6542025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6552025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6562025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6572025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6582025-04-30T06:26:18.799ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6592025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6602025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6612025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6622025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6632025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6642025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6652025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6662025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6672025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6682025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6692025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6702025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6712025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6722025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6732025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6742025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6752025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6762025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6772025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6782025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6792025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6802025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6812025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6822025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6832025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6842025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6852025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6862025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6872025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6882025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6892025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6902025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6912025-04-30T06:26:18.800ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6922025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6932025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6942025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6952025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6962025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6972025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6982025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6992025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7002025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7012025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7022025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:50989 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7032025-04-30T06:26:18.801ZINFOtest_disk_metrics (dropshot): listening local_addr = [::1]:50989 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7042025-04-30T06:26:18.801ZDEBGtest_disk_metrics (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:50989 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7052025-04-30T06:26:18.801ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
7062025-04-30T06:26:18.801ZWARNtest_disk_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
7072025-04-30T06:26:18.801ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 46.824\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
7082025-04-30T06:26:18.801ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
7092025-04-30T06:26:18.801ZINFOtest_disk_metrics (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:16:18.800850320 UTC
7102025-04-30T06:26:18.801ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
7112025-04-30T06:26:18.801ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
7122025-04-30T06:26:18.802ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
7132025-04-30T06:26:18.802ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
7142025-04-30T06:26:18.802ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
7152025-04-30T06:26:18.802ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7162025-04-30T06:26:18.802ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
7172025-04-30T06:26:18.802ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7182025-04-30T06:26:18.802ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7192025-04-30T06:26:18.802ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
7202025-04-30T06:26:18.802ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
7212025-04-30T06:26:18.802ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7222025-04-30T06:26:18.803ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7232025-04-30T06:26:18.803ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
7242025-04-30T06:26:18.803ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
7252025-04-30T06:26:18.803ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7262025-04-30T06:26:18.803ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7272025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
7282025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
7292025-04-30T06:26:18.804ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
7302025-04-30T06:26:18.804ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7312025-04-30T06:26:18.804ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
7322025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7332025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7342025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
7352025-04-30T06:26:18.804ZWARNtest_disk_metrics (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
7362025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 50.31\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7372025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
7382025-04-30T06:26:18.804ZWARNtest_disk_metrics (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
7392025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 16.443\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7402025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
7412025-04-30T06:26:18.804ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
7422025-04-30T06:26:18.804ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7432025-04-30T06:26:18.804ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
7442025-04-30T06:26:18.804ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7452025-04-30T06:26:18.805ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7462025-04-30T06:26:18.805ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
7472025-04-30T06:26:18.806ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
7482025-04-30T06:26:18.806ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7492025-04-30T06:26:18.806ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
7502025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7512025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7522025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
7532025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 33.898\u{b5}s iteration = 1 status = {"error":"task disabled"}
7542025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
7552025-04-30T06:26:18.806ZWARNtest_disk_metrics (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
7562025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 42.495\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7572025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
7582025-04-30T06:26:18.806ZINFOtest_disk_metrics (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
7592025-04-30T06:26:18.806ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 76.593\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
7602025-04-30T06:26:18.807ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
7612025-04-30T06:26:18.807ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
7622025-04-30T06:26:18.807ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:26:18.807ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
7642025-04-30T06:26:18.807ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7652025-04-30T06:26:18.807ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7662025-04-30T06:26:18.807ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
7672025-04-30T06:26:18.807ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
7682025-04-30T06:26:18.807ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7692025-04-30T06:26:18.807ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
7702025-04-30T06:26:18.807ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7712025-04-30T06:26:18.808ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7722025-04-30T06:26:18.808ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
7732025-04-30T06:26:18.808ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
7742025-04-30T06:26:18.808ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7752025-04-30T06:26:18.809ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7762025-04-30T06:26:18.809ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
7772025-04-30T06:26:18.809ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7782025-04-30T06:26:18.809ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7792025-04-30T06:26:18.809ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7802025-04-30T06:26:18.809ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7812025-04-30T06:26:18.809ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7822025-04-30T06:26:18.809ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7832025-04-30T06:26:18.810ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7842025-04-30T06:26:18.810ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7852025-04-30T06:26:18.810ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7862025-04-30T06:26:18.810ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7872025-04-30T06:26:18.810ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7882025-04-30T06:26:18.810ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7892025-04-30T06:26:18.810ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7902025-04-30T06:26:18.810ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7912025-04-30T06:26:18.810ZINFOtest_disk_metrics (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7922025-04-30T06:26:18.810ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = instance_updater elapsed = 283.485\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7932025-04-30T06:26:18.811ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7942025-04-30T06:26:18.811ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7952025-04-30T06:26:18.811ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7962025-04-30T06:26:18.811ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7972025-04-30T06:26:18.811ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7982025-04-30T06:26:18.811ZDEBGtest_disk_metrics (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7992025-04-30T06:26:18.811ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
8002025-04-30T06:26:18.811ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8012025-04-30T06:26:18.812ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8022025-04-30T06:26:18.812ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
8032025-04-30T06:26:18.812ZINFOtest_disk_metrics (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
8042025-04-30T06:26:18.812ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
8052025-04-30T06:26:18.812ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8062025-04-30T06:26:18.812ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
8072025-04-30T06:26:18.812ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8082025-04-30T06:26:18.813ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8092025-04-30T06:26:18.814ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
8102025-04-30T06:26:18.814ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
8112025-04-30T06:26:18.814ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
8122025-04-30T06:26:18.815ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
8132025-04-30T06:26:18.815ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
8142025-04-30T06:26:18.815ZDEBGtest_disk_metrics (SagaRecoverer): listing candidate sagas for recovery
8152025-04-30T06:26:18.815ZTRACtest_disk_metrics (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
8162025-04-30T06:26:18.815ZDEBGtest_disk_metrics (SagaRecoverer): roles roles = RoleSet { roles: {} }
8172025-04-30T06:26:18.817ZDEBGtest_disk_metrics (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
8182025-04-30T06:26:18.817ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
8192025-04-30T06:26:18.817ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
8202025-04-30T06:26:18.817ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
8212025-04-30T06:26:18.817ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
8222025-04-30T06:26:18.817ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
8232025-04-30T06:26:18.817ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
8242025-04-30T06:26:18.818ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8252025-04-30T06:26:18.820ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8262025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
8272025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
8282025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8292025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8302025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
8312025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
8322025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8332025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8342025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
8352025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
8362025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8372025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8382025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
8392025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
8402025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8412025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
8422025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8432025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8442025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
8452025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
8462025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8472025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8482025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
8492025-04-30T06:26:18.821ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
8502025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
8512025-04-30T06:26:18.821ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
8522025-04-30T06:26:18.822ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
8532025-04-30T06:26:18.822ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
8542025-04-30T06:26:18.822ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
8552025-04-30T06:26:18.822ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
8562025-04-30T06:26:18.822ZDEBGtest_disk_metrics (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8572025-04-30T06:26:18.823ZDEBGtest_disk_metrics (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:39551/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8582025-04-30T06:26:18.824ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
8592025-04-30T06:26:18.824ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8602025-04-30T06:26:18.824ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
8612025-04-30T06:26:18.824ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8622025-04-30T06:26:18.824ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8632025-04-30T06:26:18.824ZINFOtest_disk_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:39551 remote_addr = 127.0.0.1:55304
8642025-04-30T06:26:18.825ZTRACtest_disk_metrics (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: c4 f7 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:62356 req_id = 7d8b372b-48e5-4f9c-a1a7-9044e61ded0f
8652025-04-30T06:26:18.825ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50423,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62356 req_id = 7d8b372b-48e5-4f9c-a1a7-9044e61ded0f
8662025-04-30T06:26:18.825ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50423, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62356 req_id = 7d8b372b-48e5-4f9c-a1a7-9044e61ded0f
8672025-04-30T06:26:18.826ZTRACtest_disk_metrics (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 0f 76 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 .v..........._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56413 req_id = b6b65edb-aa8f-4ca3-a0d1-dffeab75f191
8682025-04-30T06:26:18.826ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3958,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56413 req_id = b6b65edb-aa8f-4ca3-a0d1-dffeab75f191
8692025-04-30T06:26:18.826ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3958, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56413 req_id = b6b65edb-aa8f-4ca3-a0d1-dffeab75f191
8702025-04-30T06:26:18.826ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8712025-04-30T06:26:18.826ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8722025-04-30T06:26:18.827ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8732025-04-30T06:26:18.827ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8742025-04-30T06:26:18.828ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8752025-04-30T06:26:18.828ZWARNtest_disk_metrics (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8762025-04-30T06:26:18.828ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 93.723127ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8772025-04-30T06:26:18.828ZWARNtest_disk_metrics (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:26:18.828ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 91.698933ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:26:18.829ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:55304 req_id = a6bca091-ac5d-4634-8294-e55d96241dd4 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:26:18.831ZINFOtest_disk_metrics (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:26:18.841ZINFOtest_disk_metrics (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:16:18.800850320 UTC pruned = {}
8822025-04-30T06:26:18.841ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 40.994459ms iteration = 1 status = {"expiration":"2025-04-30T06:16:18.800850320Z","pruned":[]}
8832025-04-30T06:26:18.875ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:26:18.875ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8852025-04-30T06:26:18.879ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: de 64 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .d..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56343 req_id = 7c1fa730-cfaf-4c70-aafb-f0d1305808fd
8862025-04-30T06:26:18.880ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56932,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56343 req_id = 7c1fa730-cfaf-4c70-aafb-f0d1305808fd
8872025-04-30T06:26:18.881ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56932, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56343 req_id = 7c1fa730-cfaf-4c70-aafb-f0d1305808fd
8882025-04-30T06:26:18.881ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: b0 65 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .e..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58679 req_id = cdcd7948-2473-43ad-9ce3-e9108691388e
8892025-04-30T06:26:18.882ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45157,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58679 req_id = cdcd7948-2473-43ad-9ce3-e9108691388e
8902025-04-30T06:26:18.883ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45157, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58679 req_id = cdcd7948-2473-43ad-9ce3-e9108691388e
8912025-04-30T06:26:18.883ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8922025-04-30T06:26:18.890ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8932025-04-30T06:26:18.890ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
8942025-04-30T06:26:18.890ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8952025-04-30T06:26:18.891ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8962025-04-30T06:26:18.896ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 92.162347ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8972025-04-30T06:26:18.898ZERROtest_disk_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8982025-04-30T06:26:18.898ZERROtest_disk_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8992025-04-30T06:26:18.898ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 96.108209ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9002025-04-30T06:26:18.899ZERROtest_disk_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9012025-04-30T06:26:18.899ZERROtest_disk_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9022025-04-30T06:26:18.899ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 95.928111ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9032025-04-30T06:26:18.928ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9042025-04-30T06:26:18.935ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9052025-04-30T06:26:18.936ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9062025-04-30T06:26:18.936ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9072025-04-30T06:26:18.937ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9082025-04-30T06:26:18.938ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-04-30T06:26:18.953ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9102025-04-30T06:26:18.953ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9112025-04-30T06:26:18.953ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9122025-04-30T06:26:18.954ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9132025-04-30T06:26:19.057ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9142025-04-30T06:26:19.064ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9152025-04-30T06:26:19.064ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9162025-04-30T06:26:19.064ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9172025-04-30T06:26:19.066ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9182025-04-30T06:26:19.066ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9192025-04-30T06:26:19.074ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9202025-04-30T06:26:19.079ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9212025-04-30T06:26:19.079ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9222025-04-30T06:26:19.080ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9232025-04-30T06:26:19.080ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242025-04-30T06:26:19.084ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9252025-04-30T06:26:19.084ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9262025-04-30T06:26:19.084ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9272025-04-30T06:26:19.086ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9282025-04-30T06:26:19.086ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9292025-04-30T06:26:19.092ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9302025-04-30T06:26:19.092ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9312025-04-30T06:26:19.092ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9322025-04-30T06:26:19.093ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9332025-04-30T06:26:19.097ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9342025-04-30T06:26:19.097ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9352025-04-30T06:26:19.099ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9362025-04-30T06:26:19.099ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9372025-04-30T06:26:19.099ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9382025-04-30T06:26:19.099ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9392025-04-30T06:26:19.132ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9402025-04-30T06:26:19.132ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9412025-04-30T06:26:19.133ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9422025-04-30T06:26:19.289ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 471.748819ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9432025-04-30T06:26:19.294ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9442025-04-30T06:26:19.300ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9452025-04-30T06:26:19.300ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9462025-04-30T06:26:19.300ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9472025-04-30T06:26:19.300ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9482025-04-30T06:26:19.300ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9492025-04-30T06:26:19.301ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9502025-04-30T06:26:19.523ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 701.054024ms iteration = 1 status = {"by_rx":{},"error":null}
9512025-04-30T06:26:19.525ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9522025-04-30T06:26:19.529ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9532025-04-30T06:26:19.529ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9542025-04-30T06:26:19.529ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9552025-04-30T06:26:19.529ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9562025-04-30T06:26:19.529ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9572025-04-30T06:26:19.529ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9582025-04-30T06:26:19.530ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9592025-04-30T06:26:19.531ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9602025-04-30T06:26:19.531ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9612025-04-30T06:26:19.534ZTRACtest_disk_metrics (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9622025-04-30T06:26:19.534ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9632025-04-30T06:26:19.534ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9642025-04-30T06:26:19.535ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9652025-04-30T06:26:19.535ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9662025-04-30T06:26:19.535ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9672025-04-30T06:26:19.536ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9682025-04-30T06:26:19.550ZDEBGtest_disk_metrics (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9692025-04-30T06:26:19.550ZINFOtest_disk_metrics (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9702025-04-30T06:26:19.550ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 735.917518ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9712025-04-30T06:26:19.573ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9722025-04-30T06:26:19.573ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9732025-04-30T06:26:19.573ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9742025-04-30T06:26:19.573ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9752025-04-30T06:26:19.573ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9762025-04-30T06:26:19.600ZINFOtest_disk_metrics (SagaRecoverer): listed in-progress sagas count = 0
9772025-04-30T06:26:19.601ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 785.59147ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9782025-04-30T06:26:19.604ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9792025-04-30T06:26:19.610ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9802025-04-30T06:26:19.611ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 790.423789ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9812025-04-30T06:26:19.611ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 792.645515ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9822025-04-30T06:26:19.611ZWARNtest_disk_metrics (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9832025-04-30T06:26:19.611ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 807.526123ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9842025-04-30T06:26:19.613ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 806.063489ms iteration = 1 status = {}
9852025-04-30T06:26:19.614ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 795.222569ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9862025-04-30T06:26:19.617ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9872025-04-30T06:26:19.622ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9882025-04-30T06:26:19.631ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9892025-04-30T06:26:19.631ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9902025-04-30T06:26:19.632ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9912025-04-30T06:26:19.682ZWARNtest_disk_metrics (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
9922025-04-30T06:26:19.682ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 875.083678ms iteration = 1 status = {"error":"inventory collection is None"}
9932025-04-30T06:26:19.682ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9942025-04-30T06:26:19.682ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9952025-04-30T06:26:19.683ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 23 91 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 #............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64969 req_id = baeca0f2-e4fc-46d9-b747-5241ec0c5381
9962025-04-30T06:26:19.683ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9105,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64969 req_id = baeca0f2-e4fc-46d9-b747-5241ec0c5381
9972025-04-30T06:26:19.683ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9105, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64969 req_id = baeca0f2-e4fc-46d9-b747-5241ec0c5381
9982025-04-30T06:26:19.684ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9992025-04-30T06:26:19.684ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 879.558132ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
10002025-04-30T06:26:19.755ZWARNtest_disk_metrics (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
10012025-04-30T06:26:19.755ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 1.019589535s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
10022025-04-30T06:26:19.756ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10032025-04-30T06:26:19.756ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10042025-04-30T06:26:19.757ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10052025-04-30T06:26:19.765ZINFOtest_disk_metrics (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10062025-04-30T06:26:19.765ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 955.440603ms iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10072025-04-30T06:26:19.769ZWARNtest_disk_metrics (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
10082025-04-30T06:26:19.769ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 1.036414518s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
10092025-04-30T06:26:19.771ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10102025-04-30T06:26:19.776ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10112025-04-30T06:26:19.776ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10122025-04-30T06:26:19.776ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10132025-04-30T06:26:19.777ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10142025-04-30T06:26:19.823ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10152025-04-30T06:26:19.827ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10162025-04-30T06:26:19.827ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10172025-04-30T06:26:19.827ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10182025-04-30T06:26:19.827ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10192025-04-30T06:26:19.828ZTRACtest_disk_metrics (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10202025-04-30T06:26:19.828ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10212025-04-30T06:26:19.828ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10222025-04-30T06:26:19.828ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10232025-04-30T06:26:19.869ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10242025-04-30T06:26:19.870ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10252025-04-30T06:26:19.870ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10262025-04-30T06:26:19.870ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10272025-04-30T06:26:19.870ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10282025-04-30T06:26:19.871ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 1.061445831s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10292025-04-30T06:26:19.871ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = region_replacement elapsed = 1.062179072s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10302025-04-30T06:26:19.871ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10312025-04-30T06:26:19.871ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10322025-04-30T06:26:19.871ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10332025-04-30T06:26:19.871ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10342025-04-30T06:26:19.871ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10352025-04-30T06:26:19.872ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10362025-04-30T06:26:19.872ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 1043191 local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:55304 req_id = a6bca091-ac5d-4634-8294-e55d96241dd4 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10372025-04-30T06:26:19.872ZDEBGtest_disk_metrics (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:39551/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "a6bca091-ac5d-4634-8294-e55d96241dd4", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10382025-04-30T06:26:19.873ZDEBGtest_disk_metrics (init_with_steps): Running step start_crucible_pantry
10392025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10402025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10412025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10422025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10432025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10442025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10452025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10462025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10472025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10482025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10492025-04-30T06:26:19.875ZDEBGtest_disk_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:35636 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10502025-04-30T06:26:19.875ZINFOtest_disk_metrics (dropshot): listening kind = pantry local_addr = [::1]:35636 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10512025-04-30T06:26:19.876ZDEBGtest_disk_metrics (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:35636 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10522025-04-30T06:26:19.876ZINFOtest_disk_metrics (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:35636 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10532025-04-30T06:26:19.876ZDEBGtest_disk_metrics (init_with_steps): Running step populate_internal_dns
10542025-04-30T06:26:19.876ZDEBGtest_disk_metrics: Populating Internal DNS
10552025-04-30T06:26:19.953ZINFOtest_disk_metrics: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:26:19.951555234Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "8a5e515c-ecc5-49c3-a9dd-d7a585200ff2.host": [ Aaaa( ::1, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 34295, target: "b8202ae0-9a1d-4662-af90-b806c57a906e.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 35636, target: "58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 54200, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 57001, target: "0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal", }, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36104, target: "8a5e515c-ecc5-49c3-a9dd-d7a585200ff2.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 39213, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "8a5e515c-ecc5-49c3-a9dd-d7a585200ff2.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "58007544-db95-41ae-9c86-658edc7605fe.host": [ Aaaa( ::1, ), ], "0bfd02ed-dd55-49ea-af78-a36627e3c575.host": [ Aaaa( ::1, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "8a5e515c-ecc5-49c3-a9dd-d7a585200ff2.host.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "8a5e515c-ecc5-49c3-a9dd-d7a585200ff2.host.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 37651, target: "abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 39551, target: "06ed8ede-6e59-42fb-a0cc-51ac048d6bbe.host.control-plane.oxide.internal", }, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 58149, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "b8202ae0-9a1d-4662-af90-b806c57a906e.host": [ Aaaa( ::1, ), ], "06ed8ede-6e59-42fb-a0cc-51ac048d6bbe.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "abe98b47-4c43-4b99-b630-92d39b0fa0e2.host": [ Aaaa( ::1, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 40605, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], }, }, ], }
10562025-04-30T06:26:19.953ZDEBGtest_disk_metrics: client request body = Some(Body) method = PUT uri = http://[::1]:57001/config
10572025-04-30T06:26:19.957ZINFOtest_disk_metrics (http): accepted connection kind = dns local_addr = [::1]:57001 remote_addr = [::1]:45530
10582025-04-30T06:26:19.959ZTRACtest_disk_metrics (http): incoming request kind = dns local_addr = [::1]:57001 method = PUT remote_addr = [::1]:45530 req_id = bb28d8c2-6671-4210-a6aa-b7cceabb1445 uri = /config
10592025-04-30T06:26:19.963ZINFOtest_disk_metrics (store): attempting generation update new_generation = 1 req_id = bb28d8c2-6671-4210-a6aa-b7cceabb1445
10602025-04-30T06:26:19.963ZINFOtest_disk_metrics (store): pruning trees for generations newer than 0
10612025-04-30T06:26:19.963ZDEBGtest_disk_metrics (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10622025-04-30T06:26:19.965ZDEBGtest_disk_metrics (store): updating current config
10632025-04-30T06:26:19.965ZDEBGtest_disk_metrics (store): flushing default tree
10642025-04-30T06:26:19.965ZINFOtest_disk_metrics (store): pruning trees for generations older than 1 keep = 3
10652025-04-30T06:26:19.965ZINFOtest_disk_metrics (store): updated generation new_generation = 1 req_id = bb28d8c2-6671-4210-a6aa-b7cceabb1445
10662025-04-30T06:26:19.966ZINFOtest_disk_metrics (http): request completed kind = dns latency_us = 6744 local_addr = [::1]:57001 method = PUT remote_addr = [::1]:45530 req_id = bb28d8c2-6671-4210-a6aa-b7cceabb1445 response_code = 204 uri = /config
10672025-04-30T06:26:19.974ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10682025-04-30T06:26:19.974ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10692025-04-30T06:26:19.978ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10702025-04-30T06:26:19.980ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10712025-04-30T06:26:19.988ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10722025-04-30T06:26:19.989ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10732025-04-30T06:26:19.989ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10742025-04-30T06:26:19.989ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10752025-04-30T06:26:20.011ZDEBGtest_disk_metrics: client response result = Ok(Response { url: "http://[::1]:57001/config", status: 204, headers: {"x-request-id": "bb28d8c2-6671-4210-a6aa-b7cceabb1445", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
10762025-04-30T06:26:20.013ZDEBGtest_disk_metrics (init_with_steps): Running step configure_sled_agents
10772025-04-30T06:26:20.071ZDEBGtest_disk_metrics: client request body = Some(Body) method = PUT uri = http://[::1]:50989/omicron-config
10782025-04-30T06:26:20.075ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10792025-04-30T06:26:20.075ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10802025-04-30T06:26:20.075ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10812025-04-30T06:26:20.075ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10822025-04-30T06:26:20.076ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10832025-04-30T06:26:20.078ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 1.260182361s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10842025-04-30T06:26:20.080ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:37947 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10852025-04-30T06:26:20.084ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = PUT remote_addr = [::1]:37947 req_id = 3abc6dd4-b337-43ca-b405-a13d09ae5600 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10862025-04-30T06:26:20.088ZINFOtest_disk_metrics (dropshot): request completed latency_us = 4036 local_addr = [::1]:50989 method = PUT remote_addr = [::1]:37947 req_id = 3abc6dd4-b337-43ca-b405-a13d09ae5600 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10872025-04-30T06:26:20.091ZDEBGtest_disk_metrics: client response result = Ok(Response { url: "http://[::1]:50989/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3abc6dd4-b337-43ca-b405-a13d09ae5600", "content-length": "26", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
10882025-04-30T06:26:20.092ZDEBGtest_disk_metrics: client request body = Some(Body) method = PUT uri = http://[::1]:50989/network-bootstore-config
10892025-04-30T06:26:20.093ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = PUT remote_addr = [::1]:37947 req_id = 9278a3e2-d066-47c6-bebc-ff7daf7aae51 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10902025-04-30T06:26:20.096ZINFOtest_disk_metrics (dropshot): request completed latency_us = 2389 local_addr = [::1]:50989 method = PUT remote_addr = [::1]:37947 req_id = 9278a3e2-d066-47c6-bebc-ff7daf7aae51 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10912025-04-30T06:26:20.097ZDEBGtest_disk_metrics: client response result = Ok(Response { url: "http://[::1]:50989/network-bootstore-config", status: 204, headers: {"x-request-id": "9278a3e2-d066-47c6-bebc-ff7daf7aae51", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
10922025-04-30T06:26:20.098ZDEBGtest_disk_metrics (init_with_steps): Running step start_nexus_external
10932025-04-30T06:26:20.098ZDEBGtest_disk_metrics: Starting Nexus (external API)
10942025-04-30T06:26:23.497ZTRACtest_disk_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
10952025-04-30T06:26:23.497ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10962025-04-30T06:26:23.497ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
10972025-04-30T06:26:23.498ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
10982025-04-30T06:26:23.499ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10992025-04-30T06:26:23.590ZINFOtest_disk_metrics (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11002025-04-30T06:26:23.591ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:50989/vpc-routes
11012025-04-30T06:26:23.591ZTRACtest_disk_metrics (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11022025-04-30T06:26:23.591ZINFOtest_disk_metrics (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11032025-04-30T06:26:23.592ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 4.781087214s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11042025-04-30T06:26:23.595ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11052025-04-30T06:26:23.595ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11062025-04-30T06:26:23.595ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11072025-04-30T06:26:23.595ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11082025-04-30T06:26:23.595ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11092025-04-30T06:26:23.683ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11102025-04-30T06:26:23.683ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11112025-04-30T06:26:23.684ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11122025-04-30T06:26:23.684ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11132025-04-30T06:26:23.685ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11142025-04-30T06:26:23.687ZTRACtest_disk_metrics (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
11152025-04-30T06:26:23.688ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 4.867143486s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
11162025-04-30T06:26:23.692ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:56093 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11172025-04-30T06:26:23.694ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11182025-04-30T06:26:23.698ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
11192025-04-30T06:26:23.699ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11202025-04-30T06:26:23.699ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11212025-04-30T06:26:23.699ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11222025-04-30T06:26:23.701ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:56093 req_id = 5ea43dc4-7c57-4114-af83-6e7efc505605 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11232025-04-30T06:26:23.753ZINFOtest_disk_metrics (dropshot): request completed latency_us = 51976 local_addr = [::1]:50989 method = GET remote_addr = [::1]:56093 req_id = 5ea43dc4-7c57-4114-af83-6e7efc505605 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11242025-04-30T06:26:23.755ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11252025-04-30T06:26:23.766ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11262025-04-30T06:26:23.766ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11272025-04-30T06:26:23.766ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11282025-04-30T06:26:23.766ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11292025-04-30T06:26:23.767ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.948323472s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11302025-04-30T06:26:23.767ZWARNtest_disk_metrics (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11312025-04-30T06:26:23.767ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.965677648s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11322025-04-30T06:26:23.770ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:50989/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ea43dc4-7c57-4114-af83-6e7efc505605", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
11332025-04-30T06:26:23.770ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11342025-04-30T06:26:23.770ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11352025-04-30T06:26:23.771ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11362025-04-30T06:26:23.772ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11372025-04-30T06:26:23.773ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11382025-04-30T06:26:23.773ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11392025-04-30T06:26:23.773ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11402025-04-30T06:26:23.779ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11412025-04-30T06:26:23.785ZDEBGtest_disk_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11422025-04-30T06:26:23.785ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11432025-04-30T06:26:23.785ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
11442025-04-30T06:26:23.785ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11452025-04-30T06:26:23.788ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11462025-04-30T06:26:23.801ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11472025-04-30T06:26:23.801ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11482025-04-30T06:26:23.801ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11492025-04-30T06:26:23.801ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11502025-04-30T06:26:23.801ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11512025-04-30T06:26:23.801ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11522025-04-30T06:26:23.807ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11532025-04-30T06:26:23.807ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11542025-04-30T06:26:23.827ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fa d3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61659 req_id = 48c900ac-156b-42de-aec7-dcecdd265ed7
11552025-04-30T06:26:23.827ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64211,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61659 req_id = 48c900ac-156b-42de-aec7-dcecdd265ed7
11562025-04-30T06:26:23.827ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
11572025-04-30T06:26:23.827ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11582025-04-30T06:26:23.828ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61659 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 48c900ac-156b-42de-aec7-dcecdd265ed7
11592025-04-30T06:26:23.828ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11602025-04-30T06:26:23.835ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11612025-04-30T06:26:23.835ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11622025-04-30T06:26:23.835ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11632025-04-30T06:26:23.836ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11642025-04-30T06:26:23.841ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11652025-04-30T06:26:23.841ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11662025-04-30T06:26:23.841ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11672025-04-30T06:26:23.843ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11682025-04-30T06:26:23.851ZINFOtest_disk_metrics (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11692025-04-30T06:26:23.851ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 5.045495039s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11702025-04-30T06:26:23.866ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3248, tv_nsec: 664947981 } })
11712025-04-30T06:26:23.866ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
11722025-04-30T06:26:23.868ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11732025-04-30T06:26:23.868ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
11742025-04-30T06:26:23.869ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11752025-04-30T06:26:23.873ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11762025-04-30T06:26:23.873ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11772025-04-30T06:26:23.873ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11782025-04-30T06:26:23.873ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11792025-04-30T06:26:23.876ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11802025-04-30T06:26:23.877ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11812025-04-30T06:26:23.885ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11822025-04-30T06:26:23.885ZTRACtest_disk_metrics (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11832025-04-30T06:26:23.885ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11842025-04-30T06:26:23.886ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11852025-04-30T06:26:23.886ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11862025-04-30T06:26:23.886ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11872025-04-30T06:26:23.886ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11882025-04-30T06:26:23.894ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 48 15 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e H...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55629 req_id = b183e6a3-ebcf-4aaa-a049-351f3bc064d0
11892025-04-30T06:26:23.894ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18453,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55629 req_id = b183e6a3-ebcf-4aaa-a049-351f3bc064d0
11902025-04-30T06:26:23.894ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11912025-04-30T06:26:23.895ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55629 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b183e6a3-ebcf-4aaa-a049-351f3bc064d0
11922025-04-30T06:26:23.900ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11932025-04-30T06:26:23.914ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11942025-04-30T06:26:23.914ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11952025-04-30T06:26:23.914ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11962025-04-30T06:26:23.915ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11972025-04-30T06:26:23.917ZINFOtest_disk_metrics (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
11982025-04-30T06:26:23.918ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:50989/eip-gateways
11992025-04-30T06:26:23.918ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12002025-04-30T06:26:23.925ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
12012025-04-30T06:26:23.927ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
12022025-04-30T06:26:23.927ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12032025-04-30T06:26:23.929ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12042025-04-30T06:26:23.931ZINFOtest_disk_metrics (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12052025-04-30T06:26:23.931ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12062025-04-30T06:26:23.931ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12072025-04-30T06:26:23.932ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12082025-04-30T06:26:23.934ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = PUT remote_addr = [::1]:56093 req_id = 3c711641-0d87-4842-ac5a-7e2ac925f416 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12092025-04-30T06:26:23.988ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
12102025-04-30T06:26:23.990ZINFOtest_disk_metrics (dropshot): request completed latency_us = 56032 local_addr = [::1]:50989 method = PUT remote_addr = [::1]:56093 req_id = 3c711641-0d87-4842-ac5a-7e2ac925f416 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12112025-04-30T06:26:23.991ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:50989/artifacts-config
12122025-04-30T06:26:23.992ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12132025-04-30T06:26:23.992ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12142025-04-30T06:26:23.992ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12152025-04-30T06:26:23.995ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12162025-04-30T06:26:24.099ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:50989/v2p
12172025-04-30T06:26:24.102ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:50447 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12182025-04-30T06:26:24.102ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:42196 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12192025-04-30T06:26:24.106ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:50989/eip-gateways", status: 204, headers: {"x-request-id": "3c711641-0d87-4842-ac5a-7e2ac925f416", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
12202025-04-30T06:26:24.109ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 5.297083412s iteration = 1 status = {}
12212025-04-30T06:26:24.110ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12222025-04-30T06:26:24.110ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12232025-04-30T06:26:24.110ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12242025-04-30T06:26:24.111ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12252025-04-30T06:26:24.114ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12262025-04-30T06:26:24.114ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12272025-04-30T06:26:24.114ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12282025-04-30T06:26:24.114ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12292025-04-30T06:26:24.115ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12302025-04-30T06:26:24.116ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = PUT remote_addr = [::1]:50447 req_id = 4d2acd8e-0bd7-4f49-b4da-9a948f4ae8e8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12312025-04-30T06:26:24.116ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:42196 req_id = 33f80b19-68a4-4a7e-adf5-4ebe1f74abea sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12322025-04-30T06:26:24.118ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f9 aa 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61679 req_id = e7194380-d72f-4651-8e94-971323f77c7d
12332025-04-30T06:26:24.118ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63914,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61679 req_id = e7194380-d72f-4651-8e94-971323f77c7d
12342025-04-30T06:26:24.118ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
12352025-04-30T06:26:24.118ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61679 req_id = e7194380-d72f-4651-8e94-971323f77c7d
12362025-04-30T06:26:24.119ZINFOtest_disk_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpSz5UZI/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12372025-04-30T06:26:24.119ZDEBGtest_disk_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpSz5UZI/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12382025-04-30T06:26:24.120ZINFOtest_disk_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpTLbgOh/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12392025-04-30T06:26:24.120ZDEBGtest_disk_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpTLbgOh/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12402025-04-30T06:26:24.120ZINFOtest_disk_metrics (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpSz5UZI/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12412025-04-30T06:26:24.120ZINFOtest_disk_metrics (dropshot): request completed latency_us = 3635 local_addr = [::1]:50989 method = GET remote_addr = [::1]:42196 req_id = 33f80b19-68a4-4a7e-adf5-4ebe1f74abea response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12422025-04-30T06:26:24.126ZINFOtest_disk_metrics (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpTLbgOh/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12432025-04-30T06:26:24.129ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12442025-04-30T06:26:24.145ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12452025-04-30T06:26:24.145ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12462025-04-30T06:26:24.145ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12472025-04-30T06:26:24.149ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12482025-04-30T06:26:24.150ZINFOtest_disk_metrics (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12492025-04-30T06:26:24.151ZINFOtest_disk_metrics (dropshot): request completed latency_us = 34885 local_addr = [::1]:50989 method = PUT remote_addr = [::1]:50447 req_id = 4d2acd8e-0bd7-4f49-b4da-9a948f4ae8e8 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12502025-04-30T06:26:24.157ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
12512025-04-30T06:26:24.157ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
12522025-04-30T06:26:24.157ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
12532025-04-30T06:26:24.158ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:50989/artifacts-config", status: 204, headers: {"x-request-id": "4d2acd8e-0bd7-4f49-b4da-9a948f4ae8e8", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
12542025-04-30T06:26:24.159ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:50989/artifacts
12552025-04-30T06:26:24.159ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:50989/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "33f80b19-68a4-4a7e-adf5-4ebe1f74abea", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
12562025-04-30T06:26:24.159ZINFOtest_disk_metrics (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12572025-04-30T06:26:24.159ZINFOtest_disk_metrics (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12582025-04-30T06:26:24.159ZINFOtest_disk_metrics (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12592025-04-30T06:26:24.161ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 5.353629427s iteration = 1 status = {}
12602025-04-30T06:26:24.182ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12612025-04-30T06:26:24.182ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12622025-04-30T06:26:24.183ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12632025-04-30T06:26:24.184ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12642025-04-30T06:26:24.191ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12652025-04-30T06:26:24.191ZDEBGtest_disk_metrics (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12662025-04-30T06:26:24.191ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12672025-04-30T06:26:24.191ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12682025-04-30T06:26:24.192ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12692025-04-30T06:26:24.193ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:50447 req_id = b8f7d7f8-a4ad-4ee9-952e-d106f4e16419 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12702025-04-30T06:26:24.197ZINFOtest_disk_metrics (dropshot): request completed latency_us = 4084 local_addr = [::1]:50989 method = GET remote_addr = [::1]:50447 req_id = b8f7d7f8-a4ad-4ee9-952e-d106f4e16419 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12712025-04-30T06:26:24.200ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12722025-04-30T06:26:24.200ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12732025-04-30T06:26:24.201ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12742025-04-30T06:26:24.201ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:50989/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f7d7f8-a4ad-4ee9-952e-d106f4e16419", "content-length": "26", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
12752025-04-30T06:26:24.204ZINFOtest_disk_metrics (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:50989
12762025-04-30T06:26:24.207ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 5.387647145s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:26:24.158927879Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:26:24.204106855Z"}]}
12772025-04-30T06:26:24.211ZINFOtest_disk_metrics (InternalApi): Using static configuration for external switchports
12782025-04-30T06:26:24.211ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12792025-04-30T06:26:24.211ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12802025-04-30T06:26:24.212ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12812025-04-30T06:26:24.214ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12822025-04-30T06:26:24.215ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12832025-04-30T06:26:24.215ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12842025-04-30T06:26:24.220ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12852025-04-30T06:26:24.221ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12862025-04-30T06:26:24.221ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12872025-04-30T06:26:24.221ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12882025-04-30T06:26:24.222ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12892025-04-30T06:26:24.243ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4416198e-880a-4ebf-8d5e-c9d830cd4e16", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
12902025-04-30T06:26:24.244ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
12912025-04-30T06:26:24.246ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12922025-04-30T06:26:24.246ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12932025-04-30T06:26:24.246ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12942025-04-30T06:26:24.247ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12952025-04-30T06:26:24.258ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:26:24.258ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:26:24.258ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:26:24.259ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:26:24.267ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbae7f99-d67b-41f2-a634-b29614a53c17", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
13002025-04-30T06:26:24.268ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
13012025-04-30T06:26:24.268ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13022025-04-30T06:26:24.275ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13032025-04-30T06:26:24.275ZTRACtest_disk_metrics (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13042025-04-30T06:26:24.275ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13052025-04-30T06:26:24.275ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13062025-04-30T06:26:24.276ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13072025-04-30T06:26:24.276ZINFOtest_disk_metrics (InternalApi): Recording Rack Network Configuration
13082025-04-30T06:26:24.276ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13092025-04-30T06:26:24.276ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13102025-04-30T06:26:24.277ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13112025-04-30T06:26:24.280ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13122025-04-30T06:26:24.280ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13132025-04-30T06:26:24.281ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13142025-04-30T06:26:24.285ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13152025-04-30T06:26:24.285ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13162025-04-30T06:26:24.285ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13172025-04-30T06:26:24.290ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13182025-04-30T06:26:24.291ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13192025-04-30T06:26:24.292ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13202025-04-30T06:26:24.333ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:26:24.333ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:26:24.333ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:26:24.333ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:26:24.333ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:26:24.369ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:26:24.369ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:26:24.369ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:26:24.370ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:26:24.375ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302025-04-30T06:26:24.375ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13312025-04-30T06:26:24.375ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13322025-04-30T06:26:24.377ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13332025-04-30T06:26:24.381ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13342025-04-30T06:26:24.391ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13352025-04-30T06:26:24.392ZTRACtest_disk_metrics (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13362025-04-30T06:26:24.392ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13372025-04-30T06:26:24.392ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13382025-04-30T06:26:24.392ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13392025-04-30T06:26:24.392ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13402025-04-30T06:26:24.393ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13412025-04-30T06:26:24.406ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5907804-e0d3-4566-95e8-309587cdc894", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
13422025-04-30T06:26:24.406ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
13432025-04-30T06:26:24.407ZTRACtest_disk_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13442025-04-30T06:26:24.407ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13452025-04-30T06:26:24.407ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13462025-04-30T06:26:24.407ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13472025-04-30T06:26:24.426ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13482025-04-30T06:26:24.431ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13492025-04-30T06:26:24.431ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13502025-04-30T06:26:24.431ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13512025-04-30T06:26:24.431ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13522025-04-30T06:26:24.440ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13532025-04-30T06:26:24.443ZDEBGtest_disk_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13542025-04-30T06:26:24.444ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13552025-04-30T06:26:24.444ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13562025-04-30T06:26:24.444ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13572025-04-30T06:26:24.450ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13582025-04-30T06:26:24.450ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13592025-04-30T06:26:24.450ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13602025-04-30T06:26:24.450ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13612025-04-30T06:26:24.460ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13622025-04-30T06:26:24.469ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13632025-04-30T06:26:24.469ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13642025-04-30T06:26:24.469ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13652025-04-30T06:26:24.469ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13662025-04-30T06:26:24.471ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 9f455afc-a941-4e5d-9bdf-0fff0c5b8b11, lookup_type: ByName("oxide-service-pool") }
13672025-04-30T06:26:24.471ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13682025-04-30T06:26:24.471ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13692025-04-30T06:26:24.471ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13702025-04-30T06:26:24.471ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13712025-04-30T06:26:24.483ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13722025-04-30T06:26:24.483ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13732025-04-30T06:26:24.484ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13742025-04-30T06:26:24.487ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13752025-04-30T06:26:24.498ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 9f455afc-a941-4e5d-9bdf-0fff0c5b8b11, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13762025-04-30T06:26:24.498ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13772025-04-30T06:26:24.498ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13782025-04-30T06:26:24.498ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13792025-04-30T06:26:24.502ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13802025-04-30T06:26:24.502ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13812025-04-30T06:26:24.502ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13822025-04-30T06:26:24.536ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13832025-04-30T06:26:24.536ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13842025-04-30T06:26:24.537ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13852025-04-30T06:26:24.548ZTRACtest_disk_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 9f455afc-a941-4e5d-9bdf-0fff0c5b8b11, lookup_type: ByName("oxide-service-pool") }
13862025-04-30T06:26:24.548ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13872025-04-30T06:26:24.548ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13882025-04-30T06:26:24.548ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13892025-04-30T06:26:24.548ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13902025-04-30T06:26:24.554ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13912025-04-30T06:26:24.554ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13922025-04-30T06:26:24.555ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13932025-04-30T06:26:24.557ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13942025-04-30T06:26:24.559ZDEBGtest_disk_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 9f455afc-a941-4e5d-9bdf-0fff0c5b8b11, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13952025-04-30T06:26:24.563ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7e6d3ef-b640-4f2f-9b2a-2c0d3d3b7291", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
13962025-04-30T06:26:24.563ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
13972025-04-30T06:26:24.564ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13982025-04-30T06:26:24.564ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13992025-04-30T06:26:24.564ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14002025-04-30T06:26:24.564ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14012025-04-30T06:26:24.564ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14022025-04-30T06:26:24.568ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14032025-04-30T06:26:24.568ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14042025-04-30T06:26:24.568ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14052025-04-30T06:26:24.568ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14062025-04-30T06:26:24.570ZTRACtest_disk_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 9f455afc-a941-4e5d-9bdf-0fff0c5b8b11, lookup_type: ByName("oxide-service-pool") }
14072025-04-30T06:26:24.570ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14082025-04-30T06:26:24.570ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14092025-04-30T06:26:24.570ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
14102025-04-30T06:26:24.570ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14112025-04-30T06:26:24.571ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14122025-04-30T06:26:24.571ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14132025-04-30T06:26:24.571ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14142025-04-30T06:26:24.572ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14152025-04-30T06:26:24.574ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14162025-04-30T06:26:24.579ZDEBGtest_disk_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 9f455afc-a941-4e5d-9bdf-0fff0c5b8b11, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14172025-04-30T06:26:24.580ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14182025-04-30T06:26:24.589ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14192025-04-30T06:26:24.589ZTRACtest_disk_metrics (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14202025-04-30T06:26:24.589ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14212025-04-30T06:26:24.589ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14222025-04-30T06:26:24.589ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14232025-04-30T06:26:24.589ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14242025-04-30T06:26:24.590ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14252025-04-30T06:26:24.591ZTRACtest_disk_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14262025-04-30T06:26:24.591ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14272025-04-30T06:26:24.591ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14282025-04-30T06:26:24.591ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
14292025-04-30T06:26:24.592ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14302025-04-30T06:26:24.596ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14312025-04-30T06:26:24.596ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14322025-04-30T06:26:24.596ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14332025-04-30T06:26:24.597ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14342025-04-30T06:26:24.599ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14352025-04-30T06:26:24.603ZDEBGtest_disk_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14362025-04-30T06:26:24.604ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14372025-04-30T06:26:24.604ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14382025-04-30T06:26:24.604ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14392025-04-30T06:26:24.604ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14402025-04-30T06:26:24.613ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7232b49a-5a02-43c0-9172-351f9432ff51", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
14412025-04-30T06:26:24.613ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
14422025-04-30T06:26:24.613ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14432025-04-30T06:26:24.648ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14442025-04-30T06:26:24.648ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14452025-04-30T06:26:24.648ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14462025-04-30T06:26:24.649ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14472025-04-30T06:26:24.665ZDEBGtest_disk_metrics (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
14482025-04-30T06:26:24.665ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14492025-04-30T06:26:24.665ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14502025-04-30T06:26:24.665ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14512025-04-30T06:26:24.665ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14522025-04-30T06:26:24.666ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14532025-04-30T06:26:24.672ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14542025-04-30T06:26:24.678ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14552025-04-30T06:26:24.678ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14562025-04-30T06:26:24.678ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14572025-04-30T06:26:24.679ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14582025-04-30T06:26:24.681ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3f61bf-0241-4b18-a15d-a14a7c2b26d3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
14592025-04-30T06:26:24.682ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
14602025-04-30T06:26:24.684ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14809267-9f8e-4bb8-92da-c38e6317c2e1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
14612025-04-30T06:26:24.695ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
14622025-04-30T06:26:24.695ZTRACtest_disk_metrics (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14632025-04-30T06:26:24.695ZDEBGtest_disk_metrics (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14642025-04-30T06:26:24.695ZDEBGtest_disk_metrics (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14652025-04-30T06:26:24.695ZDEBGtest_disk_metrics (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14662025-04-30T06:26:24.708ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab88351f-6565-4312-9acc-1ef227ceb0c2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14672025-04-30T06:26:24.712ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
14682025-04-30T06:26:24.713ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14f4ab27-80b1-4bf8-b35b-02b1ea001281", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14692025-04-30T06:26:24.713ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
14702025-04-30T06:26:24.722ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "234d3b35-b133-4044-9a64-f27d9364568d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14712025-04-30T06:26:24.722ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
14722025-04-30T06:26:24.724ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e08e653d-3e9f-4f95-9cbb-e91c86207b08", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14732025-04-30T06:26:24.724ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
14742025-04-30T06:26:24.725ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3672b71a-f12f-4ce2-bb81-2d902b36ee13", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14752025-04-30T06:26:24.725ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
14762025-04-30T06:26:24.727ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34927948-ec4f-4f07-bbab-314cd3b26e81", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14772025-04-30T06:26:24.727ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:26:24.729ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa50fb03-f174-4df5-9825-7fb859d89aee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14792025-04-30T06:26:24.729ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:26:24.732ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f696814-9cc2-4025-89f0-5c4199e617e6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14812025-04-30T06:26:24.732ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:26:24.735ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37af498d-0734-4374-bf03-221036350b3b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14832025-04-30T06:26:24.735ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:26:24.738ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46fffbac-8079-4a8d-b64b-6ef6019357c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14852025-04-30T06:26:24.743ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:26:24.755ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "926cd370-aa3b-4147-bfdd-372798274dd3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14872025-04-30T06:26:24.756ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:26:24.761ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1c74044-e52d-458e-a51f-e8686a804c3b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14892025-04-30T06:26:24.762ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
14902025-04-30T06:26:24.764ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7031e209-8281-471a-bfdf-2d96632a3065", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14912025-04-30T06:26:24.764ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
14922025-04-30T06:26:24.765ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e26f660-7635-4f04-83e0-e74d0ae9d54b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14932025-04-30T06:26:24.765ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
14942025-04-30T06:26:24.770ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ea5d4b1-3d68-4670-a28f-246bcbab3e6f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14952025-04-30T06:26:24.770ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
14962025-04-30T06:26:24.771ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d41836f-8764-49c8-89a7-efd378003e6d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14972025-04-30T06:26:24.771ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
14982025-04-30T06:26:24.781ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd247f9c-d84b-42ac-ae8a-254970b437ab", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
14992025-04-30T06:26:24.782ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
15002025-04-30T06:26:24.784ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1476498-878c-464e-887e-cdf69a0b89bd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15012025-04-30T06:26:24.784ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
15022025-04-30T06:26:24.786ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8469187-032a-4e6b-a4d7-1f4f61fbd679", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15032025-04-30T06:26:24.788ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
15042025-04-30T06:26:24.790ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75f88e6b-794b-440a-9903-f8ffed71c9cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15052025-04-30T06:26:24.791ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
15062025-04-30T06:26:24.794ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05d891e5-2772-42f2-9439-21cc6bfa7d05", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15072025-04-30T06:26:24.794ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
15082025-04-30T06:26:24.796ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93c30143-6e96-4cb7-9c15-2191c17e35eb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15092025-04-30T06:26:24.798ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
15102025-04-30T06:26:24.799ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d72ea82c-7d83-460f-9973-6bfa7f32d388", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15112025-04-30T06:26:24.799ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
15122025-04-30T06:26:24.801ZINFOtest_disk_metrics (InternalApi): inserted blueprint blueprint_id = 8103503d-1d38-4dca-aa32-944ee6611aaf
15132025-04-30T06:26:24.801ZTRACtest_disk_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15142025-04-30T06:26:24.801ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15152025-04-30T06:26:24.801ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15162025-04-30T06:26:24.801ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
15172025-04-30T06:26:24.803ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15182025-04-30T06:26:24.803ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37110c4e-c507-44a9-a450-dacb7be3ff91", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15192025-04-30T06:26:24.803ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
15202025-04-30T06:26:24.806ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61be2af8-4dbe-455d-9126-f0a947ac8e46", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15212025-04-30T06:26:24.806ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
15222025-04-30T06:26:24.806ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15232025-04-30T06:26:24.810ZDEBGtest_disk_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15242025-04-30T06:26:24.811ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c74041b6-bf5f-430e-ac19-f689f6c14c8b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15252025-04-30T06:26:24.811ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
15262025-04-30T06:26:24.812ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b39ac6ce-c877-4662-9613-49e970435b91", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15272025-04-30T06:26:24.812ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
15282025-04-30T06:26:24.814ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1119fc58-6a55-4d87-881c-023ee30dffd9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15292025-04-30T06:26:24.814ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
15302025-04-30T06:26:24.816ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85f99f1f-e037-465a-96cc-8ea1885fa406", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15312025-04-30T06:26:24.817ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
15322025-04-30T06:26:24.818ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ccfaf38-c77f-40ef-b881-62618c022ac2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15332025-04-30T06:26:24.818ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
15342025-04-30T06:26:24.820ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26d5ae9a-1195-46cd-80ba-8fa2878ae5c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15352025-04-30T06:26:24.820ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
15362025-04-30T06:26:24.822ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3cdd080-803c-4f9b-939c-e2f08b5364ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15372025-04-30T06:26:24.822ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
15382025-04-30T06:26:24.824ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17b3ed1a-ab06-4f57-a9b7-d63062f3ad68", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15392025-04-30T06:26:24.825ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
15402025-04-30T06:26:24.827ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ce35ac4-b0a6-433b-b00f-50efda0860bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15412025-04-30T06:26:24.827ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
15422025-04-30T06:26:24.828ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "896ed670-5225-414f-93f3-e6f77b93f154", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15432025-04-30T06:26:24.828ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
15442025-04-30T06:26:24.829ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "772e529f-2d68-427f-8f03-2005c499f7fd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15452025-04-30T06:26:24.829ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
15462025-04-30T06:26:24.831ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fb411e2-de48-4a37-9e14-c6e3afa9aced", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15472025-04-30T06:26:24.831ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
15482025-04-30T06:26:24.832ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46e96ab4-621b-4a0c-a194-46ccaad8c4ea", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15492025-04-30T06:26:24.832ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15502025-04-30T06:26:24.832ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15512025-04-30T06:26:24.832ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15522025-04-30T06:26:24.832ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15532025-04-30T06:26:24.833ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15542025-04-30T06:26:24.890ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15552025-04-30T06:26:24.902ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15562025-04-30T06:26:24.903ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15572025-04-30T06:26:24.903ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15582025-04-30T06:26:24.903ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15592025-04-30T06:26:24.903ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15602025-04-30T06:26:24.903ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15612025-04-30T06:26:24.910ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15622025-04-30T06:26:24.917ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15632025-04-30T06:26:24.917ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15642025-04-30T06:26:24.917ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15652025-04-30T06:26:24.918ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15662025-04-30T06:26:24.942ZDEBGtest_disk_metrics (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
15672025-04-30T06:26:24.942ZDEBGtest_disk_metrics (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15682025-04-30T06:26:24.943ZINFOtest_disk_metrics (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15692025-04-30T06:26:24.943ZDEBGtest_disk_metrics (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 6.131249494s
15702025-04-30T06:26:24.943ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 6.131298892s iteration = 1 status = {}
15712025-04-30T06:26:24.943ZINFOtest_disk_metrics (InternalApi): Inserted networking records for nexus service
15722025-04-30T06:26:24.943ZINFOtest_disk_metrics (InternalApi): No networking records needed for internal_dns service
15732025-04-30T06:26:24.943ZINFOtest_disk_metrics (InternalApi): No networking records needed for crucible_pantry service
15742025-04-30T06:26:24.943ZINFOtest_disk_metrics (InternalApi): No networking records needed for clickhouse service
15752025-04-30T06:26:25.059ZINFOtest_disk_metrics (InternalApi): Inserted networking records for external_dns service
15762025-04-30T06:26:25.059ZINFOtest_disk_metrics (InternalApi): No networking records needed for cockroach_db service
15772025-04-30T06:26:25.059ZINFOtest_disk_metrics (InternalApi): Inserted service networking records
15782025-04-30T06:26:25.059ZINFOtest_disk_metrics (InternalApi): Inserted physical disks
15792025-04-30T06:26:25.059ZINFOtest_disk_metrics (InternalApi): Inserted zpools
15802025-04-30T06:26:25.059ZINFOtest_disk_metrics (InternalApi): Inserted datasets
15812025-04-30T06:26:25.148ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
15822025-04-30T06:26:25.148ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
15832025-04-30T06:26:25.149ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:34546 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15842025-04-30T06:26:25.152ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:34546 req_id = 43e5e72a-4aea-42d7-a00f-f78d47051bb5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15852025-04-30T06:26:25.152ZINFOtest_disk_metrics (dropshot): request completed latency_us = 569 local_addr = [::1]:50989 method = GET remote_addr = [::1]:34546 req_id = 43e5e72a-4aea-42d7-a00f-f78d47051bb5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15862025-04-30T06:26:25.153ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e5e72a-4aea-42d7-a00f-f78d47051bb5", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
15872025-04-30T06:26:25.155ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15882025-04-30T06:26:25.155ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15892025-04-30T06:26:25.155ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
15902025-04-30T06:26:25.156ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15912025-04-30T06:26:25.157ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15922025-04-30T06:26:25.157ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15932025-04-30T06:26:25.157ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15942025-04-30T06:26:25.157ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15952025-04-30T06:26:25.158ZINFOtest_disk_metrics (InternalApi): Populated DNS tables for internal DNS
15962025-04-30T06:26:25.161ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15972025-04-30T06:26:25.165ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
15982025-04-30T06:26:25.165ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15992025-04-30T06:26:25.165ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16002025-04-30T06:26:25.166ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16012025-04-30T06:26:25.182ZINFOtest_disk_metrics (InternalApi): Populated DNS tables for external DNS
16022025-04-30T06:26:25.183ZTRACtest_disk_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16032025-04-30T06:26:25.183ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16042025-04-30T06:26:25.183ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16052025-04-30T06:26:25.183ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16062025-04-30T06:26:25.184ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16072025-04-30T06:26:25.219ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-04-30T06:26:25.228ZDEBGtest_disk_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16092025-04-30T06:26:25.228ZTRACtest_disk_metrics (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16102025-04-30T06:26:25.228ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16112025-04-30T06:26:25.228ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16122025-04-30T06:26:25.228ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16132025-04-30T06:26:25.230ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16142025-04-30T06:26:25.243ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-04-30T06:26:25.255ZDEBGtest_disk_metrics (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16162025-04-30T06:26:25.325ZTRACtest_disk_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16172025-04-30T06:26:25.325ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16182025-04-30T06:26:25.326ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16192025-04-30T06:26:25.326ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16202025-04-30T06:26:25.326ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16212025-04-30T06:26:25.341ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-04-30T06:26:25.356ZDEBGtest_disk_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16232025-04-30T06:26:25.356ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16242025-04-30T06:26:25.356ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16252025-04-30T06:26:25.357ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16262025-04-30T06:26:25.357ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16272025-04-30T06:26:25.357ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16282025-04-30T06:26:25.359ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:26:25.379ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:26:25.396ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:26:25.396ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:26:25.399ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:26:25.399ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:26:25.402ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:26:25.411ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:26:25.417ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:26:25.425ZINFOtest_disk_metrics (InternalApi): Created recovery silo
16382025-04-30T06:26:25.426ZINFOtest_disk_metrics (InternalApi): Created recovery user
16392025-04-30T06:26:25.433ZINFOtest_disk_metrics (InternalApi): Created recovery user's password
16402025-04-30T06:26:25.433ZTRACtest_disk_metrics (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: b9d8e134-c9b0-412f-84f7-8316d5ab0644, lookup_type: ById(b9d8e134-c9b0-412f-84f7-8316d5ab0644) }
16412025-04-30T06:26:25.433ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = b9d8e134-c9b0-412f-84f7-8316d5ab0644 resource_type = Silo
16422025-04-30T06:26:25.433ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:26:25.433ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:26:25.434ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:26:25.436ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:26:25.436ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:26:25.436ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:26:25.437ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:26:25.440ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:26:25.442ZDEBGtest_disk_metrics (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: b9d8e134-c9b0-412f-84f7-8316d5ab0644, lookup_type: ById(b9d8e134-c9b0-412f-84f7-8316d5ab0644) } result = Ok(())
16512025-04-30T06:26:25.443ZDEBGtest_disk_metrics (InternalApi): Generated role assignment queries
16522025-04-30T06:26:25.446ZINFOtest_disk_metrics (InternalApi): Granted Silo privileges
16532025-04-30T06:26:25.446ZTRACtest_disk_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:26:25.446ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:26:25.446ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:26:25.446ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:26:25.450ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:26:25.457ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:26:25.460ZDEBGtest_disk_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:26:25.478ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:26:25.478ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:26:25.483ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:26:25.487ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:26:25.487ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:26:25.488ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:26:25.492ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:26:25.492ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:26:25.493ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:26:25.502ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:26:25.502ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:26:25.502ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:26:25.502ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:26:25.504ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:26:25.506ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:26:25.508ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:26:25.508ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:26:25.509ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:26:25.512ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:26:25.512ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:26:25.512ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:26:25.513ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:26:25.518ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:26:25.526ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:26:25.526ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:26:25.526ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:26:25.526ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:26:25.526ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:26:25.526ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:26:25.526ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:26:25.529ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:26:25.529ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:26:25.529ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:26:25.530ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:26:25.531ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:26:25.532ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:26:25.532ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:26:25.532ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:26:25.536ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:26:25.545ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:26:25.545ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:26:25.545ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:26:25.545ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:26:25.554ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:26:25.554ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:26:25.556ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:26:25.558ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:26:25.558ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:26:25.558ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:26:25.560ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:26:25.560ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:26:25.564ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:26:25.566ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:26:25.566ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:26:25.566ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:26:25.566ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:26:25.566ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:26:25.569ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:26:25.569ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:26:25.569ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:26:25.569ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:26:25.572ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:26:25.572ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:26:25.572ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:26:25.574ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:26:25.577ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:26:25.592ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:26:25.593ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:26:25.593ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:26:25.593ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:26:25.600ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:26:25.606ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:26:25.606ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:26:25.607ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:26:25.613ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:26:25.613ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:26:25.614ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:26:25.619ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:26:25.619ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:26:25.620ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:26:25.622ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:26:25.624ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:26:25.624ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:26:25.624ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:26:25.624ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:26:25.627ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17462025-04-30T06:26:25.627ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17472025-04-30T06:26:25.628ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17482025-04-30T06:26:25.629ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17492025-04-30T06:26:25.636ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17502025-04-30T06:26:25.636ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:26:25.636ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:26:25.637ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:26:25.754ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17542025-04-30T06:26:25.764ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17552025-04-30T06:26:25.764ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17562025-04-30T06:26:25.765ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17572025-04-30T06:26:25.765ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17582025-04-30T06:26:25.765ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:26:25.765ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:26:25.765ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:26:25.770ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17622025-04-30T06:26:25.770ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17632025-04-30T06:26:25.770ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17642025-04-30T06:26:25.771ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17652025-04-30T06:26:25.773ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17662025-04-30T06:26:25.773ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:26:25.773ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:26:25.773ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:26:25.778ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17702025-04-30T06:26:25.787ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17712025-04-30T06:26:25.787ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17722025-04-30T06:26:25.787ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17732025-04-30T06:26:25.788ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17742025-04-30T06:26:25.794ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:26:25.794ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:26:25.795ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:26:25.799ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17782025-04-30T06:26:25.799ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17792025-04-30T06:26:25.801ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17802025-04-30T06:26:25.803ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17812025-04-30T06:26:25.804ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17822025-04-30T06:26:25.805ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17832025-04-30T06:26:25.808ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17842025-04-30T06:26:25.808ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17852025-04-30T06:26:25.809ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17862025-04-30T06:26:25.809ZINFOtest_disk_metrics (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 80cb6c5e-8eff-4092-8c0b-d0e2fead80c7
17872025-04-30T06:26:25.809ZDEBGtest_disk_metrics (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 80cb6c5e-8eff-4092-8c0b-d0e2fead80c7 time_started = 2025-04-30 06:26:24.157331 UTC
17882025-04-30T06:26:25.809ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.937964666s iteration = 2 status = {"collection_id":"80cb6c5e-8eff-4092-8c0b-d0e2fead80c7","time_done":"2025-04-30 06:26:25.155293 UTC","time_started":"2025-04-30 06:26:24.157331 UTC"}
17892025-04-30T06:26:25.809ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
17902025-04-30T06:26:25.809ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 45.28\u{b5}s iteration = 2 status = {"error":"task disabled"}
17912025-04-30T06:26:25.809ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
17922025-04-30T06:26:25.809ZWARNtest_disk_metrics (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
17932025-04-30T06:26:25.809ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 16.663\u{b5}s iteration = 2 status = {"error":"no blueprint"}
17942025-04-30T06:26:25.813ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17952025-04-30T06:26:25.813ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17962025-04-30T06:26:25.813ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:26:25.813ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:26:25.814ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:26:25.816ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18002025-04-30T06:26:25.816ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:26:25.816ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:26:25.817ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:26:25.821ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18042025-04-30T06:26:25.821ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:26:25.821ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:26:25.822ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:26:25.825ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-04-30T06:26:25.835ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18092025-04-30T06:26:25.835ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18102025-04-30T06:26:25.835ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18112025-04-30T06:26:25.835ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18122025-04-30T06:26:25.835ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:26:25.835ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:26:25.836ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:26:25.842ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18162025-04-30T06:26:25.842ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18172025-04-30T06:26:25.842ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18182025-04-30T06:26:25.843ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18192025-04-30T06:26:25.849ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18202025-04-30T06:26:25.849ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18212025-04-30T06:26:25.849ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18222025-04-30T06:26:25.849ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18232025-04-30T06:26:25.852ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-04-30T06:26:25.860ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18252025-04-30T06:26:25.860ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18262025-04-30T06:26:25.860ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18272025-04-30T06:26:25.861ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18282025-04-30T06:26:25.883ZDEBGtest_disk_metrics (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: c14e81ea-5508-4e28-922b-87676d66580c, kind: Service { id: 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe }, name: Name("nexus-06ed8ede-6e59-42fb-a0cc-51ac048d6bbe"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 83d74e6a-b808-43fc-b550-3cbbfe2a2065, kind: Service { id: abe98b47-4c43-4b99-b630-92d39b0fa0e2 }, name: Name("external-dns-abe98b47-4c43-4b99-b630-92d39b0fa0e2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:26:25.883ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:26:25.883ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:26:25.883ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:26:25.883ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:26:25.885ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:26:25.888ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:26:25.892ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:26:25.893ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:26:25.893ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:26:25.893ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:26:25.896ZTRACtest_disk_metrics (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:26:25.460035Z, allowed_ips: None }
18402025-04-30T06:26:25.896ZDEBGtest_disk_metrics (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:26:25.897ZDEBGtest_disk_metrics (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 83d74e6a-b808-43fc-b550-3cbbfe2a2065, kind: Service { id: abe98b47-4c43-4b99-b630-92d39b0fa0e2 }, name: Name("external-dns-abe98b47-4c43-4b99-b630-92d39b0fa0e2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: c14e81ea-5508-4e28-922b-87676d66580c, kind: Service { id: 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe }, name: Name("nexus-06ed8ede-6e59-42fb-a0cc-51ac048d6bbe"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:26:25.897ZDEBGtest_disk_metrics (nexus): resolved 2 rules for sleds
18432025-04-30T06:26:25.935ZDEBGtest_disk_metrics (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:26:18.831805Z, time_modified: 2025-04-30T06:26:18.831805Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(50989), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(58149) }]
18442025-04-30T06:26:25.935ZDEBGtest_disk_metrics (nexus): sending firewall rules to sled agents
18452025-04-30T06:26:25.936ZTRACtest_disk_metrics (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:26:25.936ZDEBGtest_disk_metrics (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:26:25.936ZDEBGtest_disk_metrics (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:26:25.946ZTRACtest_disk_metrics (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:26:25.946ZTRACtest_disk_metrics (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:26:25.946ZTRACtest_disk_metrics (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:26:25.946ZDEBGtest_disk_metrics (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:26:25.947ZDEBGtest_disk_metrics (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:26:25.952ZDEBGtest_disk_metrics (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:26:25.963ZDEBGtest_disk_metrics (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:26:26.008ZDEBGtest_disk_metrics (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:50989/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:26:26.010ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:64881 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:26:26.019ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = PUT remote_addr = [::1]:64881 req_id = 5ee664d3-bfa2-40ec-82ef-9c6a6a355cd7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:26:26.024ZINFOtest_disk_metrics (dropshot): request completed latency_us = 4718 local_addr = [::1]:50989 method = PUT remote_addr = [::1]:64881 req_id = 5ee664d3-bfa2-40ec-82ef-9c6a6a355cd7 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:26:26.025ZDEBGtest_disk_metrics (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:50989/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "5ee664d3-bfa2-40ec-82ef-9c6a6a355cd7", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
18602025-04-30T06:26:26.027ZINFOtest_disk_metrics (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:26:26.027ZTRACtest_disk_metrics (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:26:26.027ZDEBGtest_disk_metrics (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:26:26.027ZDEBGtest_disk_metrics (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:26:26.028ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:26:26.028ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:26:26.028ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:26:26.028ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:26:26.029ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:26:26.029ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:26:26.029ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:26:26.029ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:26:26.029ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:26:26.029ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:26:26.029ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:26:26.030ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:26:26.030ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:26:26.030ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:26:26.030ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:26:26.030ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:26:26.030ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:26:26.030ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:26:26.030ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:26:26.030ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:26:26.031ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:26:26.031ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:26:26.031ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:26:26.031ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:26:26.031ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:26:26.031ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:26:26.037ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
18912025-04-30T06:26:26.037ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18922025-04-30T06:26:26.037ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18932025-04-30T06:26:26.038ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18942025-04-30T06:26:26.038ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18952025-04-30T06:26:26.040ZTRACtest_disk_metrics (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: d9 dd 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:56169 req_id = c06024d1-00a8-4441-9c74-c74e070b6cac
18962025-04-30T06:26:26.040ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55773,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56169 req_id = c06024d1-00a8-4441-9c74-c74e070b6cac
18972025-04-30T06:26:26.040ZDEBGtest_disk_metrics (store): query key key = _nameservice._tcp
18982025-04-30T06:26:26.040ZDEBGtest_disk_metrics (store): query key key = 0bfd02ed-dd55-49ea-af78-a36627e3c575.host
18992025-04-30T06:26:26.040ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56169 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 57001, target: Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal") })) }] req_id = c06024d1-00a8-4441-9c74-c74e070b6cac
19002025-04-30T06:26:26.041ZTRACtest_disk_metrics (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: a5 95 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54293 req_id = 4c66438f-d2e4-4344-b3da-bc6a47637f77
19012025-04-30T06:26:26.041ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42389,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54293 req_id = 4c66438f-d2e4-4344-b3da-bc6a47637f77
19022025-04-30T06:26:26.041ZDEBGtest_disk_metrics (store): query key key = _external-dns._tcp
19032025-04-30T06:26:26.041ZDEBGtest_disk_metrics (store): query key key = abe98b47-4c43-4b99-b630-92d39b0fa0e2.host
19042025-04-30T06:26:26.041ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54293 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 37651, target: Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal") })) }] req_id = 4c66438f-d2e4-4344-b3da-bc6a47637f77
19052025-04-30T06:26:26.041ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19062025-04-30T06:26:26.041ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19072025-04-30T06:26:26.041ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19082025-04-30T06:26:26.041ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19092025-04-30T06:26:26.042ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19102025-04-30T06:26:26.042ZTRACtest_disk_metrics (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19112025-04-30T06:26:26.042ZTRACtest_disk_metrics (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19122025-04-30T06:26:26.042ZTRACtest_disk_metrics (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19132025-04-30T06:26:26.042ZDEBGtest_disk_metrics (ServiceBalancer): roles roles = RoleSet { roles: {} }
19142025-04-30T06:26:26.044ZDEBGtest_disk_metrics (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19152025-04-30T06:26:26.045ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 57001, target: Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3250, tv_nsec: 843269464 } })
19162025-04-30T06:26:26.045ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
19172025-04-30T06:26:26.045ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 37651, target: Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3250, tv_nsec: 843524422 } })
19182025-04-30T06:26:26.045ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
19192025-04-30T06:26:26.045ZTRACtest_disk_metrics (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 44 fe 01 00 00 01 00 00 00 00 00 01 24 30 62 66 D...........$0bf\n0010: 64 30 32 65 64 2d 64 64 35 35 2d 34 39 65 61 2d d02ed-dd55-49ea-\n0020: 61 66 37 38 2d 61 33 36 36 32 37 65 33 63 35 37 af78-a36627e3c57\n0030: 35 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 5.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:55455 req_id = cb9cc4c2-5656-4295-9298-3dfc682ee8fd
19202025-04-30T06:26:26.045ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17662,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 48,\n 98,\n 102,\n 100,\n 48,\n 50,\n 101,\n 100,\n 45,\n 100,\n 100,\n 53,\n 53,\n 45,\n 52,\n 57,\n 101,\n 97,\n 45,\n 97,\n 102,\n 55,\n 56,\n 45,\n 97,\n 51,\n 54,\n 54,\n 50,\n 55,\n 101,\n 51,\n 99,\n 53,\n 55,\n 53,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55455 req_id = cb9cc4c2-5656-4295-9298-3dfc682ee8fd
19212025-04-30T06:26:26.045ZDEBGtest_disk_metrics (store): query key key = 0bfd02ed-dd55-49ea-af78-a36627e3c575.host
19222025-04-30T06:26:26.045ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55455 query = LowerQuery { name: LowerName(Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal.")), original: Query { name: Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("0bfd02ed-dd55-49ea-af78-a36627e3c575.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cb9cc4c2-5656-4295-9298-3dfc682ee8fd
19232025-04-30T06:26:26.046ZTRACtest_disk_metrics (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 68 5e 01 00 00 01 00 00 00 00 00 01 24 61 62 65 h^..........$abe\n0010: 39 38 62 34 37 2d 34 63 34 33 2d 34 62 39 39 2d 98b47-4c43-4b99-\n0020: 62 36 33 30 2d 39 32 64 33 39 62 30 66 61 30 65 b630-92d39b0fa0e\n0030: 32 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 2.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:58223 req_id = d3b477d0-17df-4cd5-b906-66e57a417dae
19242025-04-30T06:26:26.046ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26718,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 97,\n 98,\n 101,\n 57,\n 56,\n 98,\n 52,\n 55,\n 45,\n 52,\n 99,\n 52,\n 51,\n 45,\n 52,\n 98,\n 57,\n 57,\n 45,\n 98,\n 54,\n 51,\n 48,\n 45,\n 57,\n 50,\n 100,\n 51,\n 57,\n 98,\n 48,\n 102,\n 97,\n 48,\n 101,\n 50,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58223 req_id = d3b477d0-17df-4cd5-b906-66e57a417dae
19252025-04-30T06:26:26.046ZDEBGtest_disk_metrics (store): query key key = abe98b47-4c43-4b99-b630-92d39b0fa0e2.host
19262025-04-30T06:26:26.046ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58223 query = LowerQuery { name: LowerName(Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal.")), original: Query { name: Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("abe98b47-4c43-4b99-b630-92d39b0fa0e2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d3b477d0-17df-4cd5-b906-66e57a417dae
19272025-04-30T06:26:26.047ZINFOtest_disk_metrics (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:57001] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:26:26.047ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 18.082108ms iteration = 2 status = {"addresses":["[::1]:57001"]}
19292025-04-30T06:26:26.047ZINFOtest_disk_metrics (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:37651] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:26:26.048ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 17.9479ms iteration = 2 status = {"addresses":["[::1]:37651"]}
19312025-04-30T06:26:26.048ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:26:26.048ZWARNtest_disk_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:26:26.048ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 23.787\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:26:26.048ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:26:26.048ZWARNtest_disk_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:26:26.048ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 10.05\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:26:26.049ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:26:26.054ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19392025-04-30T06:26:26.054ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19402025-04-30T06:26:26.054ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19412025-04-30T06:26:26.054ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19422025-04-30T06:26:26.055ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:26:26.061ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19442025-04-30T06:26:26.061ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19452025-04-30T06:26:26.065ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19462025-04-30T06:26:26.065ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19472025-04-30T06:26:26.065ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19482025-04-30T06:26:26.065ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19492025-04-30T06:26:26.066ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19502025-04-30T06:26:26.071ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19512025-04-30T06:26:26.072ZDEBGtest_disk_metrics (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19522025-04-30T06:26:26.072ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19532025-04-30T06:26:26.072ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19542025-04-30T06:26:26.072ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19552025-04-30T06:26:26.072ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19562025-04-30T06:26:26.072ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19572025-04-30T06:26:26.072ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19582025-04-30T06:26:26.073ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19592025-04-30T06:26:26.074ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
19602025-04-30T06:26:26.074ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19612025-04-30T06:26:26.074ZDEBGtest_disk_metrics (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19622025-04-30T06:26:26.074ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19632025-04-30T06:26:26.074ZDEBGtest_disk_metrics (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19642025-04-30T06:26:26.074ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19652025-04-30T06:26:26.075ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19662025-04-30T06:26:26.075ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19672025-04-30T06:26:26.075ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19682025-04-30T06:26:26.075ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19692025-04-30T06:26:26.076ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19702025-04-30T06:26:26.076ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19712025-04-30T06:26:26.076ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19722025-04-30T06:26:26.076ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19732025-04-30T06:26:26.077ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19742025-04-30T06:26:26.077ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19752025-04-30T06:26:26.082ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19762025-04-30T06:26:26.082ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19772025-04-30T06:26:26.082ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19782025-04-30T06:26:26.083ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19792025-04-30T06:26:26.084ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19802025-04-30T06:26:26.090ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19812025-04-30T06:26:26.090ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19822025-04-30T06:26:26.090ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19832025-04-30T06:26:26.091ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19842025-04-30T06:26:26.091ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19852025-04-30T06:26:26.096ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19862025-04-30T06:26:26.096ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19872025-04-30T06:26:26.096ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19882025-04-30T06:26:26.097ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19892025-04-30T06:26:26.097ZDEBGtest_disk_metrics (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
19902025-04-30T06:26:26.097ZDEBGtest_disk_metrics (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 0ea02e39-96ce-4633-b297-cd9e389fe2dd dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
19912025-04-30T06:26:26.097ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19922025-04-30T06:26:26.098ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19932025-04-30T06:26:26.098ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19942025-04-30T06:26:26.098ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19952025-04-30T06:26:26.098ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19962025-04-30T06:26:26.099ZDEBGtest_disk_metrics (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
19972025-04-30T06:26:26.099ZDEBGtest_disk_metrics (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01684541-7a4f-4a2b-9008-4c9dc3add00e dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
19982025-04-30T06:26:26.099ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19992025-04-30T06:26:26.100ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20002025-04-30T06:26:26.100ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20012025-04-30T06:26:26.100ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20022025-04-30T06:26:26.100ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20032025-04-30T06:26:26.101ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20042025-04-30T06:26:26.101ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20052025-04-30T06:26:26.101ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20062025-04-30T06:26:26.101ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20072025-04-30T06:26:26.102ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20082025-04-30T06:26:26.103ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20092025-04-30T06:26:26.112ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20102025-04-30T06:26:26.112ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20112025-04-30T06:26:26.112ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20122025-04-30T06:26:26.114ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20132025-04-30T06:26:26.115ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20142025-04-30T06:26:26.135ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20152025-04-30T06:26:26.135ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20162025-04-30T06:26:26.135ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20172025-04-30T06:26:26.136ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20182025-04-30T06:26:26.143ZDEBGtest_disk_metrics (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20192025-04-30T06:26:26.151ZDEBGtest_disk_metrics (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
20202025-04-30T06:26:26.151ZINFOtest_disk_metrics (nexus): Rack initialized
20212025-04-30T06:26:26.151ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20222025-04-30T06:26:26.151ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20232025-04-30T06:26:26.152ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20242025-04-30T06:26:26.152ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 55 41 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 UA..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56348 req_id = 42c4b2a6-8b85-4c16-baf5-65a72bbc8dfa
20252025-04-30T06:26:26.153ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21825,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56348 req_id = 42c4b2a6-8b85-4c16-baf5-65a72bbc8dfa
20262025-04-30T06:26:26.153ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
20272025-04-30T06:26:26.153ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20282025-04-30T06:26:26.153ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56348 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 42c4b2a6-8b85-4c16-baf5-65a72bbc8dfa
20292025-04-30T06:26:26.153ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20302025-04-30T06:26:26.162ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20312025-04-30T06:26:26.162ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20322025-04-30T06:26:26.162ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20332025-04-30T06:26:26.163ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20342025-04-30T06:26:26.165ZDEBGtest_disk_metrics (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 0ea02e39-96ce-4633-b297-cd9e389fe2dd dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20352025-04-30T06:26:26.165ZDEBGtest_disk_metrics (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20362025-04-30T06:26:26.165ZINFOtest_disk_metrics (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20372025-04-30T06:26:26.165ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 136.084349ms iteration = 2 status = {"generation":1}
20382025-04-30T06:26:26.165ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20392025-04-30T06:26:26.233ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:57001 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:57001] } uri = http://[::1]:57001/config
20402025-04-30T06:26:26.234ZINFOtest_disk_metrics (http): accepted connection kind = dns local_addr = [::1]:57001 remote_addr = [::1]:58066
20412025-04-30T06:26:26.236ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3251, tv_nsec: 34871058 } })
20422025-04-30T06:26:26.236ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
20432025-04-30T06:26:26.237ZTRACtest_disk_metrics (http): incoming request kind = dns local_addr = [::1]:57001 method = PUT remote_addr = [::1]:58066 req_id = 745120cd-67f2-4377-9b50-416d7a979596 uri = /config
20442025-04-30T06:26:26.238ZDEBGtest_disk_metrics (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01684541-7a4f-4a2b-9008-4c9dc3add00e dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20452025-04-30T06:26:26.238ZDEBGtest_disk_metrics (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20462025-04-30T06:26:26.238ZINFOtest_disk_metrics (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20472025-04-30T06:26:26.238ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 209.196671ms iteration = 2 status = {"generation":2}
20482025-04-30T06:26:26.241ZINFOtest_disk_metrics (store): attempting generation update new_generation = 1 req_id = 745120cd-67f2-4377-9b50-416d7a979596
20492025-04-30T06:26:26.241ZINFOtest_disk_metrics (store): updated generation new_generation = 1 req_id = 745120cd-67f2-4377-9b50-416d7a979596
20502025-04-30T06:26:26.241ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20512025-04-30T06:26:26.303ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:37651 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:37651] } uri = http://[::1]:37651/config
20522025-04-30T06:26:26.303ZINFOtest_disk_metrics (http): request completed kind = dns latency_us = 65799 local_addr = [::1]:57001 method = PUT remote_addr = [::1]:58066 req_id = 745120cd-67f2-4377-9b50-416d7a979596 response_code = 204 uri = /config
20532025-04-30T06:26:26.304ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20542025-04-30T06:26:26.304ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20552025-04-30T06:26:26.305ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20562025-04-30T06:26:26.305ZINFOtest_disk_metrics (http): accepted connection kind = dns local_addr = [::1]:37651 remote_addr = [::1]:59374
20572025-04-30T06:26:26.306ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0b 7e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .~..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60941 req_id = df4fcb71-598a-440c-a5e4-679a046a3df6
20582025-04-30T06:26:26.306ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2942,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60941 req_id = df4fcb71-598a-440c-a5e4-679a046a3df6
20592025-04-30T06:26:26.306ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20602025-04-30T06:26:26.306ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60941 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = df4fcb71-598a-440c-a5e4-679a046a3df6
20612025-04-30T06:26:26.306ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:57001 generation = 1 result = Ok(Response { url: "http://[::1]:57001/config", status: 204, headers: {"x-request-id": "745120cd-67f2-4377-9b50-416d7a979596", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} }) servers = DnsServersList { addresses: [[::1]:57001] }
20622025-04-30T06:26:26.306ZINFOtest_disk_metrics (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:57001 generation = 1 servers = DnsServersList { addresses: [[::1]:57001] }
20632025-04-30T06:26:26.309ZINFOtest_disk_metrics (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:57001] }
20642025-04-30T06:26:26.309ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 145.242877ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:57001":{"Ok":null}}}
20652025-04-30T06:26:26.385ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20662025-04-30T06:26:26.386ZTRACtest_disk_metrics (http): incoming request kind = dns local_addr = [::1]:37651 method = PUT remote_addr = [::1]:59374 req_id = b74cee87-27a2-4684-86c0-21966512e5aa uri = /config
20672025-04-30T06:26:26.387ZINFOtest_disk_metrics (store): attempting generation update new_generation = 2 req_id = b74cee87-27a2-4684-86c0-21966512e5aa
20682025-04-30T06:26:26.387ZINFOtest_disk_metrics (store): pruning trees for generations newer than 0
20692025-04-30T06:26:26.388ZDEBGtest_disk_metrics (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20702025-04-30T06:26:26.388ZDEBGtest_disk_metrics (store): updating current config
20712025-04-30T06:26:26.389ZDEBGtest_disk_metrics (store): flushing default tree
20722025-04-30T06:26:26.389ZINFOtest_disk_metrics (store): pruning trees for generations older than 2 keep = 3
20732025-04-30T06:26:26.389ZINFOtest_disk_metrics (store): updated generation new_generation = 2 req_id = b74cee87-27a2-4684-86c0-21966512e5aa
20742025-04-30T06:26:26.389ZINFOtest_disk_metrics (http): request completed kind = dns latency_us = 2948 local_addr = [::1]:37651 method = PUT remote_addr = [::1]:59374 req_id = b74cee87-27a2-4684-86c0-21966512e5aa response_code = 204 uri = /config
20752025-04-30T06:26:26.389ZINFOtest_disk_metrics (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:26:17.276114Z, time_modified: 2025-04-30T06:26:17.276114Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: b9d8e134-c9b0-412f-84f7-8316d5ab0644, db_silo: Silo { identity: SiloIdentity { id: b9d8e134-c9b0-412f-84f7-8316d5ab0644, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:26:25.183325Z, time_modified: 2025-04-30T06:26:25.183325Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: b9d8e134-c9b0-412f-84f7-8316d5ab0644, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b9d8e134-c9b0-412f-84f7-8316d5ab0644, db_silo: Silo { identity: SiloIdentity { id: b9d8e134-c9b0-412f-84f7-8316d5ab0644, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:26:25.183325Z, time_modified: 2025-04-30T06:26:25.183325Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20762025-04-30T06:26:26.390ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 360.011841ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"b9d8e134-c9b0-412f-84f7-8316d5ab0644","tls_certs":[]}},"default_endpoint":{"silo_id":"b9d8e134-c9b0-412f-84f7-8316d5ab0644","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo b9d8e134-c9b0-412f-84f7-8316d5ab0644 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20772025-04-30T06:26:26.391ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20782025-04-30T06:26:26.402ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20792025-04-30T06:26:26.402ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20802025-04-30T06:26:26.403ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:37651 generation = 2 result = Ok(Response { url: "http://[::1]:37651/config", status: 204, headers: {"x-request-id": "b74cee87-27a2-4684-86c0-21966512e5aa", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} }) servers = DnsServersList { addresses: [[::1]:37651] }
20812025-04-30T06:26:26.403ZINFOtest_disk_metrics (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:37651 generation = 2 servers = DnsServersList { addresses: [[::1]:37651] }
20822025-04-30T06:26:26.405ZINFOtest_disk_metrics (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:37651] }
20832025-04-30T06:26:26.405ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 163.369153ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:37651":{"Ok":null}}}
20842025-04-30T06:26:26.407ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e5 5d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .]..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61592 req_id = 0afd8630-51fc-4c76-a06b-771e9dafa2e4
20852025-04-30T06:26:26.407ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58717,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61592 req_id = 0afd8630-51fc-4c76-a06b-771e9dafa2e4
20862025-04-30T06:26:26.407ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
20872025-04-30T06:26:26.408ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58717, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61592 req_id = 0afd8630-51fc-4c76-a06b-771e9dafa2e4
20882025-04-30T06:26:26.408ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20892025-04-30T06:26:26.408ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20902025-04-30T06:26:26.408ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20912025-04-30T06:26:26.409ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20922025-04-30T06:26:26.411ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20932025-04-30T06:26:26.411ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
20942025-04-30T06:26:26.411ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
20952025-04-30T06:26:26.411ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
20962025-04-30T06:26:26.427ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20972025-04-30T06:26:26.427ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20982025-04-30T06:26:26.427ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20992025-04-30T06:26:26.429ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21002025-04-30T06:26:26.430ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6f30bbd-2985-481a-b41c-27c90b0a64fc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21012025-04-30T06:26:26.430ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
21022025-04-30T06:26:26.442ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21032025-04-30T06:26:26.442ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21042025-04-30T06:26:26.442ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21052025-04-30T06:26:26.443ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21062025-04-30T06:26:26.445ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21072025-04-30T06:26:26.454ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21082025-04-30T06:26:26.454ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21092025-04-30T06:26:26.454ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21102025-04-30T06:26:26.454ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21112025-04-30T06:26:26.454ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21122025-04-30T06:26:26.455ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21132025-04-30T06:26:26.455ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21142025-04-30T06:26:26.457ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffa71f49-d268-4bc7-9443-0e0a9546b32c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21152025-04-30T06:26:26.457ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
21162025-04-30T06:26:26.460ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21172025-04-30T06:26:26.460ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21182025-04-30T06:26:26.460ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21192025-04-30T06:26:26.461ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21202025-04-30T06:26:26.467ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21212025-04-30T06:26:26.467ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21222025-04-30T06:26:26.467ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21232025-04-30T06:26:26.468ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21242025-04-30T06:26:26.469ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdbf62ad-d93e-4c9e-91d5-9858bb2afd1e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21252025-04-30T06:26:26.470ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
21262025-04-30T06:26:26.471ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21272025-04-30T06:26:26.484ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21282025-04-30T06:26:26.484ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21292025-04-30T06:26:26.484ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21302025-04-30T06:26:26.485ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21312025-04-30T06:26:26.488ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21322025-04-30T06:26:26.488ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21332025-04-30T06:26:26.489ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21342025-04-30T06:26:26.490ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8a5abff-0660-42a5-9402-2871251e63ca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21352025-04-30T06:26:26.492ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
21362025-04-30T06:26:26.497ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21372025-04-30T06:26:26.497ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21382025-04-30T06:26:26.498ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21392025-04-30T06:26:26.499ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffa5b4f4-1640-4258-a523-c5ca477bedaa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21402025-04-30T06:26:26.499ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
21412025-04-30T06:26:26.500ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21422025-04-30T06:26:26.500ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21432025-04-30T06:26:26.501ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21442025-04-30T06:26:26.503ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c377444-e746-42a9-905a-34e8dfa1c040", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21452025-04-30T06:26:26.503ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
21462025-04-30T06:26:26.504ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21472025-04-30T06:26:26.504ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21482025-04-30T06:26:26.504ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21492025-04-30T06:26:26.504ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21502025-04-30T06:26:26.506ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21512025-04-30T06:26:26.508ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31ab55b3-274e-4127-b438-3bd60ec1a2d2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21522025-04-30T06:26:26.508ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
21532025-04-30T06:26:26.509ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21542025-04-30T06:26:26.509ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21552025-04-30T06:26:26.509ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21562025-04-30T06:26:26.511ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21572025-04-30T06:26:26.519ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21582025-04-30T06:26:26.519ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21592025-04-30T06:26:26.519ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21602025-04-30T06:26:26.520ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21612025-04-30T06:26:26.522ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21622025-04-30T06:26:26.531ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21632025-04-30T06:26:26.531ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21642025-04-30T06:26:26.531ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21652025-04-30T06:26:26.531ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21662025-04-30T06:26:26.532ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21672025-04-30T06:26:26.533ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06010d59-1f57-4bba-9822-0e90a9083f45", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21682025-04-30T06:26:26.533ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
21692025-04-30T06:26:26.534ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21702025-04-30T06:26:26.534ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21712025-04-30T06:26:26.535ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21722025-04-30T06:26:26.536ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81e723f2-1227-40b1-ba9a-b48fce781f61", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21732025-04-30T06:26:26.536ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
21742025-04-30T06:26:26.538ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21752025-04-30T06:26:26.538ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21762025-04-30T06:26:26.540ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21772025-04-30T06:26:26.542ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af76d3e6-da7a-4810-823a-8349b2f45872", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21782025-04-30T06:26:26.544ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
21792025-04-30T06:26:26.544ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21802025-04-30T06:26:26.544ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21812025-04-30T06:26:26.545ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21822025-04-30T06:26:26.545ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65377c93-09aa-470a-a3ec-60a7138b1151", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21832025-04-30T06:26:26.545ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
21842025-04-30T06:26:26.547ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
21852025-04-30T06:26:26.548ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21862025-04-30T06:26:26.548ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21872025-04-30T06:26:26.548ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21882025-04-30T06:26:26.548ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21892025-04-30T06:26:26.549ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceaeb50b-0b61-44fd-b4d1-01883c55a953", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21902025-04-30T06:26:26.549ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
21912025-04-30T06:26:26.551ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21922025-04-30T06:26:26.551ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21932025-04-30T06:26:26.551ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21942025-04-30T06:26:26.552ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21952025-04-30T06:26:26.552ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4b9ca13-d592-423b-ad7c-6d2bf3d54b20", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
21962025-04-30T06:26:26.552ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
21972025-04-30T06:26:26.557ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21982025-04-30T06:26:26.557ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21992025-04-30T06:26:26.557ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22002025-04-30T06:26:26.557ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22012025-04-30T06:26:26.561ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22022025-04-30T06:26:26.573ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22032025-04-30T06:26:26.573ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22042025-04-30T06:26:26.573ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22052025-04-30T06:26:26.574ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22062025-04-30T06:26:26.574ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22072025-04-30T06:26:26.574ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22082025-04-30T06:26:26.575ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22092025-04-30T06:26:26.577ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ff3d69c-bf3c-4f82-8aff-5bcc138d0349", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22102025-04-30T06:26:26.577ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
22112025-04-30T06:26:26.578ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22122025-04-30T06:26:26.578ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22132025-04-30T06:26:26.578ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22142025-04-30T06:26:26.579ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22152025-04-30T06:26:26.583ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22162025-04-30T06:26:26.583ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22172025-04-30T06:26:26.583ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22182025-04-30T06:26:26.584ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22192025-04-30T06:26:26.587ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18d7b97b-a8d1-4246-af92-207c0c8cd48b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22202025-04-30T06:26:26.587ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
22212025-04-30T06:26:26.590ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22222025-04-30T06:26:26.617ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22232025-04-30T06:26:26.617ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22242025-04-30T06:26:26.617ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22252025-04-30T06:26:26.617ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22262025-04-30T06:26:26.618ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "886a52e3-64cc-45d5-81fc-fca1ea089679", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22272025-04-30T06:26:26.618ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
22282025-04-30T06:26:26.621ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d561ee41-ddb9-4c5a-b1f3-d79dac976b64", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22292025-04-30T06:26:26.621ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
22302025-04-30T06:26:26.623ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22312025-04-30T06:26:26.623ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22322025-04-30T06:26:26.624ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22332025-04-30T06:26:26.628ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22342025-04-30T06:26:26.628ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22352025-04-30T06:26:26.629ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22362025-04-30T06:26:26.632ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "098c5202-4d97-4e9e-8d2e-d2924294c1ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22372025-04-30T06:26:26.633ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
22382025-04-30T06:26:26.633ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22392025-04-30T06:26:26.633ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22402025-04-30T06:26:26.635ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22412025-04-30T06:26:26.638ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22422025-04-30T06:26:26.638ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22432025-04-30T06:26:26.639ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22442025-04-30T06:26:26.640ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "764c2b67-3cfc-449c-9ef0-739b96a6e580", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22452025-04-30T06:26:26.640ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
22462025-04-30T06:26:26.641ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d076e28-a719-4f71-95c6-1dfac0abd74d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22472025-04-30T06:26:26.642ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
22482025-04-30T06:26:26.642ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22492025-04-30T06:26:26.642ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22502025-04-30T06:26:26.642ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22512025-04-30T06:26:26.642ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22522025-04-30T06:26:26.644ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22532025-04-30T06:26:26.647ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22542025-04-30T06:26:26.647ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22552025-04-30T06:26:26.647ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22562025-04-30T06:26:26.648ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22572025-04-30T06:26:26.649ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ea7d936-bbe8-4ca2-ac08-76dfe38cf3f6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22582025-04-30T06:26:26.649ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
22592025-04-30T06:26:26.651ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43d3f3f7-41c9-4bc8-9363-4ff9cf29b8de", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22602025-04-30T06:26:26.651ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
22612025-04-30T06:26:26.652ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22622025-04-30T06:26:26.652ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22632025-04-30T06:26:26.652ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22642025-04-30T06:26:26.654ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22652025-04-30T06:26:26.657ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fff7db2f-1351-4bcb-a0bc-1041ea29de3b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22662025-04-30T06:26:26.657ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
22672025-04-30T06:26:26.658ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22682025-04-30T06:26:26.671ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22692025-04-30T06:26:26.671ZTRACtest_disk_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22702025-04-30T06:26:26.671ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22712025-04-30T06:26:26.671ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22722025-04-30T06:26:26.671ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22732025-04-30T06:26:26.671ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22742025-04-30T06:26:26.674ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22752025-04-30T06:26:26.676ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d87bb9e-2e79-4504-8da1-bea904320116", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22762025-04-30T06:26:26.676ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
22772025-04-30T06:26:26.678ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22782025-04-30T06:26:26.678ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22792025-04-30T06:26:26.678ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22802025-04-30T06:26:26.679ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22812025-04-30T06:26:26.683ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22822025-04-30T06:26:26.683ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22832025-04-30T06:26:26.683ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22842025-04-30T06:26:26.684ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22852025-04-30T06:26:26.685ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e21ed43-75bf-42f8-bd12-bed5a976af4e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22862025-04-30T06:26:26.686ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
22872025-04-30T06:26:26.686ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22882025-04-30T06:26:26.698ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22892025-04-30T06:26:26.698ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22902025-04-30T06:26:26.698ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22912025-04-30T06:26:26.699ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22922025-04-30T06:26:26.704ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6df3ffdb-fb4c-4b40-8ea8-12115b5c15bb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22932025-04-30T06:26:26.704ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
22942025-04-30T06:26:26.708ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0f2323c-04a6-4fc6-8c7a-c2b9770c5e9e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
22952025-04-30T06:26:26.708ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
22962025-04-30T06:26:26.713ZDEBGtest_disk_metrics (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 83d74e6a-b808-43fc-b550-3cbbfe2a2065, kind: Service { id: abe98b47-4c43-4b99-b630-92d39b0fa0e2 }, name: Name("external-dns-abe98b47-4c43-4b99-b630-92d39b0fa0e2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: c14e81ea-5508-4e28-922b-87676d66580c, kind: Service { id: 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe }, name: Name("nexus-06ed8ede-6e59-42fb-a0cc-51ac048d6bbe"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
22972025-04-30T06:26:26.713ZTRACtest_disk_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
22982025-04-30T06:26:26.713ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22992025-04-30T06:26:26.713ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23002025-04-30T06:26:26.713ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23012025-04-30T06:26:26.713ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23022025-04-30T06:26:26.720ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23032025-04-30T06:26:26.724ZDEBGtest_disk_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23042025-04-30T06:26:26.724ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23052025-04-30T06:26:26.724ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23062025-04-30T06:26:26.725ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23072025-04-30T06:26:26.726ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3e8ee86-3892-4518-a626-c2faa057be24", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23082025-04-30T06:26:26.727ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
23092025-04-30T06:26:26.728ZTRACtest_disk_metrics (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:26:25.460035Z, allowed_ips: None }
23102025-04-30T06:26:26.728ZDEBGtest_disk_metrics (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23112025-04-30T06:26:26.728ZDEBGtest_disk_metrics (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 83d74e6a-b808-43fc-b550-3cbbfe2a2065, kind: Service { id: abe98b47-4c43-4b99-b630-92d39b0fa0e2 }, name: Name("external-dns-abe98b47-4c43-4b99-b630-92d39b0fa0e2"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: c14e81ea-5508-4e28-922b-87676d66580c, kind: Service { id: 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe }, name: Name("nexus-06ed8ede-6e59-42fb-a0cc-51ac048d6bbe"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23122025-04-30T06:26:26.728ZDEBGtest_disk_metrics (InternalApi): resolved 2 rules for sleds
23132025-04-30T06:26:26.730ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "202238ca-d05c-4c4b-8922-e7774271d11e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23142025-04-30T06:26:26.731ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
23152025-04-30T06:26:26.733ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7dfb48c-47bf-41e9-a37c-8c6bddd86321", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23162025-04-30T06:26:26.733ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
23172025-04-30T06:26:26.734ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f57e325a-2fa7-4201-82b8-04daf9a36c09", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23182025-04-30T06:26:26.734ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
23192025-04-30T06:26:26.736ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc962003-d09b-44f6-90ed-a53ccd0fd134", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23202025-04-30T06:26:26.736ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
23212025-04-30T06:26:26.737ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63c5c674-b410-47a2-bdb2-835a52670ec1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23222025-04-30T06:26:26.737ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
23232025-04-30T06:26:26.738ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6a38310-af4c-4660-bae1-89c4158a4528", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23242025-04-30T06:26:26.739ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
23252025-04-30T06:26:26.740ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48ac9b12-c4f1-432d-b590-54a9dfa523e3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23262025-04-30T06:26:26.740ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
23272025-04-30T06:26:26.742ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f62fd5ca-74ea-4738-8c68-8c7449fb3381", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23282025-04-30T06:26:26.742ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
23292025-04-30T06:26:26.744ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33e23192-279c-41d4-887c-1aae3ef45451", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23302025-04-30T06:26:26.745ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
23312025-04-30T06:26:26.747ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52db45d2-cd37-4950-879b-3b258f66318c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23322025-04-30T06:26:26.747ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
23332025-04-30T06:26:26.749ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8377f186-ae02-4019-8c5f-35bf121d75fa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23342025-04-30T06:26:26.749ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
23352025-04-30T06:26:26.749ZDEBGtest_disk_metrics (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:26:18.831805Z, time_modified: 2025-04-30T06:26:18.831805Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(50989), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(58149) }]
23362025-04-30T06:26:26.749ZDEBGtest_disk_metrics (InternalApi): sending firewall rules to sled agents
23372025-04-30T06:26:26.750ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23382025-04-30T06:26:26.750ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23392025-04-30T06:26:26.750ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23402025-04-30T06:26:26.753ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "734b0adb-93f3-497c-a648-f082d27acf3e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23412025-04-30T06:26:26.754ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
23422025-04-30T06:26:26.754ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23432025-04-30T06:26:26.754ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23442025-04-30T06:26:26.754ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23452025-04-30T06:26:26.754ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23462025-04-30T06:26:26.755ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23472025-04-30T06:26:26.757ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17243c41-2bf2-449b-9d08-457652323132", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23482025-04-30T06:26:26.758ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
23492025-04-30T06:26:26.758ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23502025-04-30T06:26:26.764ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23512025-04-30T06:26:26.833ZDEBGtest_disk_metrics (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:50989/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23522025-04-30T06:26:26.842ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:61141 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23532025-04-30T06:26:26.850ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = PUT remote_addr = [::1]:61141 req_id = 1274fcc4-d3fb-4e4e-ae89-dd061a63d903 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23542025-04-30T06:26:26.850ZTRACtest_disk_metrics (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: cf f1 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:64280 req_id = bf30dc49-b49e-45b9-b02e-1d1de791bf19
23552025-04-30T06:26:26.850ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53233,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64280 req_id = bf30dc49-b49e-45b9-b02e-1d1de791bf19
23562025-04-30T06:26:26.851ZDEBGtest_disk_metrics (store): query key key = _crucible-pantry._tcp
23572025-04-30T06:26:26.851ZDEBGtest_disk_metrics (store): query key key = 58007544-db95-41ae-9c86-658edc7605fe.host
23582025-04-30T06:26:26.851ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64280 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35636, target: Name("58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal") })) }] req_id = bf30dc49-b49e-45b9-b02e-1d1de791bf19
23592025-04-30T06:26:26.852ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b1f1f23-1341-4f19-9211-1289a35bc5cc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23602025-04-30T06:26:26.853ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
23612025-04-30T06:26:26.854ZINFOtest_disk_metrics (dropshot): request completed latency_us = 4321 local_addr = [::1]:50989 method = PUT remote_addr = [::1]:61141 req_id = 1274fcc4-d3fb-4e4e-ae89-dd061a63d903 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23622025-04-30T06:26:26.857ZDEBGtest_disk_metrics (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:50989/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "1274fcc4-d3fb-4e4e-ae89-dd061a63d903", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
23632025-04-30T06:26:26.860ZINFOtest_disk_metrics (InternalApi): updated firewall rules on 1 sleds
23642025-04-30T06:26:26.860ZINFOtest_disk_metrics (nexus): plumbed initial IP allowlist
23652025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = / versions = all
23662025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /access versions = all
23672025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /assets/{path} versions = all
23682025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /device/auth versions = all
23692025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /device/confirm versions = all
23702025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /device/success versions = all
23712025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /device/token versions = all
23722025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /device/verify versions = all
23732025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /experimental/v1/probes versions = all
23742025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /experimental/v1/probes versions = all
23752025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /experimental/v1/probes/{probe} versions = all
23762025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /experimental/v1/probes/{probe} versions = all
23772025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /experimental/v1/system/support-bundles versions = all
23782025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /experimental/v1/system/support-bundles versions = all
23792025-04-30T06:26:26.953ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23802025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23812025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23822025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23832025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23842025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23852025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
23862025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /images versions = all
23872025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /login versions = all
23882025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /login/{silo_name}/local versions = all
23892025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
23902025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
23912025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
23922025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /lookup/{path} versions = all
23932025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /projects/{path} versions = all
23942025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /projects-new versions = all
23952025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /settings/{path} versions = all
23962025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /system/{path} versions = all
23972025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /utilization versions = all
23982025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/affinity-groups versions = all
23992025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/affinity-groups versions = all
24002025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
24012025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
24022025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
24032025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
24042025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24052025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24062025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24072025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/anti-affinity-groups versions = all
24082025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/anti-affinity-groups versions = all
24092025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24102025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24112025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24122025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24132025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24142025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24152025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24162025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/certificates versions = all
24172025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/certificates versions = all
24182025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/certificates/{certificate} versions = all
24192025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/certificates/{certificate} versions = all
24202025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/disks versions = all
24212025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/disks versions = all
24222025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/disks/{disk} versions = all
24232025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/disks/{disk} versions = all
24242025-04-30T06:26:26.954ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24252025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24262025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24272025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/disks/{disk}/finalize versions = all
24282025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24292025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/floating-ips versions = all
24302025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/floating-ips versions = all
24312025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24322025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24332025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24342025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24352025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24362025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/groups versions = all
24372025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/groups/{group_id} versions = all
24382025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/images versions = all
24392025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/images versions = all
24402025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/images/{image} versions = all
24412025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/images/{image} versions = all
24422025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/images/{image}/demote versions = all
24432025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/images/{image}/promote versions = all
24442025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances versions = all
24452025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/instances versions = all
24462025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/instances/{instance} versions = all
24472025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance} versions = all
24482025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/instances/{instance} versions = all
24492025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24502025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24512025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance}/disks versions = all
24522025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24532025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24542025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance}/external-ips versions = all
24552025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24562025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24572025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/instances/{instance}/reboot versions = all
24582025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance}/serial-console versions = all
24592025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24602025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24612025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/instances/{instance}/start versions = all
24622025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/instances/{instance}/stop versions = all
24632025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24642025-04-30T06:26:26.955ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24652025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24662025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/internet-gateway-ip-pools versions = all
24672025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/internet-gateway-ip-pools versions = all
24682025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24692025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/internet-gateways versions = all
24702025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/internet-gateways versions = all
24712025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24722025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/internet-gateways/{gateway} versions = all
24732025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/ip-pools versions = all
24742025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/ip-pools/{pool} versions = all
24752025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/login/{silo_name}/local versions = all
24762025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/logout versions = all
24772025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/me versions = all
24782025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/me/groups versions = all
24792025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/me/ssh-keys versions = all
24802025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/me/ssh-keys versions = all
24812025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
24822025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
24832025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/metrics/{metric_name} versions = all
24842025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/network-interfaces versions = all
24852025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/network-interfaces versions = all
24862025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/network-interfaces/{interface} versions = all
24872025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/network-interfaces/{interface} versions = all
24882025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/network-interfaces/{interface} versions = all
24892025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/ping versions = all
24902025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/policy versions = all
24912025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/policy versions = all
24922025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/projects versions = all
24932025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/projects versions = all
24942025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/projects/{project} versions = all
24952025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/projects/{project} versions = all
24962025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/projects/{project} versions = all
24972025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/projects/{project}/policy versions = all
24982025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/projects/{project}/policy versions = all
24992025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/snapshots versions = all
25002025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/snapshots versions = all
25012025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/snapshots/{snapshot} versions = all
25022025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/snapshots/{snapshot} versions = all
25032025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/disks versions = all
25042025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
25052025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
25062025-04-30T06:26:26.956ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/racks versions = all
25072025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25082025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/sleds versions = all
25092025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/hardware/sleds versions = all
25102025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25112025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25122025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25132025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25142025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25152025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/switch-port versions = all
25162025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25172025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25182025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25192025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25202025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25212025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/switches versions = all
25222025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25232025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/identity-providers versions = all
25242025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/identity-providers/local/users versions = all
25252025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25262025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25272025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/identity-providers/saml versions = all
25282025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25292025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/ip-pools versions = all
25302025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/ip-pools versions = all
25312025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25322025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/ip-pools/{pool} versions = all
25332025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25342025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25352025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25362025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25372025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25382025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25392025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25402025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25412025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25422025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/ip-pools-service versions = all
25432025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25442025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25452025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25462025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/metrics/{metric_name} versions = all
25472025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/address-lot versions = all
25482025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/networking/address-lot versions = all
25492025-04-30T06:26:26.957ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25502025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25512025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/allow-list versions = all
25522025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/networking/allow-list versions = all
25532025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/networking/bfd-disable versions = all
25542025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/networking/bfd-enable versions = all
25552025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bfd-status versions = all
25562025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/networking/bgp versions = all
25572025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bgp versions = all
25582025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/networking/bgp versions = all
25592025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25602025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25612025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25622025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25632025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bgp-exported versions = all
25642025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bgp-message-history versions = all
25652025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25662025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/bgp-status versions = all
25672025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/loopback-address versions = all
25682025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/networking/loopback-address versions = all
25692025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25702025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25712025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/switch-port-settings versions = all
25722025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/networking/switch-port-settings versions = all
25732025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25742025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/policy versions = all
25752025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/policy versions = all
25762025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/roles versions = all
25772025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/roles/{role_name} versions = all
25782025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/silo-quotas versions = all
25792025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/silos versions = all
25802025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/silos versions = all
25812025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/system/silos/{silo} versions = all
25822025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/silos/{silo} versions = all
25832025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
25842025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/silos/{silo}/policy versions = all
25852025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/silos/{silo}/policy versions = all
25862025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/silos/{silo}/quotas versions = all
25872025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
25882025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/system/timeseries/query versions = all
25892025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/timeseries/schemas versions = all
25902025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/update/repository versions = all
25912025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/update/repository/{system_version} versions = all
25922025-04-30T06:26:26.958ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/update/target-release versions = all
25932025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/system/update/target-release versions = all
25942025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/users versions = all
25952025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/users/{user_id} versions = all
25962025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/users-builtin versions = all
25972025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/users-builtin/{user} versions = all
25982025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/utilization/silos versions = all
25992025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/system/utilization/silos/{silo} versions = all
26002025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/timeseries/query versions = all
26012025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/users versions = all
26022025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/utilization versions = all
26032025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-firewall-rules versions = all
26042025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/vpc-firewall-rules versions = all
26052025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-router-routes versions = all
26062025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/vpc-router-routes versions = all
26072025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26082025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-router-routes/{route} versions = all
26092025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26102025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-routers versions = all
26112025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/vpc-routers versions = all
26122025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/vpc-routers/{router} versions = all
26132025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-routers/{router} versions = all
26142025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/vpc-routers/{router} versions = all
26152025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-subnets versions = all
26162025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/vpc-subnets versions = all
26172025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26182025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26192025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26202025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26212025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpcs versions = all
26222025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/vpcs versions = all
26232025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/vpcs/{vpc} versions = all
26242025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/vpcs/{vpc} versions = all
26252025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/vpcs/{vpc} versions = all
26262025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/webhooks/deliveries versions = all
26272025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26282025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/webhooks/event-classes versions = all
26292025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/webhooks/receivers versions = all
26302025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/webhooks/receivers versions = all
26312025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26322025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26332025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26342025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26352025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26362025-04-30T06:26:26.959ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26372025-04-30T06:26:26.960ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = GET path = /v1/webhooks/secrets versions = all
26382025-04-30T06:26:26.960ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = POST path = /v1/webhooks/secrets versions = all
26392025-04-30T06:26:26.960ZDEBGtest_disk_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:33614 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26402025-04-30T06:26:26.960ZINFOtest_disk_metrics (dropshot_external): listening local_addr = 127.0.0.1:33614
26412025-04-30T06:26:26.960ZDEBGtest_disk_metrics (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:33614
26422025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = / versions = all
26432025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /access versions = all
26442025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /assets/{path} versions = all
26452025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /device/auth versions = all
26462025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /device/confirm versions = all
26472025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /device/success versions = all
26482025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /device/token versions = all
26492025-04-30T06:26:27.020ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /device/verify versions = all
26502025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /experimental/v1/probes versions = all
26512025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /experimental/v1/probes versions = all
26522025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26532025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /experimental/v1/probes/{probe} versions = all
26542025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /experimental/v1/system/support-bundles versions = all
26552025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /experimental/v1/system/support-bundles versions = all
26562025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26572025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26582025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26592025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26602025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26612025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26622025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26632025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /images versions = all
26642025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /login versions = all
26652025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /login/{silo_name}/local versions = all
26662025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26672025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26682025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26692025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /lookup/{path} versions = all
26702025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /projects/{path} versions = all
26712025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /projects-new versions = all
26722025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /settings/{path} versions = all
26732025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /system/{path} versions = all
26742025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /utilization versions = all
26752025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/affinity-groups versions = all
26762025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/affinity-groups versions = all
26772025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
26782025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
26792025-04-30T06:26:27.021ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
26802025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
26812025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26822025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26832025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26842025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/anti-affinity-groups versions = all
26852025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/anti-affinity-groups versions = all
26862025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26872025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26882025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26892025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
26902025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26912025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26922025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26932025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/certificates versions = all
26942025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/certificates versions = all
26952025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/certificates/{certificate} versions = all
26962025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/certificates/{certificate} versions = all
26972025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/disks versions = all
26982025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/disks versions = all
26992025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/disks/{disk} versions = all
27002025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/disks/{disk} versions = all
27012025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/disks/{disk}/bulk-write versions = all
27022025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
27032025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
27042025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/disks/{disk}/finalize versions = all
27052025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
27062025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/floating-ips versions = all
27072025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/floating-ips versions = all
27082025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27092025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27102025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27112025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27122025-04-30T06:26:27.022ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27132025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/groups versions = all
27142025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/groups/{group_id} versions = all
27152025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/images versions = all
27162025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/images versions = all
27172025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/images/{image} versions = all
27182025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/images/{image} versions = all
27192025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/images/{image}/demote versions = all
27202025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/images/{image}/promote versions = all
27212025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances versions = all
27222025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/instances versions = all
27232025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/instances/{instance} versions = all
27242025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance} versions = all
27252025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/instances/{instance} versions = all
27262025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27272025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27282025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance}/disks versions = all
27292025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27302025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27312025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance}/external-ips versions = all
27322025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27332025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27342025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/instances/{instance}/reboot versions = all
27352025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance}/serial-console versions = all
27362025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27372025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27382025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/instances/{instance}/start versions = all
27392025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/instances/{instance}/stop versions = all
27402025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27412025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27422025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27432025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/internet-gateway-ip-pools versions = all
27442025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/internet-gateway-ip-pools versions = all
27452025-04-30T06:26:27.023ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27462025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/internet-gateways versions = all
27472025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/internet-gateways versions = all
27482025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27492025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/internet-gateways/{gateway} versions = all
27502025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/ip-pools versions = all
27512025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/ip-pools/{pool} versions = all
27522025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/login/{silo_name}/local versions = all
27532025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/logout versions = all
27542025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/me versions = all
27552025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/me/groups versions = all
27562025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/me/ssh-keys versions = all
27572025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/me/ssh-keys versions = all
27582025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27592025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27602025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/metrics/{metric_name} versions = all
27612025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/network-interfaces versions = all
27622025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/network-interfaces versions = all
27632025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27642025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/network-interfaces/{interface} versions = all
27652025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/network-interfaces/{interface} versions = all
27662025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/ping versions = all
27672025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/policy versions = all
27682025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/policy versions = all
27692025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/projects versions = all
27702025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/projects versions = all
27712025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/projects/{project} versions = all
27722025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/projects/{project} versions = all
27732025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/projects/{project} versions = all
27742025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/projects/{project}/policy versions = all
27752025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/projects/{project}/policy versions = all
27762025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/snapshots versions = all
27772025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/snapshots versions = all
27782025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/snapshots/{snapshot} versions = all
27792025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/snapshots/{snapshot} versions = all
27802025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/disks versions = all
27812025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27822025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
27832025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/racks versions = all
27842025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
27852025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/sleds versions = all
27862025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/hardware/sleds versions = all
27872025-04-30T06:26:27.024ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
27882025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
27892025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
27902025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
27912025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
27922025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/switch-port versions = all
27932025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27942025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27952025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
27962025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
27972025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
27982025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/switches versions = all
27992025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
28002025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/identity-providers versions = all
28012025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/identity-providers/local/users versions = all
28022025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
28032025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
28042025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/identity-providers/saml versions = all
28052025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
28062025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/ip-pools versions = all
28072025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/ip-pools versions = all
28082025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28092025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/ip-pools/{pool} versions = all
28102025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28112025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28122025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28132025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28142025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28152025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28162025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28172025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28182025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28192025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/ip-pools-service versions = all
28202025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28212025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28222025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28232025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/metrics/{metric_name} versions = all
28242025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/address-lot versions = all
28252025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/networking/address-lot versions = all
28262025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28272025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28282025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/allow-list versions = all
28292025-04-30T06:26:27.025ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/networking/allow-list versions = all
28302025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/networking/bfd-disable versions = all
28312025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/networking/bfd-enable versions = all
28322025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bfd-status versions = all
28332025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/networking/bgp versions = all
28342025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bgp versions = all
28352025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/networking/bgp versions = all
28362025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28372025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28382025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28392025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28402025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bgp-exported versions = all
28412025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bgp-message-history versions = all
28422025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28432025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/bgp-status versions = all
28442025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/loopback-address versions = all
28452025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/networking/loopback-address versions = all
28462025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28472025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28482025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/switch-port-settings versions = all
28492025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/networking/switch-port-settings versions = all
28502025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28512025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/policy versions = all
28522025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/policy versions = all
28532025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/roles versions = all
28542025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/roles/{role_name} versions = all
28552025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/silo-quotas versions = all
28562025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/silos versions = all
28572025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/silos versions = all
28582025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/system/silos/{silo} versions = all
28592025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/silos/{silo} versions = all
28602025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28612025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/silos/{silo}/policy versions = all
28622025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28632025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28642025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28652025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/system/timeseries/query versions = all
28662025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/timeseries/schemas versions = all
28672025-04-30T06:26:27.026ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/update/repository versions = all
28682025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/update/repository/{system_version} versions = all
28692025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/update/target-release versions = all
28702025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/system/update/target-release versions = all
28712025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/users versions = all
28722025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/users/{user_id} versions = all
28732025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/users-builtin versions = all
28742025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/users-builtin/{user} versions = all
28752025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/utilization/silos versions = all
28762025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28772025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/timeseries/query versions = all
28782025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/users versions = all
28792025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/utilization versions = all
28802025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-firewall-rules versions = all
28812025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/vpc-firewall-rules versions = all
28822025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-router-routes versions = all
28832025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/vpc-router-routes versions = all
28842025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
28852025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-router-routes/{route} versions = all
28862025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/vpc-router-routes/{route} versions = all
28872025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-routers versions = all
28882025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/vpc-routers versions = all
28892025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/vpc-routers/{router} versions = all
28902025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-routers/{router} versions = all
28912025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/vpc-routers/{router} versions = all
28922025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-subnets versions = all
28932025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/vpc-subnets versions = all
28942025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
28952025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-subnets/{subnet} versions = all
28962025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
28972025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
28982025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpcs versions = all
28992025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/vpcs versions = all
29002025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/vpcs/{vpc} versions = all
29012025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/vpcs/{vpc} versions = all
29022025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/vpcs/{vpc} versions = all
29032025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/webhooks/deliveries versions = all
29042025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
29052025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/webhooks/event-classes versions = all
29062025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/webhooks/receivers versions = all
29072025-04-30T06:26:27.027ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/webhooks/receivers versions = all
29082025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29092025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29102025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29112025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29122025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29132025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29142025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = GET path = /v1/webhooks/secrets versions = all
29152025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = POST path = /v1/webhooks/secrets versions = all
29162025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:35580 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29172025-04-30T06:26:27.028ZINFOtest_disk_metrics (dropshot_external_techport): listening local_addr = 127.0.0.1:35580
29182025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:35580
29192025-04-30T06:26:27.028ZDEBGtest_disk_metrics (nexus-producer-server): registered DTrace probes
29202025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = 127.0.0.1:39352 method = GET path = /{producer_id} versions = all
29212025-04-30T06:26:27.028ZINFOtest_disk_metrics (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:39352
29222025-04-30T06:26:27.028ZDEBGtest_disk_metrics (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:39352
29232025-04-30T06:26:27.028ZDEBGtest_disk_metrics (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:39352
29242025-04-30T06:26:27.028ZDEBGtest_disk_metrics (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:39551
29252025-04-30T06:26:27.028ZDEBGtest_disk_metrics (nexus-producer-server): starting producer registration task
29262025-04-30T06:26:27.028ZINFOtest_disk_metrics (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:39352 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
29272025-04-30T06:26:27.028ZDEBGtest_disk_metrics (init_with_steps): Running step start_oximeter
29282025-04-30T06:26:27.028ZDEBGtest_disk_metrics: Starting Oximeter
29292025-04-30T06:26:27.028ZDEBGtest_disk_metrics (oximeter): registered DTrace probes
29302025-04-30T06:26:27.028ZINFOtest_disk_metrics (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29312025-04-30T06:26:27.028ZDEBGtest_disk_metrics (oximeter): creating ClickHouse client
29322025-04-30T06:26:27.028ZDEBGtest_disk_metrics (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
29332025-04-30T06:26:27.028ZDEBGtest_disk_metrics (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29342025-04-30T06:26:27.028ZDEBGtest_disk_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:39551
29352025-04-30T06:26:27.097ZDEBGtest_disk_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:39551/metrics/producers
29362025-04-30T06:26:27.099ZINFOtest_disk_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:39551 remote_addr = 127.0.0.1:60338
29372025-04-30T06:26:27.100ZTRACtest_disk_metrics (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: b0 ba 01 00 00 01 00 00 00 00 00 01 24 35 38 30 ............$580\n0010: 30 37 35 34 34 2d 64 62 39 35 2d 34 31 61 65 2d 07544-db95-41ae-\n0020: 39 63 38 36 2d 36 35 38 65 64 63 37 36 30 35 66 9c86-658edc7605f\n0030: 65 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 e.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:56268 req_id = 6a5be025-f61f-4b9a-9cd7-6f08fe72bfbf
29382025-04-30T06:26:27.100ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45242,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 53,\n 56,\n 48,\n 48,\n 55,\n 53,\n 52,\n 52,\n 45,\n 100,\n 98,\n 57,\n 53,\n 45,\n 52,\n 49,\n 97,\n 101,\n 45,\n 57,\n 99,\n 56,\n 54,\n 45,\n 54,\n 53,\n 56,\n 101,\n 100,\n 99,\n 55,\n 54,\n 48,\n 53,\n 102,\n 101,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56268 req_id = 6a5be025-f61f-4b9a-9cd7-6f08fe72bfbf
29392025-04-30T06:26:27.100ZDEBGtest_disk_metrics (store): query key key = 58007544-db95-41ae-9c86-658edc7605fe.host
29402025-04-30T06:26:27.102ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56268 query = LowerQuery { name: LowerName(Name("58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal.")), original: Query { name: Name("58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("58007544-db95-41ae-9c86-658edc7605fe.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6a5be025-f61f-4b9a-9cd7-6f08fe72bfbf
29412025-04-30T06:26:27.104ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:60338 req_id = 2babe779-6f1f-4547-9c56-3d7d47484832 uri = /metrics/producers
29422025-04-30T06:26:27.105ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "00981bfc-09a8-4ba2-8749-e0eba1781cc2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:27 GMT"} })
29432025-04-30T06:26:27.105ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
29442025-04-30T06:26:27.106ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:60338 req_id = 2babe779-6f1f-4547-9c56-3d7d47484832 resource = Database uri = /metrics/producers
29452025-04-30T06:26:27.106ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:60338 req_id = 2babe779-6f1f-4547-9c56-3d7d47484832 roles = RoleSet { roles: {} } uri = /metrics/producers
29462025-04-30T06:26:27.107ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:60338 req_id = 2babe779-6f1f-4547-9c56-3d7d47484832 resource = Database result = Ok(()) uri = /metrics/producers
29472025-04-30T06:26:27.191ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29482025-04-30T06:26:27.202ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:56672 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29492025-04-30T06:26:27.207ZWARNtest_disk_metrics (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
29502025-04-30T06:26:27.207ZDEBGtest_disk_metrics (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29512025-04-30T06:26:27.207ZINFOtest_disk_metrics (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
29522025-04-30T06:26:27.209ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:56672 req_id = 9fef9f4b-7952-48a9-a153-b5e643e25058 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29532025-04-30T06:26:27.211ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 2036 local_addr = [::1]:35636 method = GET remote_addr = [::1]:56672 req_id = 9fef9f4b-7952-48a9-a153-b5e643e25058 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29542025-04-30T06:26:27.211ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29552025-04-30T06:26:27.687ZINFOtest_disk_metrics (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 583123 local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:60338 req_id = 2babe779-6f1f-4547-9c56-3d7d47484832 response_code = 503 uri = /metrics/producers
29562025-04-30T06:26:27.688ZWARNtest_disk_metrics (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
29572025-04-30T06:26:27.688ZINFOtest_disk_metrics (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 version = 0
29582025-04-30T06:26:27.688ZINFOtest_disk_metrics (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
29592025-04-30T06:26:27.688ZDEBGtest_disk_metrics (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
29602025-04-30T06:26:27.689ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:38874 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29612025-04-30T06:26:27.689ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:59520 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29622025-04-30T06:26:27.689ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:44330 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29632025-04-30T06:26:27.689ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:57183 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29642025-04-30T06:26:27.689ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:52260 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29652025-04-30T06:26:27.689ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:64243 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29662025-04-30T06:26:27.689ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:41288 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29672025-04-30T06:26:27.690ZDEBGtest_disk_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "2babe779-6f1f-4547-9c56-3d7d47484832", "content-length": "133", "date": "Wed, 30 Apr 2025 06:26:27 GMT"} })
29682025-04-30T06:26:27.691ZWARNtest_disk_metrics (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 662.874415ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"2babe779-6f1f-4547-9c56-3d7d47484832\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:26:27 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"2babe779-6f1f-4547-9c56-3d7d47484832\\" }" file = oximeter/producer/src/lib.rs:371
29692025-04-30T06:26:27.692ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = CREATE DATABASE IF NOT EXISTS oximeter
29702025-04-30T06:26:27.693ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:38874 req_id = a8d4d8c3-7e16-4add-a572-11585ec6b801 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29712025-04-30T06:26:27.693ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:59520 req_id = 4618d0f5-17fb-470c-87dd-14daec7ad5f9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29722025-04-30T06:26:27.693ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:44330 req_id = 04b889d2-d604-494e-b067-38da2dc5847d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29732025-04-30T06:26:27.693ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:57183 req_id = 3d41228a-86c1-4ee0-b2a8-cb43cdda1a7c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29742025-04-30T06:26:27.694ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:52260 req_id = b72cd217-d465-4561-95de-da8daabed25f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29752025-04-30T06:26:27.694ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:64243 req_id = c5522dde-78f1-49ec-9ba5-8ef262997a89 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29762025-04-30T06:26:27.694ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:41288 req_id = 0f153771-1534-41b6-b1d7-d180fccd5e40 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29772025-04-30T06:26:27.696ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 3633 local_addr = [::1]:35636 method = GET remote_addr = [::1]:38874 req_id = a8d4d8c3-7e16-4add-a572-11585ec6b801 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29782025-04-30T06:26:27.696ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 3344 local_addr = [::1]:35636 method = GET remote_addr = [::1]:59520 req_id = 4618d0f5-17fb-470c-87dd-14daec7ad5f9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29792025-04-30T06:26:27.696ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 3121 local_addr = [::1]:35636 method = GET remote_addr = [::1]:44330 req_id = 04b889d2-d604-494e-b067-38da2dc5847d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29802025-04-30T06:26:27.696ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 3031 local_addr = [::1]:35636 method = GET remote_addr = [::1]:57183 req_id = 3d41228a-86c1-4ee0-b2a8-cb43cdda1a7c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29812025-04-30T06:26:27.697ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 3040 local_addr = [::1]:35636 method = GET remote_addr = [::1]:52260 req_id = b72cd217-d465-4561-95de-da8daabed25f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29822025-04-30T06:26:27.697ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 3008 local_addr = [::1]:35636 method = GET remote_addr = [::1]:64243 req_id = c5522dde-78f1-49ec-9ba5-8ef262997a89 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29832025-04-30T06:26:27.697ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 3256 local_addr = [::1]:35636 method = GET remote_addr = [::1]:41288 req_id = 0f153771-1534-41b6-b1d7-d180fccd5e40 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29842025-04-30T06:26:27.697ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29852025-04-30T06:26:27.698ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90eb55c1-6843-4a94-afb1-40dff3638de2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:27 GMT"} })
29862025-04-30T06:26:27.698ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
29872025-04-30T06:26:27.702ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29882025-04-30T06:26:27.707ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29892025-04-30T06:26:27.708ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce4ca5c3-5654-41bf-a69e-1ce453ae731a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:27 GMT"} })
29902025-04-30T06:26:27.708ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29912025-04-30T06:26:27.709ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29922025-04-30T06:26:27.709ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29932025-04-30T06:26:27.709ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29942025-04-30T06:26:27.710ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29952025-04-30T06:26:27.710ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29962025-04-30T06:26:27.718ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29972025-04-30T06:26:27.721ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29982025-04-30T06:26:27.725ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29992025-04-30T06:26:27.725ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
30002025-04-30T06:26:27.725ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
30012025-04-30T06:26:27.725ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30022025-04-30T06:26:27.725ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30032025-04-30T06:26:27.726ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30042025-04-30T06:26:27.727ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30052025-04-30T06:26:27.743ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30062025-04-30T06:26:27.747ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
30072025-04-30T06:26:27.747ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30082025-04-30T06:26:27.747ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30092025-04-30T06:26:27.748ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30102025-04-30T06:26:27.748ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30112025-04-30T06:26:27.753ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30122025-04-30T06:26:27.813ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
30132025-04-30T06:26:27.813ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
30142025-04-30T06:26:27.813ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30152025-04-30T06:26:27.814ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:49739 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30162025-04-30T06:26:27.816ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:49739 req_id = 37d9e04c-31f9-4ed3-812f-a859daecf6ef sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
30172025-04-30T06:26:27.817ZINFOtest_disk_metrics (dropshot): request completed latency_us = 550 local_addr = [::1]:50989 method = GET remote_addr = [::1]:49739 req_id = 37d9e04c-31f9-4ed3-812f-a859daecf6ef response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
30182025-04-30T06:26:27.817ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30192025-04-30T06:26:27.819ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "37d9e04c-31f9-4ed3-812f-a859daecf6ef", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:27 GMT"} })
30202025-04-30T06:26:27.821ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
30212025-04-30T06:26:27.821ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
30222025-04-30T06:26:27.821ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
30232025-04-30T06:26:27.823ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
30242025-04-30T06:26:27.823ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
30252025-04-30T06:26:27.823ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30262025-04-30T06:26:27.823ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30272025-04-30T06:26:27.824ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30282025-04-30T06:26:27.825ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30292025-04-30T06:26:27.827ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30302025-04-30T06:26:27.828ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30312025-04-30T06:26:27.835ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
30322025-04-30T06:26:27.835ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30332025-04-30T06:26:27.836ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30342025-04-30T06:26:27.836ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30352025-04-30T06:26:27.837ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30362025-04-30T06:26:27.843ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30372025-04-30T06:26:27.847ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30382025-04-30T06:26:27.863ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30392025-04-30T06:26:27.877ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30402025-04-30T06:26:27.883ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30412025-04-30T06:26:27.890ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30422025-04-30T06:26:27.899ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30432025-04-30T06:26:27.903ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30442025-04-30T06:26:27.908ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30452025-04-30T06:26:27.912ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30462025-04-30T06:26:27.916ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30472025-04-30T06:26:27.923ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30482025-04-30T06:26:27.929ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30492025-04-30T06:26:27.933ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30502025-04-30T06:26:27.937ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30512025-04-30T06:26:27.960ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30522025-04-30T06:26:27.962ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30532025-04-30T06:26:27.968ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30542025-04-30T06:26:27.968ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.937294993s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30552025-04-30T06:26:27.969ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30562025-04-30T06:26:27.970ZDEBGtest_disk_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:39551
30572025-04-30T06:26:28.030ZDEBGtest_disk_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:39551/metrics/producers
30582025-04-30T06:26:28.031ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30592025-04-30T06:26:28.031ZINFOtest_disk_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:39551 remote_addr = 127.0.0.1:47760
30602025-04-30T06:26:28.038ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:47760 req_id = 3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9 uri = /metrics/producers
30612025-04-30T06:26:28.038ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:47760 req_id = 3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9 resource = Database uri = /metrics/producers
30622025-04-30T06:26:28.039ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:47760 req_id = 3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9 roles = RoleSet { roles: {} } uri = /metrics/producers
30632025-04-30T06:26:28.039ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:47760 req_id = 3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9 resource = Database result = Ok(()) uri = /metrics/producers
30642025-04-30T06:26:28.049ZINFOtest_disk_metrics (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 11057 local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:47760 req_id = 3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9 response_code = 503 uri = /metrics/producers
30652025-04-30T06:26:28.050ZDEBGtest_disk_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9", "content-length": "133", "date": "Wed, 30 Apr 2025 06:26:27 GMT"} })
30662025-04-30T06:26:28.055ZWARNtest_disk_metrics (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 1.026305587s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:26:27 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"3d3c530e-cd3e-49f9-9b71-a0ef1d42f3b9\\" }" file = oximeter/producer/src/lib.rs:371
30672025-04-30T06:26:28.055ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30682025-04-30T06:26:28.062ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30692025-04-30T06:26:28.070ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30702025-04-30T06:26:28.087ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30712025-04-30T06:26:28.093ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30722025-04-30T06:26:28.097ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:26:28.104ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:26:28.112ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:26:28.128ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:26:28.143ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:26:28.164ZINFOtest_disk_metrics (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 version = 14
30782025-04-30T06:26:28.164ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:26:28.170ZDEBGtest_disk_metrics (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 0802d2b4-35f7-4479-bc2e-0db17fd5ecf2
30802025-04-30T06:26:28.171ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:35267 method = GET path = /info versions = all
30812025-04-30T06:26:28.171ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:35267 method = GET path = /producers versions = all
30822025-04-30T06:26:28.172ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:35267 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:26:28.172ZDEBGtest_disk_metrics (dropshot): registered endpoint local_addr = [::1]:35267 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:26:28.172ZINFOtest_disk_metrics (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:35267
30852025-04-30T06:26:28.173ZDEBGtest_disk_metrics (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:35267
30862025-04-30T06:26:28.174ZDEBGtest_disk_metrics (oximeter): registered USDT probes
30872025-04-30T06:26:28.174ZDEBGtest_disk_metrics (oximeter): contacting nexus
30882025-04-30T06:26:28.738ZDEBGtest_disk_metrics (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:39551/metrics/collectors
30892025-04-30T06:26:28.739ZDEBGtest_disk_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:39551
30902025-04-30T06:26:28.834ZDEBGtest_disk_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:39551/metrics/producers
30912025-04-30T06:26:28.835ZINFOtest_disk_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:39551 remote_addr = 127.0.0.1:49839
30922025-04-30T06:26:28.835ZINFOtest_disk_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:39551 remote_addr = 127.0.0.1:62234
30932025-04-30T06:26:28.836ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:62234 req_id = 5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2 uri = /metrics/producers
30942025-04-30T06:26:28.836ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:62234 req_id = 5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2 resource = Database uri = /metrics/producers
30952025-04-30T06:26:28.836ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:62234 req_id = 5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2 roles = RoleSet { roles: {} } uri = /metrics/producers
30962025-04-30T06:26:28.838ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:62234 req_id = 5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2 resource = Database result = Ok(()) uri = /metrics/producers
30972025-04-30T06:26:28.838ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:49839 req_id = 8bb8aa13-476f-413a-9e75-13666dfb9e4b uri = /metrics/collectors
30982025-04-30T06:26:28.839ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:49839 req_id = 8bb8aa13-476f-413a-9e75-13666dfb9e4b resource = Database uri = /metrics/collectors
30992025-04-30T06:26:28.839ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:49839 req_id = 8bb8aa13-476f-413a-9e75-13666dfb9e4b roles = RoleSet { roles: {} } uri = /metrics/collectors
31002025-04-30T06:26:28.839ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:49839 req_id = 8bb8aa13-476f-413a-9e75-13666dfb9e4b resource = Database result = Ok(()) uri = /metrics/collectors
31012025-04-30T06:26:28.845ZINFOtest_disk_metrics (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 9540 local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:62234 req_id = 5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2 response_code = 503 uri = /metrics/producers
31022025-04-30T06:26:28.845ZINFOtest_disk_metrics (nexus): registered new oximeter metric collection server address = [::1]:35267 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31032025-04-30T06:26:28.846ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 7386 local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:49839 req_id = 8bb8aa13-476f-413a-9e75-13666dfb9e4b response_code = 204 uri = /metrics/collectors
31042025-04-30T06:26:28.847ZDEBGtest_disk_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2", "content-length": "133", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
31052025-04-30T06:26:28.848ZWARNtest_disk_metrics (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 1.819871205s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:26:28 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"5fdb8a73-8d8a-4429-a1ca-2b7fb588f5b2\\" }" file = oximeter/producer/src/lib.rs:371
31062025-04-30T06:26:28.848ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors", status: 204, headers: {"x-request-id": "8bb8aa13-476f-413a-9e75-13666dfb9e4b", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
31072025-04-30T06:26:28.849ZINFOtest_disk_metrics (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31082025-04-30T06:26:28.849ZDEBGtest_disk_metrics (init_with_steps): Running step start_producer_server
31092025-04-30T06:26:28.849ZDEBGtest_disk_metrics: Starting test metric Producer Server
31102025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31112025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31122025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31132025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31142025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31152025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31162025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31172025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31182025-04-30T06:26:28.854ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:40156 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31192025-04-30T06:26:28.854ZINFOtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:40156 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31202025-04-30T06:26:28.855ZDEBGtest_disk_metrics (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:40156 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31212025-04-30T06:26:28.855ZINFOtest_disk_metrics (SledAgent): Created Simulated Crucible Server address = [::1]:40156 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31222025-04-30T06:26:28.856ZINFOtest_disk_metrics (nexus): inserting test physical disk model = test-model physical_disk_id = 9c4cf8c9-a47a-40fe-a860-23edc2d436ac serial = totally-unique-serial: 9c4cf8c9-a47a-40fe-a860-23edc2d436ac sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31232025-04-30T06:26:28.856ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31242025-04-30T06:26:28.856ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31252025-04-30T06:26:28.856ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31262025-04-30T06:26:28.857ZINFOtest_disk_metrics (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31272025-04-30T06:26:28.906ZINFOtest_disk_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:39551 remote_addr = 127.0.0.1:36001
31282025-04-30T06:26:28.906ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31292025-04-30T06:26:28.907ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31302025-04-30T06:26:28.907ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 7bc9bc73-9cc1-4acb-a37a-110ec936dc9d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31312025-04-30T06:26:28.907ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:36001 req_id = 4ca2fa09-e413-4a87-91d2-3e1cf593c5e7 uri = /metrics/producers
31322025-04-30T06:26:28.908ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 7bc9bc73-9cc1-4acb-a37a-110ec936dc9d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31332025-04-30T06:26:28.908ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 7bc9bc73-9cc1-4acb-a37a-110ec936dc9d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31342025-04-30T06:26:28.909ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 7bc9bc73-9cc1-4acb-a37a-110ec936dc9d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31352025-04-30T06:26:28.909ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:36001 req_id = 4ca2fa09-e413-4a87-91d2-3e1cf593c5e7 resource = Database uri = /metrics/producers
31362025-04-30T06:26:28.909ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:36001 req_id = 4ca2fa09-e413-4a87-91d2-3e1cf593c5e7 roles = RoleSet { roles: {} } uri = /metrics/producers
31372025-04-30T06:26:28.910ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:36001 req_id = 4ca2fa09-e413-4a87-91d2-3e1cf593c5e7 resource = Database result = Ok(()) uri = /metrics/producers
31382025-04-30T06:26:28.911ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31392025-04-30T06:26:28.911ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31402025-04-30T06:26:28.911ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31412025-04-30T06:26:28.918ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 10981 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 7bc9bc73-9cc1-4acb-a37a-110ec936dc9d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31422025-04-30T06:26:28.919ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31432025-04-30T06:26:28.919ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31442025-04-30T06:26:28.919ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31452025-04-30T06:26:28.919ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31462025-04-30T06:26:28.919ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31472025-04-30T06:26:28.920ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc9bc73-9cc1-4acb-a37a-110ec936dc9d", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
31482025-04-30T06:26:28.920ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
31492025-04-30T06:26:28.926ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31502025-04-30T06:26:28.932ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31512025-04-30T06:26:28.970ZINFOtest_disk_metrics (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31522025-04-30T06:26:28.988ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 80332 local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:36001 req_id = 4ca2fa09-e413-4a87-91d2-3e1cf593c5e7 response_code = 201 uri = /metrics/producers
31532025-04-30T06:26:29.091ZINFOtest_disk_metrics (nexus): upserting zpool physical_disk_id = 9c4cf8c9-a47a-40fe-a860-23edc2d436ac sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 86bfe929-f0c7-453c-94b4-ad5d02ae2fd0
31542025-04-30T06:26:29.091ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31552025-04-30T06:26:29.091ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31562025-04-30T06:26:29.092ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31572025-04-30T06:26:29.093ZTRACtest_disk_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31582025-04-30T06:26:29.093ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31592025-04-30T06:26:29.093ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31602025-04-30T06:26:29.093ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31612025-04-30T06:26:29.094ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31622025-04-30T06:26:29.103ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31632025-04-30T06:26:29.109ZDEBGtest_disk_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31642025-04-30T06:26:29.121ZINFOtest_disk_metrics (nexus): upserting Crucible dataset dataset_id = 46de1d86-484a-4ae5-b5f7-cdb827f6c475 zpool_id = 86bfe929-f0c7-453c-94b4-ad5d02ae2fd0
31652025-04-30T06:26:29.130ZINFOtest_disk_metrics (nexus): inserting test physical disk model = test-model physical_disk_id = 9c4cf8c9-a47a-40fe-a860-23edc2d436ac serial = totally-unique-serial: 9c4cf8c9-a47a-40fe-a860-23edc2d436ac sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31662025-04-30T06:26:29.131ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31672025-04-30T06:26:29.131ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31682025-04-30T06:26:29.132ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31692025-04-30T06:26:29.137ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 9c4cf8c9-a47a-40fe-a860-23edc2d436ac (physical_disk), lookup_type: ById(9c4cf8c9-a47a-40fe-a860-23edc2d436ac) }
31702025-04-30T06:26:29.137ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31712025-04-30T06:26:29.137ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31722025-04-30T06:26:29.137ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31732025-04-30T06:26:29.138ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31742025-04-30T06:26:29.142ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31752025-04-30T06:26:29.152ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 9c4cf8c9-a47a-40fe-a860-23edc2d436ac (physical_disk), lookup_type: ById(9c4cf8c9-a47a-40fe-a860-23edc2d436ac) } result = Ok(())
31762025-04-30T06:26:29.152ZINFOtest_disk_metrics (nexus): upserting zpool physical_disk_id = 9c4cf8c9-a47a-40fe-a860-23edc2d436ac sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 86bfe929-f0c7-453c-94b4-ad5d02ae2fd0
31772025-04-30T06:26:29.152ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31782025-04-30T06:26:29.152ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31792025-04-30T06:26:29.153ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31802025-04-30T06:26:29.163ZTRACtest_disk_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31812025-04-30T06:26:29.163ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31822025-04-30T06:26:29.164ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31832025-04-30T06:26:29.164ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31842025-04-30T06:26:29.166ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31852025-04-30T06:26:29.171ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31862025-04-30T06:26:29.175ZDEBGtest_disk_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31872025-04-30T06:26:29.175ZTRACtest_disk_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31882025-04-30T06:26:29.175ZTRACtest_disk_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31892025-04-30T06:26:29.183ZTRACtest_disk_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31902025-04-30T06:26:29.183ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31912025-04-30T06:26:29.183ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31922025-04-30T06:26:29.183ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31932025-04-30T06:26:29.184ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31942025-04-30T06:26:29.189ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31952025-04-30T06:26:29.198ZDEBGtest_disk_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31962025-04-30T06:26:29.198ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31972025-04-30T06:26:29.199ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
31982025-04-30T06:26:29.199ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31992025-04-30T06:26:29.213ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32002025-04-30T06:26:29.213ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32012025-04-30T06:26:29.213ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32022025-04-30T06:26:29.213ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
32032025-04-30T06:26:29.215ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32042025-04-30T06:26:29.215ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32052025-04-30T06:26:29.215ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32062025-04-30T06:26:29.215ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32072025-04-30T06:26:29.215ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32082025-04-30T06:26:29.215ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32092025-04-30T06:26:29.215ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32102025-04-30T06:26:29.219ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32112025-04-30T06:26:29.226ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32122025-04-30T06:26:29.226ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32132025-04-30T06:26:29.226ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
32142025-04-30T06:26:29.227ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32152025-04-30T06:26:29.228ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32162025-04-30T06:26:29.231ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32172025-04-30T06:26:29.232ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32182025-04-30T06:26:29.232ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32192025-04-30T06:26:29.232ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32202025-04-30T06:26:29.235ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32212025-04-30T06:26:29.235ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
32222025-04-30T06:26:29.236ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32232025-04-30T06:26:29.239ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
32242025-04-30T06:26:29.239ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32252025-04-30T06:26:29.240ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0a ab 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51482 req_id = b7b31d07-13e6-46aa-ae50-9ccf131f7d0b
32262025-04-30T06:26:29.240ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2731,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51482 req_id = b7b31d07-13e6-46aa-ae50-9ccf131f7d0b
32272025-04-30T06:26:29.241ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
32282025-04-30T06:26:29.242ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32292025-04-30T06:26:29.242ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51482 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b7b31d07-13e6-46aa-ae50-9ccf131f7d0b
32302025-04-30T06:26:29.242ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3254, tv_nsec: 41714707 } })
32312025-04-30T06:26:29.242ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
32322025-04-30T06:26:29.243ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 10 0a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55811 req_id = b0074e78-1b8c-44e5-b208-358da6993b2a
32332025-04-30T06:26:29.244ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4106,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55811 req_id = b0074e78-1b8c-44e5-b208-358da6993b2a
32342025-04-30T06:26:29.244ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32352025-04-30T06:26:29.244ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55811 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b0074e78-1b8c-44e5-b208-358da6993b2a
32362025-04-30T06:26:29.309ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32372025-04-30T06:26:29.310ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0b 5b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .[..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63959 req_id = 37c51caf-adcf-4705-b923-cd91a6670510
32382025-04-30T06:26:29.310ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2907,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63959 req_id = 37c51caf-adcf-4705-b923-cd91a6670510
32392025-04-30T06:26:29.311ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
32402025-04-30T06:26:29.311ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2907, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63959 req_id = 37c51caf-adcf-4705-b923-cd91a6670510
32412025-04-30T06:26:29.311ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
32422025-04-30T06:26:29.311ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
32432025-04-30T06:26:29.311ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
32442025-04-30T06:26:29.320ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2f614ef-59f1-4e5a-9dab-cae6955a2faf", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32452025-04-30T06:26:29.320ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
32462025-04-30T06:26:29.324ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b53ce05-bdb7-4774-8f35-31d4274b9364", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32472025-04-30T06:26:29.325ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
32482025-04-30T06:26:29.331ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ade1cb2-2412-4ee1-8e98-28aac2870210", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32492025-04-30T06:26:29.331ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
32502025-04-30T06:26:29.335ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12fc6133-dd83-4299-ba39-ffc92467147d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32512025-04-30T06:26:29.335ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
32522025-04-30T06:26:29.337ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da5bffe1-2c62-47e3-a031-fff6dcddb11e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32532025-04-30T06:26:29.337ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
32542025-04-30T06:26:29.340ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c5e4aba-8ea4-456e-8bfc-27561758b0ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32552025-04-30T06:26:29.340ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
32562025-04-30T06:26:29.342ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd04bd9f-a6f3-44b8-8c8d-2e21aabbc134", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32572025-04-30T06:26:29.342ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
32582025-04-30T06:26:29.344ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54eee19b-c0cd-486f-bee2-38f9015a39bb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32592025-04-30T06:26:29.344ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
32602025-04-30T06:26:29.349ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1cf696c-e847-41b8-b5b5-e4e4edb5fcfb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32612025-04-30T06:26:29.349ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
32622025-04-30T06:26:29.350ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dc55c1a-ca6e-43e2-b5a6-44dca34493bc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32632025-04-30T06:26:29.350ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
32642025-04-30T06:26:29.356ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "603ce832-bdb2-4b5e-9b56-374b18a0c8de", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32652025-04-30T06:26:29.356ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
32662025-04-30T06:26:29.357ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "437f7bfe-7e37-4bfb-8b73-6dc4a492d5ac", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32672025-04-30T06:26:29.357ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
32682025-04-30T06:26:29.358ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd1297ce-72f3-46cb-9581-0faf6dd392fd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32692025-04-30T06:26:29.358ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
32702025-04-30T06:26:29.362ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b8896b0-05f3-4cf0-8839-5268e7d4fbbc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32712025-04-30T06:26:29.363ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
32722025-04-30T06:26:29.364ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e23f4abb-21e5-462a-8ccb-9fec14fb86b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32732025-04-30T06:26:29.364ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
32742025-04-30T06:26:29.372ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad43e644-ab01-4bc0-8c7f-91a252c70a06", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32752025-04-30T06:26:29.372ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
32762025-04-30T06:26:29.374ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7372b993-0766-4947-97f4-0686144f44ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32772025-04-30T06:26:29.374ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
32782025-04-30T06:26:29.379ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5285110f-d54d-4842-92b2-36386b5b2be5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32792025-04-30T06:26:29.379ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
32802025-04-30T06:26:29.384ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58634a5b-a51d-403f-a68d-cf48e6b56bad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32812025-04-30T06:26:29.384ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
32822025-04-30T06:26:29.385ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "414fab9d-3f56-4f19-aef8-636f3f4056dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32832025-04-30T06:26:29.385ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
32842025-04-30T06:26:29.386ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cd852c0-b5db-48d0-99ec-8b76b533e584", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32852025-04-30T06:26:29.386ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
32862025-04-30T06:26:29.387ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce062e48-4470-409c-9607-8e9c1f1d09da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32872025-04-30T06:26:29.388ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
32882025-04-30T06:26:29.397ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd6b636b-3ede-4b9a-908d-3515f86076e0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32892025-04-30T06:26:29.397ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
32902025-04-30T06:26:29.399ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
32912025-04-30T06:26:29.401ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f365772-64b2-4caa-8bb9-577e57e82241", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32922025-04-30T06:26:29.401ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
32932025-04-30T06:26:29.403ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c58207ef-ce4f-47b9-9625-e001060da733", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32942025-04-30T06:26:29.403ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
32952025-04-30T06:26:29.406ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27580631-25a1-4872-8228-0fa1716c7085", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32962025-04-30T06:26:29.407ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
32972025-04-30T06:26:29.409ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c453e84f-1b84-4455-8642-c141f84a7271", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
32982025-04-30T06:26:29.410ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
32992025-04-30T06:26:29.424ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "461b57e5-be31-465b-babb-ff71b5eddac9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33002025-04-30T06:26:29.424ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
33012025-04-30T06:26:29.424ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2560c0a-c915-43f8-9ea1-fa7bbd508b7c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33022025-04-30T06:26:29.424ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
33032025-04-30T06:26:29.431ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4617498d-52cb-4000-b040-a7092806b333", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33042025-04-30T06:26:29.432ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
33052025-04-30T06:26:29.435ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f16e282a-e7e8-48a3-ae8f-ba5986f644ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33062025-04-30T06:26:29.436ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
33072025-04-30T06:26:29.439ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "657e47dd-8671-40b3-9ce3-a4ebf8b32895", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33082025-04-30T06:26:29.440ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
33092025-04-30T06:26:29.441ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02a7cfa4-f5e4-420a-b54f-43af74eaeffa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33102025-04-30T06:26:29.442ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
33112025-04-30T06:26:29.444ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10ef9bee-ce36-4e3e-ad05-fe02e94d5a22", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33122025-04-30T06:26:29.444ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
33132025-04-30T06:26:29.446ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52225072-63f2-44ea-88f4-26dad23cf982", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33142025-04-30T06:26:29.447ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
33152025-04-30T06:26:29.449ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f30a8859-5d76-4286-9948-3e27b11e628c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33162025-04-30T06:26:29.449ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
33172025-04-30T06:26:29.450ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33182025-04-30T06:26:29.450ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33192025-04-30T06:26:29.450ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33202025-04-30T06:26:29.450ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33212025-04-30T06:26:29.451ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33222025-04-30T06:26:29.452ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4078b5ee-ec3f-46b0-9943-a354320d5a7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33232025-04-30T06:26:29.453ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
33242025-04-30T06:26:29.454ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33252025-04-30T06:26:29.462ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33262025-04-30T06:26:29.462ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33272025-04-30T06:26:29.462ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33282025-04-30T06:26:29.463ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33292025-04-30T06:26:29.465ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33302025-04-30T06:26:29.465ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33312025-04-30T06:26:29.466ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33322025-04-30T06:26:29.466ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b0fe0f2-b4b4-4ee3-b0a4-1eb59bbc43b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33332025-04-30T06:26:29.466ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
33342025-04-30T06:26:29.469ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e36ea57c-5eef-49f1-b139-773c78299233", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33352025-04-30T06:26:29.469ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
33362025-04-30T06:26:29.471ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c37bc683-62f2-4d7e-8c79-52fe495f0787", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33372025-04-30T06:26:29.473ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
33382025-04-30T06:26:29.474ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84073947-ae23-44e9-a48a-1a6389ab4888", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33392025-04-30T06:26:29.475ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
33402025-04-30T06:26:29.476ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f9ecb74-7cdd-45c1-a8b7-d6bdb28cb3ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33412025-04-30T06:26:29.476ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
33422025-04-30T06:26:29.477ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b6425de-1857-4a79-a5ec-524a5ce256c6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33432025-04-30T06:26:29.477ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
33442025-04-30T06:26:29.478ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a37866f-254a-4e9c-bb80-c4d441bf6ff4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33452025-04-30T06:26:29.478ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
33462025-04-30T06:26:29.479ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ade8fbe3-71e9-4766-bcc5-622281367294", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33472025-04-30T06:26:29.479ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33482025-04-30T06:26:29.479ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33492025-04-30T06:26:29.479ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33502025-04-30T06:26:29.479ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33512025-04-30T06:26:29.480ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33522025-04-30T06:26:29.488ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33532025-04-30T06:26:29.503ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33542025-04-30T06:26:29.503ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33552025-04-30T06:26:29.503ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33562025-04-30T06:26:29.503ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33572025-04-30T06:26:29.503ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33582025-04-30T06:26:29.514ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33592025-04-30T06:26:29.531ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33602025-04-30T06:26:29.540ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33612025-04-30T06:26:29.540ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33622025-04-30T06:26:29.541ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33632025-04-30T06:26:29.543ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33642025-04-30T06:26:29.619ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
33652025-04-30T06:26:29.619ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
33662025-04-30T06:26:29.620ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:43329 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33672025-04-30T06:26:29.621ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:43329 req_id = 069ef26a-5c3f-40e9-b191-0f968925fca7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33682025-04-30T06:26:29.622ZINFOtest_disk_metrics (dropshot): request completed latency_us = 758 local_addr = [::1]:50989 method = GET remote_addr = [::1]:43329 req_id = 069ef26a-5c3f-40e9-b191-0f968925fca7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33692025-04-30T06:26:29.623ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "069ef26a-5c3f-40e9-b191-0f968925fca7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
33702025-04-30T06:26:29.625ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33712025-04-30T06:26:29.625ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33722025-04-30T06:26:29.625ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
33732025-04-30T06:26:29.627ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
33742025-04-30T06:26:29.627ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33752025-04-30T06:26:29.627ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33762025-04-30T06:26:29.627ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33772025-04-30T06:26:29.628ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33782025-04-30T06:26:29.646ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
33792025-04-30T06:26:29.648ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33802025-04-30T06:26:29.651ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
33812025-04-30T06:26:29.652ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33822025-04-30T06:26:29.652ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33832025-04-30T06:26:29.653ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33842025-04-30T06:26:29.690ZDEBGtest_disk_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:39551
33852025-04-30T06:26:29.777ZDEBGtest_disk_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:39551/metrics/producers
33862025-04-30T06:26:29.777ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33872025-04-30T06:26:29.777ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33882025-04-30T06:26:29.778ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33892025-04-30T06:26:29.778ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33902025-04-30T06:26:29.778ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33912025-04-30T06:26:29.778ZINFOtest_disk_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:39551 remote_addr = 127.0.0.1:55071
33922025-04-30T06:26:29.779ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:55071 req_id = 3eb8c1cf-23c8-47de-bdfe-1adc4e2b3180 uri = /metrics/producers
33932025-04-30T06:26:29.780ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:55071 req_id = 3eb8c1cf-23c8-47de-bdfe-1adc4e2b3180 resource = Database uri = /metrics/producers
33942025-04-30T06:26:29.780ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:55071 req_id = 3eb8c1cf-23c8-47de-bdfe-1adc4e2b3180 roles = RoleSet { roles: {} } uri = /metrics/producers
33952025-04-30T06:26:29.780ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:55071 req_id = 3eb8c1cf-23c8-47de-bdfe-1adc4e2b3180 resource = Database result = Ok(()) uri = /metrics/producers
33962025-04-30T06:26:29.782ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33972025-04-30T06:26:29.788ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33982025-04-30T06:26:29.788ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33992025-04-30T06:26:29.788ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
34002025-04-30T06:26:29.789ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34012025-04-30T06:26:29.834ZINFOtest_disk_metrics (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
34022025-04-30T06:26:29.834ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 54961 local_addr = 127.0.0.1:39551 method = POST remote_addr = 127.0.0.1:55071 req_id = 3eb8c1cf-23c8-47de-bdfe-1adc4e2b3180 response_code = 201 uri = /metrics/producers
34032025-04-30T06:26:29.835ZDEBGtest_disk_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "3eb8c1cf-23c8-47de-bdfe-1adc4e2b3180", "content-length": "41", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
34042025-04-30T06:26:29.838ZDEBGtest_disk_metrics (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
34052025-04-30T06:26:29.838ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34062025-04-30T06:26:29.839ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34072025-04-30T06:26:29.839ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 623.981443ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34082025-04-30T06:26:29.839ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34092025-04-30T06:26:29.839ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34102025-04-30T06:26:29.839ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34112025-04-30T06:26:29.839ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34122025-04-30T06:26:29.839ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34132025-04-30T06:26:29.840ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34142025-04-30T06:26:29.840ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34152025-04-30T06:26:29.841ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
34162025-04-30T06:26:29.842ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34172025-04-30T06:26:29.847ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34182025-04-30T06:26:29.853ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34192025-04-30T06:26:29.853ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34202025-04-30T06:26:29.853ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34212025-04-30T06:26:29.854ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34222025-04-30T06:26:29.864ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
34232025-04-30T06:26:29.864ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34242025-04-30T06:26:29.865ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fc d6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65372 req_id = f522949a-7504-4942-8fdb-86ad53be29dc
34252025-04-30T06:26:29.865ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64726,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65372 req_id = f522949a-7504-4942-8fdb-86ad53be29dc
34262025-04-30T06:26:29.866ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
34272025-04-30T06:26:29.866ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34282025-04-30T06:26:29.866ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65372 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f522949a-7504-4942-8fdb-86ad53be29dc
34292025-04-30T06:26:29.870ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3254, tv_nsec: 669240962 } })
34302025-04-30T06:26:29.870ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
34312025-04-30T06:26:29.870ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 03 95 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55226 req_id = 980b6f9e-6d29-49ef-9579-b0753d21e86c
34322025-04-30T06:26:29.870ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 917,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55226 req_id = 980b6f9e-6d29-49ef-9579-b0753d21e86c
34332025-04-30T06:26:29.870ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34342025-04-30T06:26:29.871ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55226 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 980b6f9e-6d29-49ef-9579-b0753d21e86c
34352025-04-30T06:26:29.939ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34362025-04-30T06:26:29.940ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 91 b9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56558 req_id = b14c5878-2099-4d7d-84fc-382427608e2c
34372025-04-30T06:26:29.940ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37305,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56558 req_id = b14c5878-2099-4d7d-84fc-382427608e2c
34382025-04-30T06:26:29.941ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
34392025-04-30T06:26:29.941ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37305, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56558 req_id = b14c5878-2099-4d7d-84fc-382427608e2c
34402025-04-30T06:26:29.942ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
34412025-04-30T06:26:29.942ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
34422025-04-30T06:26:29.942ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
34432025-04-30T06:26:29.969ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "166e7b72-83b4-49b3-880b-c6a7e3e4b991", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34442025-04-30T06:26:29.970ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
34452025-04-30T06:26:29.986ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "856b555f-7051-4218-aff7-3c79632136ee", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34462025-04-30T06:26:29.987ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
34472025-04-30T06:26:30.001ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40201260-3454-48c5-9c2c-bb09ee471660", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34482025-04-30T06:26:30.001ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
34492025-04-30T06:26:30.017ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c766b65d-b2a5-4800-b9fe-31fead0b68b0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34502025-04-30T06:26:30.017ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
34512025-04-30T06:26:30.021ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f78314f-3bf5-4a08-8b0f-e2871255d57b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34522025-04-30T06:26:30.024ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
34532025-04-30T06:26:30.024ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
34542025-04-30T06:26:30.029ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec3358e-db5f-4dc4-bb3e-0f4302d85f0c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34552025-04-30T06:26:30.029ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
34562025-04-30T06:26:30.032ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00d86242-4e37-4faf-8814-895da30578d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34572025-04-30T06:26:30.032ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
34582025-04-30T06:26:30.034ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ad16809-2c64-432c-98ea-419db69a3716", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34592025-04-30T06:26:30.034ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
34602025-04-30T06:26:30.035ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4022fca-1751-4998-a1c6-6156a8abd35f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34612025-04-30T06:26:30.035ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
34622025-04-30T06:26:30.036ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4f7b182-e78d-4c99-9472-2abd1c0b0274", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34632025-04-30T06:26:30.037ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
34642025-04-30T06:26:30.038ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7545c99-541a-4b31-9670-e38bfbc01e2b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34652025-04-30T06:26:30.038ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
34662025-04-30T06:26:30.039ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "568305a1-9c3a-4103-955d-2bf0936ce8d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34672025-04-30T06:26:30.039ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
34682025-04-30T06:26:30.040ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afadc238-58fb-45ba-ad48-c6d7b0ef3eb2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34692025-04-30T06:26:30.041ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
34702025-04-30T06:26:30.045ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c585a04a-8139-4d14-acbf-2a4b7d6eb935", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34712025-04-30T06:26:30.045ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
34722025-04-30T06:26:30.049ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f99b824d-7688-435f-b64a-890ef3e1751e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34732025-04-30T06:26:30.050ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
34742025-04-30T06:26:30.052ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b23e893-9f31-4073-b044-c41fb50097e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34752025-04-30T06:26:30.052ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
34762025-04-30T06:26:30.053ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d506a66-adfb-46ab-9ad5-560633269bf2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34772025-04-30T06:26:30.054ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
34782025-04-30T06:26:30.055ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e30c0f97-d9c0-4e84-9e5f-133f20fed122", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34792025-04-30T06:26:30.056ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
34802025-04-30T06:26:30.057ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e0f6d53-eecd-4304-ba60-85f4dbb26ee8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34812025-04-30T06:26:30.058ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
34822025-04-30T06:26:30.058ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a316e2b-4560-45c5-852d-1bb4fc768267", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34832025-04-30T06:26:30.059ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
34842025-04-30T06:26:30.060ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e474bb-bcae-44f1-a95c-09392f3d283f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34852025-04-30T06:26:30.060ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
34862025-04-30T06:26:30.061ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a12b508b-f93c-4520-94fb-ccf8838f0327", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34872025-04-30T06:26:30.061ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
34882025-04-30T06:26:30.062ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce892cdb-92a5-4a59-92cb-2d38ce4ba54a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34892025-04-30T06:26:30.062ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
34902025-04-30T06:26:30.063ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff96494c-d6b4-414c-9f94-f57d6d586d56", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34912025-04-30T06:26:30.064ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
34922025-04-30T06:26:30.066ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e78b28d9-4031-4cf7-ade2-9068f739e863", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34932025-04-30T06:26:30.066ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
34942025-04-30T06:26:30.068ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bacf95a4-49e2-4990-bc68-66c548d0eaf9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34952025-04-30T06:26:30.068ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
34962025-04-30T06:26:30.071ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d47f5d4a-9891-4b0c-9f92-763cfffa4130", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34972025-04-30T06:26:30.071ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
34982025-04-30T06:26:30.073ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d95aeee5-1c6a-40d0-b060-fe510dbf52db", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
34992025-04-30T06:26:30.073ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
35002025-04-30T06:26:30.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb2a0079-08f2-4826-a448-da8ff82c0df3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35012025-04-30T06:26:30.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
35022025-04-30T06:26:30.075ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35032025-04-30T06:26:30.075ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35042025-04-30T06:26:30.075ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35052025-04-30T06:26:30.075ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35062025-04-30T06:26:30.076ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35072025-04-30T06:26:30.079ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35082025-04-30T06:26:30.089ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35092025-04-30T06:26:30.089ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35102025-04-30T06:26:30.089ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35112025-04-30T06:26:30.089ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35122025-04-30T06:26:30.091ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab873fd-e9a7-4593-9689-2a0835b90f06", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35132025-04-30T06:26:30.091ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
35142025-04-30T06:26:30.092ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35152025-04-30T06:26:30.092ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35162025-04-30T06:26:30.092ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35172025-04-30T06:26:30.093ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6978ce5-93b1-40e5-8b77-aa2ad695bfbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35182025-04-30T06:26:30.093ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
35192025-04-30T06:26:30.094ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13c9322e-eb18-4bac-b802-9a83ade54ebf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35202025-04-30T06:26:30.094ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
35212025-04-30T06:26:30.095ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6f97ca1-cc76-4295-a4aa-de5e144fac4c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35222025-04-30T06:26:30.095ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
35232025-04-30T06:26:30.097ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a47190-ca47-40a7-a7f1-17fc523d1466", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35242025-04-30T06:26:30.097ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
35252025-04-30T06:26:30.099ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "431d1b78-0770-43fd-9753-c9e19ca67bd7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35262025-04-30T06:26:30.099ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
35272025-04-30T06:26:30.101ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "586cc316-006d-4cee-aa81-eed9116f85c7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35282025-04-30T06:26:30.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
35292025-04-30T06:26:30.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4ab1b4a-830b-43ca-ad7b-bcb52c3fca76", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35302025-04-30T06:26:30.108ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
35312025-04-30T06:26:30.110ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85b57b0b-175b-4f56-ab9a-f3c64a34fea4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35322025-04-30T06:26:30.110ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
35332025-04-30T06:26:30.116ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9609adc-d543-4f9c-a8d7-566d954de0ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35342025-04-30T06:26:30.117ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
35352025-04-30T06:26:30.120ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd755e72-228b-4afb-ad62-9e60e538fb99", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35362025-04-30T06:26:30.120ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
35372025-04-30T06:26:30.134ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
35382025-04-30T06:26:30.137ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfdd5991-7bbe-4dde-a6b9-7bc09f94db4b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35392025-04-30T06:26:30.137ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
35402025-04-30T06:26:30.139ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbc58ea5-5b12-43fb-bc88-1067a274cd5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35412025-04-30T06:26:30.139ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
35422025-04-30T06:26:30.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f9245e3-f7ed-4e4b-8a16-0a9471142a53", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35432025-04-30T06:26:30.141ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
35442025-04-30T06:26:30.148ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd568de-27c2-4e47-9856-f830274d6da5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35452025-04-30T06:26:30.148ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
35462025-04-30T06:26:30.150ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ca64f79-f20e-4224-9a98-602a0857de52", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35472025-04-30T06:26:30.150ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35482025-04-30T06:26:30.150ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35492025-04-30T06:26:30.150ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35502025-04-30T06:26:30.150ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35512025-04-30T06:26:30.151ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35522025-04-30T06:26:30.156ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35532025-04-30T06:26:30.164ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35542025-04-30T06:26:30.164ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35552025-04-30T06:26:30.164ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35562025-04-30T06:26:30.164ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35572025-04-30T06:26:30.164ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35582025-04-30T06:26:30.165ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35592025-04-30T06:26:30.176ZTRACtest_disk_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35602025-04-30T06:26:30.176ZTRACtest_disk_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35612025-04-30T06:26:30.178ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35622025-04-30T06:26:30.186ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35632025-04-30T06:26:30.186ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35642025-04-30T06:26:30.186ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35652025-04-30T06:26:30.187ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35662025-04-30T06:26:30.188ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35672025-04-30T06:26:30.188ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35682025-04-30T06:26:30.188ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35692025-04-30T06:26:30.188ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35702025-04-30T06:26:30.189ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35712025-04-30T06:26:30.193ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35722025-04-30T06:26:30.200ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35732025-04-30T06:26:30.200ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35742025-04-30T06:26:30.200ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35752025-04-30T06:26:30.200ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35762025-04-30T06:26:30.244ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
35772025-04-30T06:26:30.244ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
35782025-04-30T06:26:30.245ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:56673 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35792025-04-30T06:26:30.245ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:56673 req_id = 217e53c6-5a6b-4504-ba0b-ee1baf67806a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35802025-04-30T06:26:30.246ZINFOtest_disk_metrics (dropshot): request completed latency_us = 557 local_addr = [::1]:50989 method = GET remote_addr = [::1]:56673 req_id = 217e53c6-5a6b-4504-ba0b-ee1baf67806a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35812025-04-30T06:26:30.246ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35822025-04-30T06:26:30.246ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35832025-04-30T06:26:30.247ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35842025-04-30T06:26:30.247ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "217e53c6-5a6b-4504-ba0b-ee1baf67806a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
35852025-04-30T06:26:30.249ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35862025-04-30T06:26:30.249ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35872025-04-30T06:26:30.249ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
35882025-04-30T06:26:30.251ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
35892025-04-30T06:26:30.251ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35902025-04-30T06:26:30.251ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35912025-04-30T06:26:30.251ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35922025-04-30T06:26:30.251ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35932025-04-30T06:26:30.254ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35942025-04-30T06:26:30.258ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
35952025-04-30T06:26:30.259ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35962025-04-30T06:26:30.259ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35972025-04-30T06:26:30.259ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35982025-04-30T06:26:30.298ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
35992025-04-30T06:26:30.299ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36002025-04-30T06:26:30.299ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 460.301952ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36012025-04-30T06:26:30.299ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36022025-04-30T06:26:30.299ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36032025-04-30T06:26:30.299ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36042025-04-30T06:26:30.299ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36052025-04-30T06:26:30.299ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36062025-04-30T06:26:30.300ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36072025-04-30T06:26:30.303ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
36082025-04-30T06:26:30.304ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36092025-04-30T06:26:30.308ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36102025-04-30T06:26:30.308ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36112025-04-30T06:26:30.309ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36122025-04-30T06:26:30.309ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36132025-04-30T06:26:30.316ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
36142025-04-30T06:26:30.316ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36152025-04-30T06:26:30.316ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ea 1c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61032 req_id = 6f76b290-83ea-42ba-9828-735eec200665
36162025-04-30T06:26:30.316ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59932,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61032 req_id = 6f76b290-83ea-42ba-9828-735eec200665
36172025-04-30T06:26:30.316ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
36182025-04-30T06:26:30.316ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36192025-04-30T06:26:30.316ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61032 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6f76b290-83ea-42ba-9828-735eec200665
36202025-04-30T06:26:30.316ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3255, tv_nsec: 113665950 } })
36212025-04-30T06:26:30.316ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
36222025-04-30T06:26:30.316ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c3 3d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .=..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60066 req_id = 521d5468-7c53-415d-8cd4-228b96404b10
36232025-04-30T06:26:30.316ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49981,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60066 req_id = 521d5468-7c53-415d-8cd4-228b96404b10
36242025-04-30T06:26:30.316ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36252025-04-30T06:26:30.316ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60066 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 521d5468-7c53-415d-8cd4-228b96404b10
36262025-04-30T06:26:30.380ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36272025-04-30T06:26:30.381ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36282025-04-30T06:26:30.381ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36292025-04-30T06:26:30.381ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36302025-04-30T06:26:30.381ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36312025-04-30T06:26:30.382ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36322025-04-30T06:26:30.382ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 87 90 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64065 req_id = fd308623-d297-47ef-9a0f-1d76e604d6c9
36332025-04-30T06:26:30.389ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34704,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64065 req_id = fd308623-d297-47ef-9a0f-1d76e604d6c9
36342025-04-30T06:26:30.389ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
36352025-04-30T06:26:30.389ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34704, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64065 req_id = fd308623-d297-47ef-9a0f-1d76e604d6c9
36362025-04-30T06:26:30.389ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
36372025-04-30T06:26:30.389ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
36382025-04-30T06:26:30.389ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
36392025-04-30T06:26:30.391ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36402025-04-30T06:26:30.399ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36412025-04-30T06:26:30.399ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36422025-04-30T06:26:30.399ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36432025-04-30T06:26:30.400ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36442025-04-30T06:26:30.404ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36452025-04-30T06:26:30.404ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36462025-04-30T06:26:30.405ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36472025-04-30T06:26:30.405ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d4adeea-d1a2-40dc-91b3-cbca480983aa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36482025-04-30T06:26:30.405ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
36492025-04-30T06:26:30.408ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85424cfb-07c9-4eea-88dd-16fbda33612f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36502025-04-30T06:26:30.408ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
36512025-04-30T06:26:30.411ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28639e2a-7886-41f4-82a5-2f6a2cf23d79", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36522025-04-30T06:26:30.411ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
36532025-04-30T06:26:30.427ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6446f321-e285-443c-9671-9d1728918aa6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36542025-04-30T06:26:30.427ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
36552025-04-30T06:26:30.431ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "528d6426-ec5b-4036-828b-26ed9b238c54", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36562025-04-30T06:26:30.431ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
36572025-04-30T06:26:30.433ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dde204ec-a5ed-4a02-ba2f-47faf8504cce", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36582025-04-30T06:26:30.433ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
36592025-04-30T06:26:30.434ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c92220e0-ff2d-48ba-a91c-222a4259f11c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36602025-04-30T06:26:30.435ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
36612025-04-30T06:26:30.436ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dee36f9-a114-450c-95d4-4f02f761fc0d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36622025-04-30T06:26:30.436ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
36632025-04-30T06:26:30.437ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f901ac48-8cce-4e2a-bc92-99db5f262fd1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36642025-04-30T06:26:30.437ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
36652025-04-30T06:26:30.438ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12ff4bbc-6363-429b-ad81-aae23875fd95", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36662025-04-30T06:26:30.438ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
36672025-04-30T06:26:30.442ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
36682025-04-30T06:26:30.443ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "190df028-eab7-4969-82f2-2dd746b75fa3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36692025-04-30T06:26:30.443ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
36702025-04-30T06:26:30.445ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eb7455c-01bb-4dd1-a1da-7ef423b03161", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36712025-04-30T06:26:30.445ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
36722025-04-30T06:26:30.446ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60dc3a05-4316-411e-b149-97e1e46e7a71", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36732025-04-30T06:26:30.446ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
36742025-04-30T06:26:30.448ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7abeb87f-fa7b-42a1-80d8-4c4faebde6a3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36752025-04-30T06:26:30.448ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
36762025-04-30T06:26:30.450ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55b85cd5-590a-45a2-ab44-881b4981631c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36772025-04-30T06:26:30.450ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
36782025-04-30T06:26:30.451ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a630b31-5df2-46a8-8b91-a3cffaf4a2b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36792025-04-30T06:26:30.451ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
36802025-04-30T06:26:30.453ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d07fea7c-93ac-4f5a-a196-59215fb87087", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36812025-04-30T06:26:30.453ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
36822025-04-30T06:26:30.455ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cd525f3-9581-475a-a571-71ea9ce53caa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36832025-04-30T06:26:30.455ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
36842025-04-30T06:26:30.457ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d39d84a-c771-4686-8b0a-f24b91da754a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36852025-04-30T06:26:30.457ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
36862025-04-30T06:26:30.458ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40bb5b2c-3c3a-4e5c-9aa5-9e9dcb3477f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36872025-04-30T06:26:30.458ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
36882025-04-30T06:26:30.459ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b23201d5-9664-4993-87ab-d32efba96569", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36892025-04-30T06:26:30.459ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
36902025-04-30T06:26:30.460ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d043db9f-3c02-41d6-961b-92afb2381769", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36912025-04-30T06:26:30.460ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
36922025-04-30T06:26:30.461ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b9f8eef-3c81-4759-9e6d-a3d5ae4c03f6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36932025-04-30T06:26:30.461ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
36942025-04-30T06:26:30.462ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70fad5b9-c218-4521-97b4-3d277184a973", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36952025-04-30T06:26:30.462ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
36962025-04-30T06:26:30.465ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9bf888b-3a0b-4ad9-914c-1347b410e860", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36972025-04-30T06:26:30.465ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
36982025-04-30T06:26:30.467ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "418fdf95-00dc-4cbb-96c3-286c1ce3c789", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
36992025-04-30T06:26:30.467ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
37002025-04-30T06:26:30.469ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48c494e8-8c99-485a-97ad-7d1615f5ad8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37012025-04-30T06:26:30.469ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
37022025-04-30T06:26:30.471ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eb876ec-470e-4d4c-aa82-c8a72a30e31b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37032025-04-30T06:26:30.471ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
37042025-04-30T06:26:30.473ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f39ddc5-0589-4372-8991-14e68acc12ec", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37052025-04-30T06:26:30.473ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
37062025-04-30T06:26:30.475ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "441dc599-0290-4948-8706-924128b990fe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37072025-04-30T06:26:30.475ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
37082025-04-30T06:26:30.475ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be74b955-8f39-4ddd-8510-3f8b82b3627f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37092025-04-30T06:26:30.476ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
37102025-04-30T06:26:30.477ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "074b75a2-c500-40a3-9c86-113d184a7a87", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37112025-04-30T06:26:30.477ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
37122025-04-30T06:26:30.479ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c4ea2a-7cd2-420f-8f0d-317557551aa8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37132025-04-30T06:26:30.479ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
37142025-04-30T06:26:30.481ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5691abf-feb1-4e57-930d-4e0a408ebc34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37152025-04-30T06:26:30.482ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
37162025-04-30T06:26:30.483ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7904a13b-a632-41be-a347-ed1d99682661", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37172025-04-30T06:26:30.483ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
37182025-04-30T06:26:30.486ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e76e17c-04db-43c6-b45c-71df0dbb5797", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37192025-04-30T06:26:30.486ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
37202025-04-30T06:26:30.490ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b36cc2d4-ffe9-4678-9fdc-cee660a7be2b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37212025-04-30T06:26:30.490ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
37222025-04-30T06:26:30.493ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37232025-04-30T06:26:30.494ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37242025-04-30T06:26:30.494ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37252025-04-30T06:26:30.494ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37262025-04-30T06:26:30.495ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37272025-04-30T06:26:30.499ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37282025-04-30T06:26:30.508ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37292025-04-30T06:26:30.508ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37302025-04-30T06:26:30.508ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37312025-04-30T06:26:30.509ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37322025-04-30T06:26:30.509ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe9a99a4-195f-4075-9010-2579a5c59161", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37332025-04-30T06:26:30.509ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
37342025-04-30T06:26:30.513ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37352025-04-30T06:26:30.513ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37362025-04-30T06:26:30.514ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37372025-04-30T06:26:30.517ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1063e592-93e6-428b-95a2-df23419e0afe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37382025-04-30T06:26:30.518ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
37392025-04-30T06:26:30.524ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb037962-29a8-4e97-801e-0ee43db563d9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37402025-04-30T06:26:30.524ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
37412025-04-30T06:26:30.527ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f322924-e57d-457b-9f71-7eb9762532e3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37422025-04-30T06:26:30.527ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
37432025-04-30T06:26:30.527ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68def866-4619-4800-ab6d-1d85810c59c5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37442025-04-30T06:26:30.528ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
37452025-04-30T06:26:30.529ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88b75b23-8983-4718-8dfe-efc712f394bd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37462025-04-30T06:26:30.530ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
37472025-04-30T06:26:30.531ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "683c8260-a83b-4070-93d0-c0e47c7b55f1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37482025-04-30T06:26:30.532ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
37492025-04-30T06:26:30.533ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "298b0720-dec2-4735-a504-5b012fedc46c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37502025-04-30T06:26:30.533ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37512025-04-30T06:26:30.533ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37522025-04-30T06:26:30.533ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37532025-04-30T06:26:30.533ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37542025-04-30T06:26:30.534ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37552025-04-30T06:26:30.536ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37562025-04-30T06:26:30.544ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37572025-04-30T06:26:30.544ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37582025-04-30T06:26:30.544ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37592025-04-30T06:26:30.544ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37602025-04-30T06:26:30.544ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37612025-04-30T06:26:30.544ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37622025-04-30T06:26:30.547ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37632025-04-30T06:26:30.557ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37642025-04-30T06:26:30.557ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37652025-04-30T06:26:30.557ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37662025-04-30T06:26:30.558ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37672025-04-30T06:26:30.636ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
37682025-04-30T06:26:30.636ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
37692025-04-30T06:26:30.638ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:58378 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
37702025-04-30T06:26:30.639ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:58378 req_id = 7ca07b6b-38e4-43f9-8999-54d1f497afc1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
37712025-04-30T06:26:30.640ZINFOtest_disk_metrics (dropshot): request completed latency_us = 488 local_addr = [::1]:50989 method = GET remote_addr = [::1]:58378 req_id = 7ca07b6b-38e4-43f9-8999-54d1f497afc1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
37722025-04-30T06:26:30.640ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
37732025-04-30T06:26:30.640ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ca07b6b-38e4-43f9-8999-54d1f497afc1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
37742025-04-30T06:26:30.642ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
37752025-04-30T06:26:30.643ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
37762025-04-30T06:26:30.643ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
37772025-04-30T06:26:30.658ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
37782025-04-30T06:26:30.658ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37792025-04-30T06:26:30.658ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37802025-04-30T06:26:30.658ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37812025-04-30T06:26:30.658ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37822025-04-30T06:26:30.703ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37832025-04-30T06:26:30.709ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
37842025-04-30T06:26:30.709ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37852025-04-30T06:26:30.709ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37862025-04-30T06:26:30.710ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37872025-04-30T06:26:30.711ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37882025-04-30T06:26:30.711ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37892025-04-30T06:26:30.711ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37902025-04-30T06:26:30.711ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37912025-04-30T06:26:30.711ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37922025-04-30T06:26:30.715ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37932025-04-30T06:26:30.728ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37942025-04-30T06:26:30.728ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37952025-04-30T06:26:30.729ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37962025-04-30T06:26:30.729ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37972025-04-30T06:26:30.734ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37982025-04-30T06:26:30.734ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37992025-04-30T06:26:30.734ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38002025-04-30T06:26:30.822ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
38012025-04-30T06:26:30.859ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
38022025-04-30T06:26:30.861ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38032025-04-30T06:26:30.861ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38042025-04-30T06:26:30.861ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38052025-04-30T06:26:30.862ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 562.433693ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38062025-04-30T06:26:30.862ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
38072025-04-30T06:26:30.862ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38082025-04-30T06:26:30.862ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38092025-04-30T06:26:30.862ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38102025-04-30T06:26:30.862ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38112025-04-30T06:26:30.863ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38122025-04-30T06:26:30.864ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 400b2fbe-4a74-4d93-9f12-91cb02e40dba uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38132025-04-30T06:26:30.864ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 400b2fbe-4a74-4d93-9f12-91cb02e40dba resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38142025-04-30T06:26:30.864ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 400b2fbe-4a74-4d93-9f12-91cb02e40dba roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38152025-04-30T06:26:30.864ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 400b2fbe-4a74-4d93-9f12-91cb02e40dba resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38162025-04-30T06:26:30.871ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38172025-04-30T06:26:30.881ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38182025-04-30T06:26:30.881ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38192025-04-30T06:26:30.881ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38202025-04-30T06:26:30.882ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38212025-04-30T06:26:30.883ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 19455 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 400b2fbe-4a74-4d93-9f12-91cb02e40dba response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38222025-04-30T06:26:30.883ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38232025-04-30T06:26:30.883ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38242025-04-30T06:26:30.883ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38252025-04-30T06:26:30.883ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38262025-04-30T06:26:30.884ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38272025-04-30T06:26:30.884ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "400b2fbe-4a74-4d93-9f12-91cb02e40dba", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38282025-04-30T06:26:30.884ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38292025-04-30T06:26:30.886ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f2bfb010-51b1-4859-a2a4-6dfe1246c7db uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38302025-04-30T06:26:30.887ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f2bfb010-51b1-4859-a2a4-6dfe1246c7db resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38312025-04-30T06:26:30.887ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f2bfb010-51b1-4859-a2a4-6dfe1246c7db roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38322025-04-30T06:26:30.889ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f2bfb010-51b1-4859-a2a4-6dfe1246c7db resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38332025-04-30T06:26:30.889ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
38342025-04-30T06:26:30.889ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38352025-04-30T06:26:30.891ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: be 71 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .q..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61388 req_id = 9f92cd73-5668-49cd-a146-0681e49ee619
38362025-04-30T06:26:30.891ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48753,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61388 req_id = 9f92cd73-5668-49cd-a146-0681e49ee619
38372025-04-30T06:26:30.892ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
38382025-04-30T06:26:30.892ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38392025-04-30T06:26:30.892ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61388 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9f92cd73-5668-49cd-a146-0681e49ee619
38402025-04-30T06:26:30.893ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 7013 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f2bfb010-51b1-4859-a2a4-6dfe1246c7db response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38412025-04-30T06:26:30.893ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38422025-04-30T06:26:30.908ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38432025-04-30T06:26:30.908ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38442025-04-30T06:26:30.908ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38452025-04-30T06:26:30.909ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38462025-04-30T06:26:30.910ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3255, tv_nsec: 708865496 } })
38472025-04-30T06:26:30.910ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
38482025-04-30T06:26:30.910ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2bfb010-51b1-4859-a2a4-6dfe1246c7db", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38492025-04-30T06:26:30.910ZDEBGtest_disk_metrics (oximeter-agent): registered new metric producer address = 127.0.0.1:39352 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
38502025-04-30T06:26:30.982ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
38512025-04-30T06:26:30.983ZDEBGtest_disk_metrics (oximeter-agent): registered new metric producer address = [::1]:57619 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38522025-04-30T06:26:30.984ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 88 ed 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56955 req_id = 38a5c2a6-98a9-4418-b243-27262b4f684a
38532025-04-30T06:26:30.984ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35053,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56955 req_id = 38a5c2a6-98a9-4418-b243-27262b4f684a
38542025-04-30T06:26:30.984ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38552025-04-30T06:26:30.985ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56955 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 38a5c2a6-98a9-4418-b243-27262b4f684a
38562025-04-30T06:26:31.084ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38572025-04-30T06:26:31.086ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38582025-04-30T06:26:31.087ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38592025-04-30T06:26:31.088ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38602025-04-30T06:26:31.167ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38612025-04-30T06:26:31.169ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
38622025-04-30T06:26:31.173ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 82 bc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61628 req_id = 0baae807-bec3-400e-830a-4a9911ae2f8f
38632025-04-30T06:26:31.173ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33468,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61628 req_id = 0baae807-bec3-400e-830a-4a9911ae2f8f
38642025-04-30T06:26:31.173ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
38652025-04-30T06:26:31.174ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33468, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61628 req_id = 0baae807-bec3-400e-830a-4a9911ae2f8f
38662025-04-30T06:26:31.176ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
38672025-04-30T06:26:31.176ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
38682025-04-30T06:26:31.177ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
38692025-04-30T06:26:31.181ZTRACtest_disk_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38702025-04-30T06:26:31.181ZTRACtest_disk_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38712025-04-30T06:26:31.193ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "28cccda1-a51e-4656-a9c3-52ed63e7af35", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38722025-04-30T06:26:31.194ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
38732025-04-30T06:26:31.241ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68fb953e-5bdc-4549-a226-d0bc7e1a95f1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38742025-04-30T06:26:31.241ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
38752025-04-30T06:26:31.247ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ca47055-87b4-4690-b007-5326cd764a9e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38762025-04-30T06:26:31.247ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
38772025-04-30T06:26:31.251ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
38782025-04-30T06:26:31.252ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dccb1c1-c090-416b-b18a-9081298b8578", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38792025-04-30T06:26:31.252ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
38802025-04-30T06:26:31.254ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ae327e2-b09c-4762-8f0b-63c4b0657f03", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38812025-04-30T06:26:31.254ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
38822025-04-30T06:26:31.258ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7edb7b9-5d08-4a05-82ea-551ce7ec8e8c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38832025-04-30T06:26:31.258ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
38842025-04-30T06:26:31.265ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb8c512-1e16-410b-89ae-74db14d4dd0e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38852025-04-30T06:26:31.273ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
38862025-04-30T06:26:31.298ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "538cefc6-a3e1-422e-b476-98db44c33348", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38872025-04-30T06:26:31.298ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
38882025-04-30T06:26:31.300ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0b0ea57-90c2-4f7c-b205-10759c672016", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38892025-04-30T06:26:31.300ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
38902025-04-30T06:26:31.303ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa09e927-8e29-4c9e-ba32-d8f6316d4bb7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38912025-04-30T06:26:31.303ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
38922025-04-30T06:26:31.303ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38932025-04-30T06:26:31.303ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38942025-04-30T06:26:31.303ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38952025-04-30T06:26:31.303ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38962025-04-30T06:26:31.306ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38972025-04-30T06:26:31.307ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4e85233-0bc7-4dce-8a30-51f858181398", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
38982025-04-30T06:26:31.307ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
38992025-04-30T06:26:31.309ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82e4e42f-95ca-47a8-8456-dc3a7de410a0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39002025-04-30T06:26:31.309ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
39012025-04-30T06:26:31.309ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39022025-04-30T06:26:31.318ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39032025-04-30T06:26:31.318ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39042025-04-30T06:26:31.318ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39052025-04-30T06:26:31.318ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39062025-04-30T06:26:31.325ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27663910-7011-4f90-8552-75e673a3e3a3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39072025-04-30T06:26:31.326ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
39082025-04-30T06:26:31.326ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39092025-04-30T06:26:31.326ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39102025-04-30T06:26:31.327ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39112025-04-30T06:26:31.333ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "953831bc-7e47-4c14-8f65-a72466428e7f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39122025-04-30T06:26:31.333ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
39132025-04-30T06:26:31.338ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "955755a3-2f15-48b5-ad84-ae3c96668b78", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39142025-04-30T06:26:31.338ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
39152025-04-30T06:26:31.376ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c00021f-5c12-4460-b229-d11865509ffc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39162025-04-30T06:26:31.376ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
39172025-04-30T06:26:31.381ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0626dda5-7826-462d-9a6e-4959ea584700", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39182025-04-30T06:26:31.381ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
39192025-04-30T06:26:31.384ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a3782d-3c3a-4343-a424-040257a4561a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39202025-04-30T06:26:31.384ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
39212025-04-30T06:26:31.391ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
39222025-04-30T06:26:31.394ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5772bc8f-3be9-47d4-a8c5-74c9a485b7e9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39232025-04-30T06:26:31.394ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
39242025-04-30T06:26:31.395ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a90a7638-44f3-4e31-940f-95b362f6e2af", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39252025-04-30T06:26:31.395ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
39262025-04-30T06:26:31.396ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d200b58-1472-481d-b9f5-d4a750cc1983", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39272025-04-30T06:26:31.396ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
39282025-04-30T06:26:31.398ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b05f998e-ab2d-46c3-9f0d-4f47cff8f3e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39292025-04-30T06:26:31.398ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
39302025-04-30T06:26:31.399ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "518b263b-1aa2-438b-bb4d-3cf9b169a356", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39312025-04-30T06:26:31.400ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
39322025-04-30T06:26:31.402ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8511712-aab7-4608-8846-d90e0604227a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39332025-04-30T06:26:31.402ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
39342025-04-30T06:26:31.404ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1959b99d-1288-4d00-b20e-8083d9471108", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39352025-04-30T06:26:31.404ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
39362025-04-30T06:26:31.406ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c711faf6-821b-4de3-874f-0d5e86fc661d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39372025-04-30T06:26:31.406ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
39382025-04-30T06:26:31.408ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd606489-3edc-4555-b8c9-afa304b6cf9b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39392025-04-30T06:26:31.408ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
39402025-04-30T06:26:31.413ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc8fca98-ee90-4b7d-8d95-3f61e89a750e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39412025-04-30T06:26:31.413ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
39422025-04-30T06:26:31.428ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6cd50c0-56a8-4bb7-a83a-af3321a235f3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39432025-04-30T06:26:31.429ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
39442025-04-30T06:26:31.445ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39452025-04-30T06:26:31.445ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39462025-04-30T06:26:31.445ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39472025-04-30T06:26:31.445ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39482025-04-30T06:26:31.446ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39492025-04-30T06:26:31.451ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39502025-04-30T06:26:31.456ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39512025-04-30T06:26:31.456ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39522025-04-30T06:26:31.456ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39532025-04-30T06:26:31.456ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39542025-04-30T06:26:31.460ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30819666-5ceb-4602-8ead-0618ba512181", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39552025-04-30T06:26:31.460ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
39562025-04-30T06:26:31.461ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39572025-04-30T06:26:31.461ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39582025-04-30T06:26:31.468ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39592025-04-30T06:26:31.470ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e93d618-5bb7-474d-8d45-636cf62b47bf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39602025-04-30T06:26:31.470ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
39612025-04-30T06:26:31.473ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "842701d8-d0e8-4df6-9b37-225707117e99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39622025-04-30T06:26:31.473ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
39632025-04-30T06:26:31.481ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "123fb70e-b3ec-41a3-b502-4eefe74b47d1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39642025-04-30T06:26:31.481ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
39652025-04-30T06:26:31.484ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b56a1f25-246a-4ca1-b65f-d9c3054a5f10", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39662025-04-30T06:26:31.484ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
39672025-04-30T06:26:31.491ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92804abd-97e1-412f-a018-612b6dee7f3b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39682025-04-30T06:26:31.491ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
39692025-04-30T06:26:31.495ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71456f2e-221b-43b5-a1a4-d4e3d9ed53a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39702025-04-30T06:26:31.495ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
39712025-04-30T06:26:31.501ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca43f04d-b58b-4c5d-a8cd-efd5c221411f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39722025-04-30T06:26:31.502ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
39732025-04-30T06:26:31.542ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
39742025-04-30T06:26:31.560ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4a7c841-34b7-4bf4-8a9c-2e2fbfc5b8e9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39752025-04-30T06:26:31.560ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
39762025-04-30T06:26:31.562ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d622ffb-a4cf-48f8-ae2c-c27b6401922a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39772025-04-30T06:26:31.562ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
39782025-04-30T06:26:31.564ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fae1917-def6-4dd5-a96e-20425c90a438", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39792025-04-30T06:26:31.564ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
39802025-04-30T06:26:31.570ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54b69681-3fc8-4e8c-b6ff-4856d05a1a5c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39812025-04-30T06:26:31.570ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
39822025-04-30T06:26:31.571ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cdf9301-84e0-43fc-ac47-4c521e6080ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39832025-04-30T06:26:31.571ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
39842025-04-30T06:26:31.574ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5296ec05-cc63-44d1-80ad-5428035bff65", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39852025-04-30T06:26:31.574ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
39862025-04-30T06:26:31.593ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2b8947e-c957-49d6-9b10-5edae2dc807d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39872025-04-30T06:26:31.593ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
39882025-04-30T06:26:31.594ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b228dd-02a9-411e-bae0-a118e4611258", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
39892025-04-30T06:26:31.595ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39902025-04-30T06:26:31.595ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39912025-04-30T06:26:31.595ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39922025-04-30T06:26:31.595ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39932025-04-30T06:26:31.596ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39942025-04-30T06:26:31.599ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39952025-04-30T06:26:31.599ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39962025-04-30T06:26:31.599ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39972025-04-30T06:26:31.599ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39982025-04-30T06:26:31.600ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39992025-04-30T06:26:31.609ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40002025-04-30T06:26:31.613ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40012025-04-30T06:26:31.613ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40022025-04-30T06:26:31.614ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40032025-04-30T06:26:31.614ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40042025-04-30T06:26:31.614ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40052025-04-30T06:26:31.614ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40062025-04-30T06:26:31.618ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40072025-04-30T06:26:31.622ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40082025-04-30T06:26:31.629ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40092025-04-30T06:26:31.629ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40102025-04-30T06:26:31.642ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40112025-04-30T06:26:31.645ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40122025-04-30T06:26:31.653ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40132025-04-30T06:26:31.653ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40142025-04-30T06:26:31.653ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40152025-04-30T06:26:31.654ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40162025-04-30T06:26:31.696ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40172025-04-30T06:26:31.696ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40182025-04-30T06:26:31.697ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40192025-04-30T06:26:31.863ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
40202025-04-30T06:26:31.863ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
40212025-04-30T06:26:31.870ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:53411 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40222025-04-30T06:26:31.871ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:53411 req_id = 7eb512e5-f68d-497f-a484-cc0dbc70c9d9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40232025-04-30T06:26:31.872ZINFOtest_disk_metrics (dropshot): request completed latency_us = 774 local_addr = [::1]:50989 method = GET remote_addr = [::1]:53411 req_id = 7eb512e5-f68d-497f-a484-cc0dbc70c9d9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40242025-04-30T06:26:31.872ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eb512e5-f68d-497f-a484-cc0dbc70c9d9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
40252025-04-30T06:26:31.874ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40262025-04-30T06:26:31.874ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40272025-04-30T06:26:31.875ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
40282025-04-30T06:26:31.876ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40292025-04-30T06:26:31.877ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40302025-04-30T06:26:31.877ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40312025-04-30T06:26:31.877ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40322025-04-30T06:26:31.878ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40332025-04-30T06:26:31.882ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40342025-04-30T06:26:31.887ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40352025-04-30T06:26:31.887ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40362025-04-30T06:26:31.887ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40372025-04-30T06:26:31.888ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40382025-04-30T06:26:31.904ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
40392025-04-30T06:26:31.958ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40402025-04-30T06:26:31.958ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40412025-04-30T06:26:31.958ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.096715374s iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40422025-04-30T06:26:31.958ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
40432025-04-30T06:26:31.958ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40442025-04-30T06:26:31.958ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40452025-04-30T06:26:31.959ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40462025-04-30T06:26:31.959ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40472025-04-30T06:26:31.960ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40482025-04-30T06:26:31.960ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40492025-04-30T06:26:31.960ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40502025-04-30T06:26:31.960ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40512025-04-30T06:26:31.960ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40522025-04-30T06:26:31.961ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40532025-04-30T06:26:31.970ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40542025-04-30T06:26:31.974ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40552025-04-30T06:26:31.974ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40562025-04-30T06:26:31.974ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40572025-04-30T06:26:31.974ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40582025-04-30T06:26:31.979ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40592025-04-30T06:26:31.985ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40602025-04-30T06:26:31.985ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40612025-04-30T06:26:31.985ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40622025-04-30T06:26:31.986ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40632025-04-30T06:26:31.987ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
40642025-04-30T06:26:31.987ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40652025-04-30T06:26:31.990ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40662025-04-30T06:26:31.990ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40672025-04-30T06:26:31.991ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40682025-04-30T06:26:31.991ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40692025-04-30T06:26:31.992ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40702025-04-30T06:26:31.992ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d1 ba 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56490 req_id = 6a139f8b-efc8-4425-9fcd-08ccfd96abec
40712025-04-30T06:26:31.992ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53690,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56490 req_id = 6a139f8b-efc8-4425-9fcd-08ccfd96abec
40722025-04-30T06:26:31.993ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
40732025-04-30T06:26:31.993ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40742025-04-30T06:26:31.993ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56490 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6a139f8b-efc8-4425-9fcd-08ccfd96abec
40752025-04-30T06:26:31.994ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3256, tv_nsec: 793167708 } })
40762025-04-30T06:26:31.994ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
40772025-04-30T06:26:31.998ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40782025-04-30T06:26:31.999ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40792025-04-30T06:26:31.999ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40802025-04-30T06:26:31.999ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40812025-04-30T06:26:31.999ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4e d2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e N...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61950 req_id = d1d10c3d-84d5-424d-a378-45860ad4ed30
40822025-04-30T06:26:32.000ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20178,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61950 req_id = d1d10c3d-84d5-424d-a378-45860ad4ed30
40832025-04-30T06:26:32.001ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40842025-04-30T06:26:32.001ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61950 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d1d10c3d-84d5-424d-a378-45860ad4ed30
40852025-04-30T06:26:32.082ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40862025-04-30T06:26:32.084ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c1 a6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54498 req_id = 4853a0c6-c654-40d8-a7e8-8c88b947d6ee
40872025-04-30T06:26:32.084ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49574,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54498 req_id = 4853a0c6-c654-40d8-a7e8-8c88b947d6ee
40882025-04-30T06:26:32.084ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
40892025-04-30T06:26:32.085ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49574, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54498 req_id = 4853a0c6-c654-40d8-a7e8-8c88b947d6ee
40902025-04-30T06:26:32.087ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
40912025-04-30T06:26:32.087ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
40922025-04-30T06:26:32.087ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
40932025-04-30T06:26:32.089ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c68596dc-195d-4d8b-9fa3-bee30d8bb5ce", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
40942025-04-30T06:26:32.090ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
40952025-04-30T06:26:32.092ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a01594-4e32-48a2-b35f-395a2c69d07d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
40962025-04-30T06:26:32.092ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
40972025-04-30T06:26:32.095ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14cf7249-19b9-4d1b-81ec-a87e426ab868", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
40982025-04-30T06:26:32.096ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
40992025-04-30T06:26:32.098ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b91c9de3-94c7-4972-ab03-9e9d6fe1966d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41002025-04-30T06:26:32.098ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
41012025-04-30T06:26:32.100ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "452ba128-68b1-47fc-b6fd-7695b8cd6294", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41022025-04-30T06:26:32.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
41032025-04-30T06:26:32.103ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "878c143c-0283-43b4-a9db-e724164f9e1a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41042025-04-30T06:26:32.103ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
41052025-04-30T06:26:32.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17167e20-1203-41bb-bbc6-c43ef05b2dd0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41062025-04-30T06:26:32.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
41072025-04-30T06:26:32.110ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
41082025-04-30T06:26:32.112ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe34254-c843-4143-97d6-84d28b986870", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41092025-04-30T06:26:32.112ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
41102025-04-30T06:26:32.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f906f42-532d-4a7e-a074-2fd2984249fa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41112025-04-30T06:26:32.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
41122025-04-30T06:26:32.115ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66475272-3aed-450b-9fb8-0d7c79a354ed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41132025-04-30T06:26:32.116ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
41142025-04-30T06:26:32.117ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "852e3525-c1e0-45a1-a67c-cccf0d03842b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41152025-04-30T06:26:32.122ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
41162025-04-30T06:26:32.124ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f5063a3-5e34-4b91-965a-04c0dbcc6e43", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41172025-04-30T06:26:32.124ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
41182025-04-30T06:26:32.126ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba0cfcc6-75c5-49af-8908-0f3f6041177b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41192025-04-30T06:26:32.126ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
41202025-04-30T06:26:32.128ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "373eb05d-2a39-47cd-99b0-3bff6952eae1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41212025-04-30T06:26:32.128ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
41222025-04-30T06:26:32.130ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4efe29f9-eb16-46a6-84d5-d1fd9e4d623d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41232025-04-30T06:26:32.130ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
41242025-04-30T06:26:32.132ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5b631cd-02ed-428b-a27b-08df1a466941", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41252025-04-30T06:26:32.132ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
41262025-04-30T06:26:32.136ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9036116e-0e61-44c8-93dc-87ce7f16ab4b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41272025-04-30T06:26:32.137ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
41282025-04-30T06:26:32.143ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "500ba259-0fcd-41d0-bfa1-c85e2d180677", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41292025-04-30T06:26:32.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
41302025-04-30T06:26:32.168ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41312025-04-30T06:26:32.168ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41322025-04-30T06:26:32.168ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41332025-04-30T06:26:32.168ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41342025-04-30T06:26:32.169ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41352025-04-30T06:26:32.173ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41362025-04-30T06:26:32.180ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41372025-04-30T06:26:32.180ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41382025-04-30T06:26:32.180ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41392025-04-30T06:26:32.182ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41402025-04-30T06:26:32.182ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee8678e5-463d-4929-915a-af7516fd07a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41412025-04-30T06:26:32.183ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
41422025-04-30T06:26:32.183ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41432025-04-30T06:26:32.183ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41442025-04-30T06:26:32.185ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
41452025-04-30T06:26:32.186ZDEBGtest_disk_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
41462025-04-30T06:26:32.186ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
41472025-04-30T06:26:32.188ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "482429be-efec-416c-a358-58ac449e3d50", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41482025-04-30T06:26:32.189ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
41492025-04-30T06:26:32.189ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41502025-04-30T06:26:32.189ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41512025-04-30T06:26:32.190ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41522025-04-30T06:26:32.190ZTRACtest_disk_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
41532025-04-30T06:26:32.190ZDEBGtest_disk_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:32.186102420Z } timeseries_name = integration_target:integration_metric
41542025-04-30T06:26:32.191ZDEBGtest_disk_metrics (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
41552025-04-30T06:26:32.191ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
41562025-04-30T06:26:32.194ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e6f67d2-55bb-4afb-832f-90d3e1441f02", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41572025-04-30T06:26:32.194ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
41582025-04-30T06:26:32.195ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aefdef34-ac00-4d12-9d52-536fa40e42da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41592025-04-30T06:26:32.195ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
41602025-04-30T06:26:32.196ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
41612025-04-30T06:26:32.197ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b73c5120-c5cb-4270-bfd9-a09d6307e5df", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41622025-04-30T06:26:32.198ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
41632025-04-30T06:26:32.199ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
41642025-04-30T06:26:32.199ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
41652025-04-30T06:26:32.200ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2bcd023-08b2-4581-bc75-c4b92ff27072", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41662025-04-30T06:26:32.200ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
41672025-04-30T06:26:32.203ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
41682025-04-30T06:26:32.203ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41692025-04-30T06:26:32.204ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "190e6b6a-96d4-4b91-b85a-c7886aca7ab3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41702025-04-30T06:26:32.204ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
41712025-04-30T06:26:32.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a22b34b-d9e0-4bb6-8742-7db0e0982308", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41722025-04-30T06:26:32.213ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
41732025-04-30T06:26:32.215ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44f54003-94a1-40a7-b31f-c06095c23f9b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41742025-04-30T06:26:32.215ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
41752025-04-30T06:26:32.217ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e25c699-c912-4218-ac4a-452aa46e970f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41762025-04-30T06:26:32.218ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
41772025-04-30T06:26:32.220ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "396d9a1f-67ed-475f-b56b-a72e73cfe789", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41782025-04-30T06:26:32.220ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
41792025-04-30T06:26:32.222ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48f842cb-348c-452b-b8d2-21a22e91f20e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41802025-04-30T06:26:32.222ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
41812025-04-30T06:26:32.223ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83882e71-1148-4dbd-8633-701350b9e146", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41822025-04-30T06:26:32.223ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
41832025-04-30T06:26:32.224ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1830a115-777c-4df1-b25c-dceee3628773", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41842025-04-30T06:26:32.224ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
41852025-04-30T06:26:32.225ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7106200-a30c-47a1-8d4d-fd5ba0b7bc35", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41862025-04-30T06:26:32.225ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
41872025-04-30T06:26:32.226ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a56443-b095-4659-9a7a-0d612430f2c4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41882025-04-30T06:26:32.227ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
41892025-04-30T06:26:32.228ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6ddbb34-6220-43ce-a16e-ade8ee3e6988", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41902025-04-30T06:26:32.228ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
41912025-04-30T06:26:32.232ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
41922025-04-30T06:26:32.243ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d62b6bd-6a9e-41ca-9d46-89563afe2e02", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41932025-04-30T06:26:32.243ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
41942025-04-30T06:26:32.246ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62d07577-80ed-406d-8e88-ec74a1efefa5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41952025-04-30T06:26:32.247ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
41962025-04-30T06:26:32.250ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec2ace9f-e6bc-48e3-bf8b-44a948af30fb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41972025-04-30T06:26:32.250ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
41982025-04-30T06:26:32.255ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "940fe8e7-a011-405c-92e3-62b0b813ffc0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
41992025-04-30T06:26:32.255ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
42002025-04-30T06:26:32.258ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b39d0473-7c9b-48dd-8533-51b98e4ddaff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
42012025-04-30T06:26:32.258ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
42022025-04-30T06:26:32.262ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "746b1594-1f65-43ed-870b-6ef3b878aee5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
42032025-04-30T06:26:32.262ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
42042025-04-30T06:26:32.264ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09cc1771-cee0-4502-9e17-fce22e0693e9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
42052025-04-30T06:26:32.264ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
42062025-04-30T06:26:32.266ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af9896e4-76de-47a7-ab3d-a4817a3b8a78", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
42072025-04-30T06:26:32.266ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
42082025-04-30T06:26:32.267ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f911b82f-6fca-4979-8a50-72ccf40404e4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
42092025-04-30T06:26:32.268ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
42102025-04-30T06:26:32.270ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa9ac65a-12fd-4cd7-8b40-3769bc3cceb3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
42112025-04-30T06:26:32.270ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42122025-04-30T06:26:32.270ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42132025-04-30T06:26:32.270ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42142025-04-30T06:26:32.270ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42152025-04-30T06:26:32.271ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42162025-04-30T06:26:32.290ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42172025-04-30T06:26:32.290ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42182025-04-30T06:26:32.290ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42192025-04-30T06:26:32.290ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42202025-04-30T06:26:32.291ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42212025-04-30T06:26:32.291ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
42222025-04-30T06:26:32.291ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42232025-04-30T06:26:32.296ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42242025-04-30T06:26:32.299ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42252025-04-30T06:26:32.299ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42262025-04-30T06:26:32.300ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42272025-04-30T06:26:32.300ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42282025-04-30T06:26:32.300ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42292025-04-30T06:26:32.303ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42302025-04-30T06:26:32.312ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42312025-04-30T06:26:32.313ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42322025-04-30T06:26:32.313ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42332025-04-30T06:26:32.313ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42342025-04-30T06:26:32.314ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42352025-04-30T06:26:32.319ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42362025-04-30T06:26:32.319ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42372025-04-30T06:26:32.319ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42382025-04-30T06:26:32.319ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42392025-04-30T06:26:32.322ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42402025-04-30T06:26:32.322ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42412025-04-30T06:26:32.323ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42422025-04-30T06:26:32.381ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
42432025-04-30T06:26:32.381ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
42442025-04-30T06:26:32.382ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:36343 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42452025-04-30T06:26:32.383ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:36343 req_id = c7058f5d-f174-48bb-93e6-2c1d42878233 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42462025-04-30T06:26:32.383ZINFOtest_disk_metrics (dropshot): request completed latency_us = 738 local_addr = [::1]:50989 method = GET remote_addr = [::1]:36343 req_id = c7058f5d-f174-48bb-93e6-2c1d42878233 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42472025-04-30T06:26:32.384ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7058f5d-f174-48bb-93e6-2c1d42878233", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
42482025-04-30T06:26:32.385ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42492025-04-30T06:26:32.385ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42502025-04-30T06:26:32.385ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
42512025-04-30T06:26:32.386ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42522025-04-30T06:26:32.386ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42532025-04-30T06:26:32.386ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42542025-04-30T06:26:32.387ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42552025-04-30T06:26:32.387ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42562025-04-30T06:26:32.390ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42572025-04-30T06:26:32.395ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42582025-04-30T06:26:32.395ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42592025-04-30T06:26:32.395ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42602025-04-30T06:26:32.396ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42612025-04-30T06:26:32.434ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
42622025-04-30T06:26:32.434ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42632025-04-30T06:26:32.434ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42642025-04-30T06:26:32.435ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 476.152613ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42652025-04-30T06:26:32.435ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
42662025-04-30T06:26:32.435ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42672025-04-30T06:26:32.435ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42682025-04-30T06:26:32.435ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42692025-04-30T06:26:32.435ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42702025-04-30T06:26:32.436ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42712025-04-30T06:26:32.437ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42722025-04-30T06:26:32.442ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42732025-04-30T06:26:32.442ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42742025-04-30T06:26:32.442ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42752025-04-30T06:26:32.443ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42762025-04-30T06:26:32.448ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
42772025-04-30T06:26:32.448ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42782025-04-30T06:26:32.448ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 67 f9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 g............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52249 req_id = 1e20b62f-5eac-4e79-b9cc-5d11ef4a3275
42792025-04-30T06:26:32.449ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26617,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52249 req_id = 1e20b62f-5eac-4e79-b9cc-5d11ef4a3275
42802025-04-30T06:26:32.449ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
42812025-04-30T06:26:32.449ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42822025-04-30T06:26:32.449ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52249 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1e20b62f-5eac-4e79-b9cc-5d11ef4a3275
42832025-04-30T06:26:32.449ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3257, tv_nsec: 248793623 } })
42842025-04-30T06:26:32.449ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
42852025-04-30T06:26:32.450ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c5 e2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51278 req_id = 5bef2e5b-086d-4cd8-8797-6ec14f44672d
42862025-04-30T06:26:32.450ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50658,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51278 req_id = 5bef2e5b-086d-4cd8-8797-6ec14f44672d
42872025-04-30T06:26:32.450ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42882025-04-30T06:26:32.450ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51278 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5bef2e5b-086d-4cd8-8797-6ec14f44672d
42892025-04-30T06:26:32.514ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42902025-04-30T06:26:32.514ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42912025-04-30T06:26:32.514ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42922025-04-30T06:26:32.514ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42932025-04-30T06:26:32.514ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42942025-04-30T06:26:32.516ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42952025-04-30T06:26:32.517ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 27 a1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c '............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50990 req_id = bc53d873-0f19-43ad-bf73-48ce383f467e
42962025-04-30T06:26:32.517ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10145,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50990 req_id = bc53d873-0f19-43ad-bf73-48ce383f467e
42972025-04-30T06:26:32.517ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
42982025-04-30T06:26:32.517ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10145, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50990 req_id = bc53d873-0f19-43ad-bf73-48ce383f467e
42992025-04-30T06:26:32.518ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
43002025-04-30T06:26:32.518ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
43012025-04-30T06:26:32.518ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
43022025-04-30T06:26:32.520ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43032025-04-30T06:26:32.529ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43042025-04-30T06:26:32.531ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43052025-04-30T06:26:32.531ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43062025-04-30T06:26:32.535ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43072025-04-30T06:26:32.537ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "38658728-a0d1-4c22-8eb2-5de14eb28171", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43082025-04-30T06:26:32.537ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
43092025-04-30T06:26:32.540ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43102025-04-30T06:26:32.540ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43112025-04-30T06:26:32.542ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43122025-04-30T06:26:32.544ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77772196-ba66-44cf-ba56-e619ed60760d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43132025-04-30T06:26:32.544ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
43142025-04-30T06:26:32.548ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1d2e168-48a3-4c9f-b3fd-48eaea6192a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43152025-04-30T06:26:32.548ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
43162025-04-30T06:26:32.551ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4773880b-621e-4724-afe8-c4ba9dc253be", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43172025-04-30T06:26:32.551ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
43182025-04-30T06:26:32.555ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad2056b9-ab70-40fd-a810-ab07801bf07a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43192025-04-30T06:26:32.555ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
43202025-04-30T06:26:32.558ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d361ace-27eb-4089-ba2c-ce1e1097e5f4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43212025-04-30T06:26:32.558ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
43222025-04-30T06:26:32.562ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46336ce6-808a-4f0b-b635-ffb0b26b95f4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43232025-04-30T06:26:32.562ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
43242025-04-30T06:26:32.566ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3d60666-b4e8-40d6-8622-b7c9d8424047", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43252025-04-30T06:26:32.566ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
43262025-04-30T06:26:32.568ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ced1ac10-1749-4c07-9d81-b291de55a5c0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43272025-04-30T06:26:32.568ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
43282025-04-30T06:26:32.571ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1c02393-b1e6-4b5a-8c0c-17de6205ebed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43292025-04-30T06:26:32.571ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
43302025-04-30T06:26:32.571ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
43312025-04-30T06:26:32.573ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a92bbc8f-8e9d-45e8-b04d-5950d3eff188", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43322025-04-30T06:26:32.573ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
43332025-04-30T06:26:32.574ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fa935db-70fb-47bc-864d-0e66fd12c8b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43342025-04-30T06:26:32.574ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
43352025-04-30T06:26:32.575ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44353d9b-d7c5-4e7c-8aea-e340c2d612d9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43362025-04-30T06:26:32.576ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
43372025-04-30T06:26:32.579ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9d10ad6-76e2-45c9-8e50-41e7ab82a35b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43382025-04-30T06:26:32.579ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
43392025-04-30T06:26:32.582ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5be5be-827a-4649-9238-36228a64b84c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43402025-04-30T06:26:32.582ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
43412025-04-30T06:26:32.584ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9303b81a-3489-4beb-9521-b8f42f5ab519", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43422025-04-30T06:26:32.584ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
43432025-04-30T06:26:32.589ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f6a2146-3754-47c0-9afc-f25c686a1a66", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43442025-04-30T06:26:32.589ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
43452025-04-30T06:26:32.593ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5683afb9-bee0-43c5-b3d3-694ff2e6c2bd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43462025-04-30T06:26:32.594ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
43472025-04-30T06:26:32.596ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86b06b3e-47d9-44d4-a3ac-7eec831e4d17", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43482025-04-30T06:26:32.596ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
43492025-04-30T06:26:32.597ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c13f2f-5ce8-435d-a916-de1e72551c1f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43502025-04-30T06:26:32.597ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
43512025-04-30T06:26:32.598ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd465363-b77f-416f-870c-b8a7e61c6403", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43522025-04-30T06:26:32.598ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
43532025-04-30T06:26:32.599ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a03eae4-e691-4594-ac09-15d17d2eb980", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43542025-04-30T06:26:32.599ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
43552025-04-30T06:26:32.600ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e289368e-350f-4474-b959-3f4f16c8cb7b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43562025-04-30T06:26:32.600ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
43572025-04-30T06:26:32.602ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "684ba905-4bd9-437f-b817-27d408b3c2ed", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43582025-04-30T06:26:32.603ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
43592025-04-30T06:26:32.605ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a01fb2a3-8ea2-40d0-b68f-5cbf233c3aba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43602025-04-30T06:26:32.605ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
43612025-04-30T06:26:32.607ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7465b823-64d0-4da3-bdc6-bbdbf36d7cdf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43622025-04-30T06:26:32.607ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
43632025-04-30T06:26:32.610ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bcbaf07-4ec8-4345-8ac4-e6b7d7da4836", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43642025-04-30T06:26:32.611ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
43652025-04-30T06:26:32.615ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "033fe6b2-d379-49b7-ba21-07e0938a96ea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43662025-04-30T06:26:32.615ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
43672025-04-30T06:26:32.617ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86004c9a-1fa4-459a-8ebd-2ed36a8ac72f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43682025-04-30T06:26:32.617ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
43692025-04-30T06:26:32.620ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efb8d0ba-baa1-417e-af93-f0efa4326230", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43702025-04-30T06:26:32.620ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
43712025-04-30T06:26:32.622ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43722025-04-30T06:26:32.622ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43732025-04-30T06:26:32.622ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43742025-04-30T06:26:32.622ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43752025-04-30T06:26:32.624ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43762025-04-30T06:26:32.624ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e9b519c-d37d-48b6-92a7-49f7974e2d14", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43772025-04-30T06:26:32.624ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
43782025-04-30T06:26:32.625ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e713958b-3402-406e-a324-76a5d70b1d2d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43792025-04-30T06:26:32.626ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
43802025-04-30T06:26:32.627ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43812025-04-30T06:26:32.633ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43822025-04-30T06:26:32.633ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43832025-04-30T06:26:32.633ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43842025-04-30T06:26:32.634ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43852025-04-30T06:26:32.634ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2926d829-c6e9-43ee-b4d0-c7ecdacae3c8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43862025-04-30T06:26:32.634ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
43872025-04-30T06:26:32.635ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d2f1f48-b31e-4793-877c-5c849e5f403d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43882025-04-30T06:26:32.636ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
43892025-04-30T06:26:32.637ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43902025-04-30T06:26:32.637ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43912025-04-30T06:26:32.645ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43922025-04-30T06:26:32.645ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42db8133-b6f1-4208-9388-680546fde4d3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43932025-04-30T06:26:32.645ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
43942025-04-30T06:26:32.650ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30b8994c-e90a-4849-bd6d-23e8a27475d6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43952025-04-30T06:26:32.651ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
43962025-04-30T06:26:32.656ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cd5d0cc-6a62-4b05-b3a7-3bef9f6ac2b6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43972025-04-30T06:26:32.656ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
43982025-04-30T06:26:32.659ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77a02903-965e-48d1-b7ec-15ab75181552", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
43992025-04-30T06:26:32.659ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
44002025-04-30T06:26:32.663ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4827c69-e663-4c79-8397-92b063e4386c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44012025-04-30T06:26:32.664ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
44022025-04-30T06:26:32.669ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a67bace5-e3d5-405e-a9ef-e7487de9c0f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44032025-04-30T06:26:32.669ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
44042025-04-30T06:26:32.675ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbdca2c9-a946-4986-a21a-002052531e3d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44052025-04-30T06:26:32.675ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
44062025-04-30T06:26:32.681ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
44072025-04-30T06:26:32.681ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21a4d049-bc25-4195-a475-0a1c20e270e8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44082025-04-30T06:26:32.682ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
44092025-04-30T06:26:32.685ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58d95767-578f-4923-b829-26198c592f94", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44102025-04-30T06:26:32.685ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
44112025-04-30T06:26:32.686ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31aacdd5-76f3-4eaa-8c8b-565c954cd6fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44122025-04-30T06:26:32.686ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
44132025-04-30T06:26:32.690ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b073776e-64fd-4554-816e-5ecaa27e6991", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44142025-04-30T06:26:32.690ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44152025-04-30T06:26:32.690ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44162025-04-30T06:26:32.690ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44172025-04-30T06:26:32.690ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44182025-04-30T06:26:32.691ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44192025-04-30T06:26:32.695ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44202025-04-30T06:26:32.699ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44212025-04-30T06:26:32.699ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44222025-04-30T06:26:32.699ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44232025-04-30T06:26:32.699ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44242025-04-30T06:26:32.699ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44252025-04-30T06:26:32.700ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44262025-04-30T06:26:32.702ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44272025-04-30T06:26:32.709ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44282025-04-30T06:26:32.709ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44292025-04-30T06:26:32.710ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44302025-04-30T06:26:32.712ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44312025-04-30T06:26:32.786ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
44322025-04-30T06:26:32.786ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
44332025-04-30T06:26:32.786ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44342025-04-30T06:26:32.786ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44352025-04-30T06:26:32.786ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44362025-04-30T06:26:32.786ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
44372025-04-30T06:26:32.788ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44382025-04-30T06:26:32.788ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:56284 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44392025-04-30T06:26:32.789ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:56284 req_id = 9c1b5fbc-51f3-4b4f-a1dd-e6fc47b543c9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44402025-04-30T06:26:32.790ZINFOtest_disk_metrics (dropshot): request completed latency_us = 465 local_addr = [::1]:50989 method = GET remote_addr = [::1]:56284 req_id = 9c1b5fbc-51f3-4b4f-a1dd-e6fc47b543c9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44412025-04-30T06:26:32.790ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c1b5fbc-51f3-4b4f-a1dd-e6fc47b543c9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
44422025-04-30T06:26:32.792ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44432025-04-30T06:26:32.792ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44442025-04-30T06:26:32.792ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
44452025-04-30T06:26:32.794ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44462025-04-30T06:26:32.794ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44472025-04-30T06:26:32.794ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44482025-04-30T06:26:32.794ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44492025-04-30T06:26:32.795ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44502025-04-30T06:26:32.795ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44512025-04-30T06:26:32.800ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44522025-04-30T06:26:32.800ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44532025-04-30T06:26:32.800ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
44542025-04-30T06:26:32.801ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44552025-04-30T06:26:32.803ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44562025-04-30T06:26:32.807ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44572025-04-30T06:26:32.807ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44582025-04-30T06:26:32.807ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44592025-04-30T06:26:32.808ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44602025-04-30T06:26:32.810ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44612025-04-30T06:26:32.810ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
44622025-04-30T06:26:32.811ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44632025-04-30T06:26:32.846ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
44642025-04-30T06:26:32.846ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
44652025-04-30T06:26:32.846ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 411.456169ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
44662025-04-30T06:26:32.846ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
44672025-04-30T06:26:32.846ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44682025-04-30T06:26:32.846ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44692025-04-30T06:26:32.846ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44702025-04-30T06:26:32.846ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44712025-04-30T06:26:32.848ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44722025-04-30T06:26:32.857ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
44732025-04-30T06:26:32.858ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44742025-04-30T06:26:32.870ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44752025-04-30T06:26:32.870ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44762025-04-30T06:26:32.870ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44772025-04-30T06:26:32.870ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44782025-04-30T06:26:32.871ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
44792025-04-30T06:26:32.872ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44802025-04-30T06:26:32.872ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = d6dbd7a4-fe5d-4515-85a5-f7b57a9fe64f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44812025-04-30T06:26:32.872ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = d6dbd7a4-fe5d-4515-85a5-f7b57a9fe64f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44822025-04-30T06:26:32.872ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = d6dbd7a4-fe5d-4515-85a5-f7b57a9fe64f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44832025-04-30T06:26:32.872ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = d6dbd7a4-fe5d-4515-85a5-f7b57a9fe64f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44842025-04-30T06:26:32.884ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
44852025-04-30T06:26:32.885ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
44862025-04-30T06:26:32.886ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ad 4d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .M..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51625 req_id = 4360763c-7610-47d7-b33e-c9fb8ad18de3
44872025-04-30T06:26:32.886ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44365,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51625 req_id = 4360763c-7610-47d7-b33e-c9fb8ad18de3
44882025-04-30T06:26:32.886ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
44892025-04-30T06:26:32.886ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44902025-04-30T06:26:32.886ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51625 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4360763c-7610-47d7-b33e-c9fb8ad18de3
44912025-04-30T06:26:32.888ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 16301 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = d6dbd7a4-fe5d-4515-85a5-f7b57a9fe64f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44922025-04-30T06:26:32.888ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3257, tv_nsec: 687630336 } })
44932025-04-30T06:26:32.888ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
44942025-04-30T06:26:32.888ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6dbd7a4-fe5d-4515-85a5-f7b57a9fe64f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
44952025-04-30T06:26:32.890ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44962025-04-30T06:26:32.891ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = ec57c7e1-7ad3-4bbe-a920-20d9fe6ae49a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44972025-04-30T06:26:32.891ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = ec57c7e1-7ad3-4bbe-a920-20d9fe6ae49a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44982025-04-30T06:26:32.891ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = ec57c7e1-7ad3-4bbe-a920-20d9fe6ae49a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44992025-04-30T06:26:32.893ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = ec57c7e1-7ad3-4bbe-a920-20d9fe6ae49a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45002025-04-30T06:26:32.893ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a5 f9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65203 req_id = cb41c756-6069-4ee8-aaa1-fb5dcbc7d484
45012025-04-30T06:26:32.893ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42489,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65203 req_id = cb41c756-6069-4ee8-aaa1-fb5dcbc7d484
45022025-04-30T06:26:32.894ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45032025-04-30T06:26:32.894ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65203 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cb41c756-6069-4ee8-aaa1-fb5dcbc7d484
45042025-04-30T06:26:32.975ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45052025-04-30T06:26:32.976ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 85365 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = ec57c7e1-7ad3-4bbe-a920-20d9fe6ae49a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45062025-04-30T06:26:32.977ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45072025-04-30T06:26:32.977ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45082025-04-30T06:26:32.977ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45092025-04-30T06:26:32.977ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45102025-04-30T06:26:32.979ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45112025-04-30T06:26:32.980ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec57c7e1-7ad3-4bbe-a920-20d9fe6ae49a", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45122025-04-30T06:26:32.980ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
45132025-04-30T06:26:32.980ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45142025-04-30T06:26:32.980ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
45152025-04-30T06:26:32.982ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d7 5a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .Z..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56976 req_id = cb04a1e9-2aea-41b7-916f-661213251981
45162025-04-30T06:26:32.983ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55130,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56976 req_id = cb04a1e9-2aea-41b7-916f-661213251981
45172025-04-30T06:26:32.983ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
45182025-04-30T06:26:32.983ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55130, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56976 req_id = cb04a1e9-2aea-41b7-916f-661213251981
45192025-04-30T06:26:32.984ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
45202025-04-30T06:26:32.984ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
45212025-04-30T06:26:32.985ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
45222025-04-30T06:26:32.985ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45232025-04-30T06:26:32.994ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45242025-04-30T06:26:32.996ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45252025-04-30T06:26:32.998ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45262025-04-30T06:26:32.998ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45272025-04-30T06:26:33.000ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45282025-04-30T06:26:33.001ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45292025-04-30T06:26:33.007ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45302025-04-30T06:26:33.008ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45312025-04-30T06:26:33.008ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45322025-04-30T06:26:33.008ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45332025-04-30T06:26:33.010ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45342025-04-30T06:26:33.010ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45352025-04-30T06:26:33.011ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45362025-04-30T06:26:33.016ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea4dda29-9665-4af7-840c-fdde8d3bc8e5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45372025-04-30T06:26:33.016ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
45382025-04-30T06:26:33.031ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "505a4659-d420-47d2-bbf4-48dee91ee91e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45392025-04-30T06:26:33.031ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
45402025-04-30T06:26:33.040ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a3f9c6b-402e-4afe-b0df-c46598afffd8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45412025-04-30T06:26:33.040ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
45422025-04-30T06:26:33.047ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "452e0f52-0fad-47e7-bfbf-66e5283536eb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45432025-04-30T06:26:33.048ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
45442025-04-30T06:26:33.053ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "800e9a03-41a1-4c63-9abe-1de136eaa25c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45452025-04-30T06:26:33.054ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
45462025-04-30T06:26:33.060ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
45472025-04-30T06:26:33.060ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d12654c-a834-42aa-8788-73e8e7a29dd9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45482025-04-30T06:26:33.060ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
45492025-04-30T06:26:33.066ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a26218c3-cf53-433d-9879-2cf4fbe54297", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45502025-04-30T06:26:33.066ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
45512025-04-30T06:26:33.074ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03ca8c85-e655-4458-bee0-a60042941c72", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45522025-04-30T06:26:33.074ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
45532025-04-30T06:26:33.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01b21e44-dcc2-4299-bb84-b5ebcd813c31", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45542025-04-30T06:26:33.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
45552025-04-30T06:26:33.077ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29fc1ce7-2e38-4615-9b30-64251b38f5ca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45562025-04-30T06:26:33.077ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
45572025-04-30T06:26:33.080ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61c5b1c1-7f35-4649-8c56-e3ddccedaa9f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45582025-04-30T06:26:33.080ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
45592025-04-30T06:26:33.081ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c24c9b10-cd1e-4d9e-afb9-7068865ae745", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45602025-04-30T06:26:33.081ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
45612025-04-30T06:26:33.086ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a4d6cdf-83ec-410f-b40d-9aa89e12de89", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45622025-04-30T06:26:33.086ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
45632025-04-30T06:26:33.094ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07a13f35-abda-4da3-8b6c-1d21fe948a52", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45642025-04-30T06:26:33.094ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
45652025-04-30T06:26:33.097ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41f7fe23-27ce-4695-9a0b-ba8ff88da34d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45662025-04-30T06:26:33.097ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
45672025-04-30T06:26:33.099ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f346a68d-8602-4b23-8b91-9cf7b73ff968", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45682025-04-30T06:26:33.099ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
45692025-04-30T06:26:33.101ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d463432c-b4cb-4c48-8e2d-6a0c8f4238e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45702025-04-30T06:26:33.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
45712025-04-30T06:26:33.103ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed011342-df90-4f80-9c69-7f52ecd99e37", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45722025-04-30T06:26:33.104ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
45732025-04-30T06:26:33.105ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ec09154-5426-43aa-a319-8dd871165c6f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45742025-04-30T06:26:33.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
45752025-04-30T06:26:33.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb8489cb-398a-4c7b-8884-8f20a738585f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45762025-04-30T06:26:33.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
45772025-04-30T06:26:33.108ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d85155a-9ada-4e0f-8c1d-33f17f69b26f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45782025-04-30T06:26:33.108ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
45792025-04-30T06:26:33.109ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb62f0f9-a72e-48aa-8108-012d9269dae8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45802025-04-30T06:26:33.109ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
45812025-04-30T06:26:33.110ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f02ce224-1245-44e8-a85d-4e0dece203b3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45822025-04-30T06:26:33.111ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
45832025-04-30T06:26:33.111ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45842025-04-30T06:26:33.111ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45852025-04-30T06:26:33.111ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45862025-04-30T06:26:33.111ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45872025-04-30T06:26:33.112ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45882025-04-30T06:26:33.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2e63165-090e-448f-bef9-536e9e1d8f83", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45892025-04-30T06:26:33.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
45902025-04-30T06:26:33.117ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45912025-04-30T06:26:33.123ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45922025-04-30T06:26:33.124ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45932025-04-30T06:26:33.125ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45942025-04-30T06:26:33.130ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45952025-04-30T06:26:33.130ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abb1f2d1-2a3a-4c6d-bb75-a3fd9be29727", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45962025-04-30T06:26:33.130ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
45972025-04-30T06:26:33.132ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1891f7c8-aa4b-456b-92ad-660dcbd0e577", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
45982025-04-30T06:26:33.132ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
45992025-04-30T06:26:33.134ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "663b2452-3ef4-4b9c-bd6c-ef55af4e817b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46002025-04-30T06:26:33.134ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
46012025-04-30T06:26:33.135ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46022025-04-30T06:26:33.135ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
46032025-04-30T06:26:33.135ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46042025-04-30T06:26:33.137ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2265485a-3f4a-4b46-bd6a-5dadcd331615", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46052025-04-30T06:26:33.137ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
46062025-04-30T06:26:33.140ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5d95611-dbf3-45a9-8194-eba6dc03cdde", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46072025-04-30T06:26:33.140ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
46082025-04-30T06:26:33.142ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95e83a8a-db5a-446c-b109-75cccaa276ff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46092025-04-30T06:26:33.142ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
46102025-04-30T06:26:33.143ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c3c70e-0705-48bf-9d35-fff5d754fa75", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46112025-04-30T06:26:33.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
46122025-04-30T06:26:33.145ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efd7bb20-55ad-4a81-bf19-790279ccf897", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46132025-04-30T06:26:33.145ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
46142025-04-30T06:26:33.147ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2908cb2b-f7a1-4b45-bc1a-bfb0bed4071f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46152025-04-30T06:26:33.147ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
46162025-04-30T06:26:33.148ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a27c865-95d1-4637-9af8-033af1879701", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46172025-04-30T06:26:33.149ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
46182025-04-30T06:26:33.151ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9a7030f-b85d-40bc-921e-3b3452b258e1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46192025-04-30T06:26:33.151ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
46202025-04-30T06:26:33.153ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dcdca3b-9728-4112-b7a3-a876fb4fb96a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46212025-04-30T06:26:33.193ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
46222025-04-30T06:26:33.194ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46232025-04-30T06:26:33.194ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46242025-04-30T06:26:33.194ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
46252025-04-30T06:26:33.196ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
46262025-04-30T06:26:33.198ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7c817b-a6ca-4051-a7ce-8d14c3eee819", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46272025-04-30T06:26:33.199ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
46282025-04-30T06:26:33.200ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
46292025-04-30T06:26:33.200ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
46302025-04-30T06:26:33.201ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44af0b12-775d-4072-ac52-af7e13d04077", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46312025-04-30T06:26:33.201ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
46322025-04-30T06:26:33.203ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
46332025-04-30T06:26:33.203ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46342025-04-30T06:26:33.204ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a399a5fc-4eb6-4f72-bafd-42d842fb751f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46352025-04-30T06:26:33.204ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
46362025-04-30T06:26:33.211ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
46372025-04-30T06:26:33.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04e10930-858d-4339-953a-e953b854bdeb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46382025-04-30T06:26:33.212ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
46392025-04-30T06:26:33.214ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd4060c2-ea83-462f-a876-40cf53d6cd25", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46402025-04-30T06:26:33.214ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
46412025-04-30T06:26:33.215ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "260cb8dd-ef7c-4fe6-aa1f-3b8190aa2519", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46422025-04-30T06:26:33.216ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
46432025-04-30T06:26:33.217ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5af5e71-72c0-44b4-a54d-ab7cd0b2928a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46442025-04-30T06:26:33.217ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
46452025-04-30T06:26:33.218ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78d1cc73-1c3d-4683-92e1-5d17cbed4c6a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46462025-04-30T06:26:33.218ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
46472025-04-30T06:26:33.219ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c3295a4-fbbb-4f5e-869a-489aa7270d87", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46482025-04-30T06:26:33.219ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46492025-04-30T06:26:33.219ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46502025-04-30T06:26:33.219ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46512025-04-30T06:26:33.219ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46522025-04-30T06:26:33.220ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46532025-04-30T06:26:33.223ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46542025-04-30T06:26:33.230ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46552025-04-30T06:26:33.230ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46562025-04-30T06:26:33.230ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46572025-04-30T06:26:33.230ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46582025-04-30T06:26:33.230ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46592025-04-30T06:26:33.232ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46602025-04-30T06:26:33.248ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46612025-04-30T06:26:33.256ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46622025-04-30T06:26:33.256ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46632025-04-30T06:26:33.256ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46642025-04-30T06:26:33.258ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46652025-04-30T06:26:33.275ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46662025-04-30T06:26:33.280ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46672025-04-30T06:26:33.280ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46682025-04-30T06:26:33.280ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
46692025-04-30T06:26:33.281ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46702025-04-30T06:26:33.336ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
46712025-04-30T06:26:33.336ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
46722025-04-30T06:26:33.337ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:42552 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46732025-04-30T06:26:33.338ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
46742025-04-30T06:26:33.338ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:42552 req_id = ad74f37c-5acd-44a7-acc9-f00207fcd14a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46752025-04-30T06:26:33.339ZINFOtest_disk_metrics (dropshot): request completed latency_us = 510 local_addr = [::1]:50989 method = GET remote_addr = [::1]:42552 req_id = ad74f37c-5acd-44a7-acc9-f00207fcd14a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46762025-04-30T06:26:33.339ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46772025-04-30T06:26:33.344ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46782025-04-30T06:26:33.345ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46792025-04-30T06:26:33.345ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
46802025-04-30T06:26:33.345ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46812025-04-30T06:26:33.346ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad74f37c-5acd-44a7-acc9-f00207fcd14a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
46822025-04-30T06:26:33.347ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46832025-04-30T06:26:33.347ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46842025-04-30T06:26:33.348ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
46852025-04-30T06:26:33.350ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46862025-04-30T06:26:33.350ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46872025-04-30T06:26:33.350ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46882025-04-30T06:26:33.350ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46892025-04-30T06:26:33.351ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46902025-04-30T06:26:33.354ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46912025-04-30T06:26:33.354ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
46922025-04-30T06:26:33.355ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46932025-04-30T06:26:33.355ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46942025-04-30T06:26:33.360ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46952025-04-30T06:26:33.360ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46962025-04-30T06:26:33.360ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46972025-04-30T06:26:33.361ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46982025-04-30T06:26:33.402ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
46992025-04-30T06:26:33.402ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
47002025-04-30T06:26:33.402ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 555.841649ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
47012025-04-30T06:26:33.402ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
47022025-04-30T06:26:33.402ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47032025-04-30T06:26:33.402ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47042025-04-30T06:26:33.402ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47052025-04-30T06:26:33.402ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47062025-04-30T06:26:33.403ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47072025-04-30T06:26:33.404ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
47082025-04-30T06:26:33.406ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47092025-04-30T06:26:33.410ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47102025-04-30T06:26:33.410ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47112025-04-30T06:26:33.410ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47122025-04-30T06:26:33.411ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47132025-04-30T06:26:33.415ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
47142025-04-30T06:26:33.415ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47152025-04-30T06:26:33.417ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 27 ac 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 '............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56597 req_id = c55bc8df-a1db-496d-a499-532266a12b18
47162025-04-30T06:26:33.417ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10156,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56597 req_id = c55bc8df-a1db-496d-a499-532266a12b18
47172025-04-30T06:26:33.418ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
47182025-04-30T06:26:33.418ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47192025-04-30T06:26:33.418ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56597 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c55bc8df-a1db-496d-a499-532266a12b18
47202025-04-30T06:26:33.419ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3258, tv_nsec: 217894553 } })
47212025-04-30T06:26:33.419ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
47222025-04-30T06:26:33.420ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5a 7d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Z}..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54990 req_id = a82924f4-59c3-4914-b187-df34e03d553f
47232025-04-30T06:26:33.420ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23165,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54990 req_id = a82924f4-59c3-4914-b187-df34e03d553f
47242025-04-30T06:26:33.420ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47252025-04-30T06:26:33.420ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54990 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a82924f4-59c3-4914-b187-df34e03d553f
47262025-04-30T06:26:33.470ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47272025-04-30T06:26:33.471ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47282025-04-30T06:26:33.471ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47292025-04-30T06:26:33.471ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47302025-04-30T06:26:33.471ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
47312025-04-30T06:26:33.472ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47322025-04-30T06:26:33.474ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d0 ee 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55120 req_id = 96fc2d02-0694-4e2f-8746-e7bdec5f087f
47332025-04-30T06:26:33.474ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53486,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55120 req_id = 96fc2d02-0694-4e2f-8746-e7bdec5f087f
47342025-04-30T06:26:33.475ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
47352025-04-30T06:26:33.475ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53486, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55120 req_id = 96fc2d02-0694-4e2f-8746-e7bdec5f087f
47362025-04-30T06:26:33.476ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
47372025-04-30T06:26:33.476ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
47382025-04-30T06:26:33.476ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
47392025-04-30T06:26:33.477ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47402025-04-30T06:26:33.483ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47412025-04-30T06:26:33.483ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47422025-04-30T06:26:33.483ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
47432025-04-30T06:26:33.484ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47442025-04-30T06:26:33.486ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47452025-04-30T06:26:33.486ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
47462025-04-30T06:26:33.487ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47472025-04-30T06:26:33.488ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "aafa5921-11ec-484e-a6ef-ff4d0105cc7c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47482025-04-30T06:26:33.488ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
47492025-04-30T06:26:33.493ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c36fb0f7-cdfc-4358-a387-65298002bdb2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47502025-04-30T06:26:33.493ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
47512025-04-30T06:26:33.497ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f55a2b5-4b92-4590-9064-bb92799b9ca4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47522025-04-30T06:26:33.497ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
47532025-04-30T06:26:33.502ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "213c9be8-c11f-4936-9b6f-aa06032d450b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47542025-04-30T06:26:33.502ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
47552025-04-30T06:26:33.506ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fbac8a2-194c-4ff5-a4fb-d5362147cedd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47562025-04-30T06:26:33.506ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
47572025-04-30T06:26:33.511ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04a37064-fc37-4d90-bb2c-fd9f5eb7c98e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47582025-04-30T06:26:33.511ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
47592025-04-30T06:26:33.516ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0452e08b-1b0d-4172-9104-7cb0f17b166f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47602025-04-30T06:26:33.517ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
47612025-04-30T06:26:33.521ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
47622025-04-30T06:26:33.522ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2af5b56-5ba6-49dd-b145-fcac264a5720", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47632025-04-30T06:26:33.522ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
47642025-04-30T06:26:33.524ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32ea69bc-9b4e-4088-b3f6-a92a05f5e50a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47652025-04-30T06:26:33.524ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
47662025-04-30T06:26:33.525ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e3f629b-6644-445e-8612-93c6e3bc3e99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47672025-04-30T06:26:33.525ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
47682025-04-30T06:26:33.527ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a68fcbbc-996a-47bb-800f-445f44aec377", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47692025-04-30T06:26:33.527ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
47702025-04-30T06:26:33.528ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c3b6ec1-f856-4f8f-a709-33703eaeafd3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47712025-04-30T06:26:33.529ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
47722025-04-30T06:26:33.531ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4645a37e-a586-4e1a-9589-99f151eb265e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47732025-04-30T06:26:33.531ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
47742025-04-30T06:26:33.534ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e77c9ba9-8d77-4255-a736-2dc1e3bee550", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47752025-04-30T06:26:33.534ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
47762025-04-30T06:26:33.539ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d756fc36-9a08-4d36-8049-d677846e73b1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47772025-04-30T06:26:33.539ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
47782025-04-30T06:26:33.542ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55bd4f86-15c6-4732-9d6d-39c865caf042", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47792025-04-30T06:26:33.542ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
47802025-04-30T06:26:33.545ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24e41d33-50ac-4b65-b029-2c009b732095", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47812025-04-30T06:26:33.545ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
47822025-04-30T06:26:33.549ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ea523dd-2226-407d-b938-85d9decc5a3d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47832025-04-30T06:26:33.550ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
47842025-04-30T06:26:33.552ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "266262bc-a8f3-4f05-babd-656cc62aa890", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47852025-04-30T06:26:33.552ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
47862025-04-30T06:26:33.553ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d55bfe3d-ac5e-47d2-be8f-06f48a934ecc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47872025-04-30T06:26:33.553ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
47882025-04-30T06:26:33.554ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f12b5b2f-ccce-487a-b9bf-9f641cf06e40", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47892025-04-30T06:26:33.554ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
47902025-04-30T06:26:33.555ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b297416-fe91-41a2-a3ab-aece69191246", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47912025-04-30T06:26:33.555ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
47922025-04-30T06:26:33.557ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83c0ebdf-f6bd-4c79-b06e-030304102e0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47932025-04-30T06:26:33.557ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
47942025-04-30T06:26:33.559ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b859c905-203a-4e88-a878-45f7ee5a6ca4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47952025-04-30T06:26:33.559ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
47962025-04-30T06:26:33.561ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a31d5f03-d1a7-4b73-b5df-a9429b1d727f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47972025-04-30T06:26:33.561ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
47982025-04-30T06:26:33.563ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a171f1b-68fc-4298-9f01-4b80c45b72e4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
47992025-04-30T06:26:33.563ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
48002025-04-30T06:26:33.565ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92cf6088-613d-430e-a2d5-8755faf165c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48012025-04-30T06:26:33.565ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
48022025-04-30T06:26:33.567ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c0afc56-7f10-47c6-a5a3-4ac940a0568b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48032025-04-30T06:26:33.567ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
48042025-04-30T06:26:33.570ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cddafee-5476-4489-8618-fe4481d7cc43", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48052025-04-30T06:26:33.570ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
48062025-04-30T06:26:33.572ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48072025-04-30T06:26:33.572ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48082025-04-30T06:26:33.572ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48092025-04-30T06:26:33.572ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48102025-04-30T06:26:33.573ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48112025-04-30T06:26:33.575ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab576c5f-ec21-49a6-ba58-f1f269485a66", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48122025-04-30T06:26:33.575ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
48132025-04-30T06:26:33.576ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48142025-04-30T06:26:33.582ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48152025-04-30T06:26:33.582ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48162025-04-30T06:26:33.582ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48172025-04-30T06:26:33.582ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48182025-04-30T06:26:33.584ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b264c6e3-356e-472f-b954-966aa1717086", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48192025-04-30T06:26:33.584ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
48202025-04-30T06:26:33.585ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48212025-04-30T06:26:33.585ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48222025-04-30T06:26:33.585ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48232025-04-30T06:26:33.586ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce73a264-6ee6-4edd-8964-2c26d554c01e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48242025-04-30T06:26:33.586ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
48252025-04-30T06:26:33.588ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8c4d983-df1d-497c-b289-02fa7d891110", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48262025-04-30T06:26:33.588ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
48272025-04-30T06:26:33.589ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a0b552e-5e44-4ec8-8ddd-e5ed39127d2e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48282025-04-30T06:26:33.590ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
48292025-04-30T06:26:33.591ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fb8bf81-0408-45e8-b583-739bb5c16c07", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48302025-04-30T06:26:33.591ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
48312025-04-30T06:26:33.594ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "345ec2b8-38c3-49fa-a0c4-eabc6a17b837", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48322025-04-30T06:26:33.594ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
48332025-04-30T06:26:33.598ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cfbe502-6fe2-439b-b4f3-c0bc423953b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48342025-04-30T06:26:33.598ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
48352025-04-30T06:26:33.602ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52143bfe-74a1-47e8-8d2a-d743dac51af4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48362025-04-30T06:26:33.602ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
48372025-04-30T06:26:33.605ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bc1064f-f1b0-4fd0-a7a9-8e811debf6b5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48382025-04-30T06:26:33.605ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
48392025-04-30T06:26:33.610ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8945eb71-ee57-4f61-9b68-4f80cdb03d6f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48402025-04-30T06:26:33.610ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
48412025-04-30T06:26:33.614ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f38fb50-bd56-4699-b256-bf6bfcdc1079", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48422025-04-30T06:26:33.614ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
48432025-04-30T06:26:33.615ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec586ab-9e39-45ab-98d9-c327363739e9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48442025-04-30T06:26:33.615ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
48452025-04-30T06:26:33.626ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
48462025-04-30T06:26:33.627ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b56ced4d-436b-4010-9f3a-0e0c95016b9f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48472025-04-30T06:26:33.627ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
48482025-04-30T06:26:33.628ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b0b9bc1-397a-4c65-8eec-36ff1edd82ab", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48492025-04-30T06:26:33.628ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
48502025-04-30T06:26:33.630ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49b3d5f5-88ca-4807-ae1a-1a47b953ea43", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48512025-04-30T06:26:33.630ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48522025-04-30T06:26:33.630ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48532025-04-30T06:26:33.630ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48542025-04-30T06:26:33.630ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48552025-04-30T06:26:33.630ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48562025-04-30T06:26:33.636ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48572025-04-30T06:26:33.642ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48582025-04-30T06:26:33.642ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48592025-04-30T06:26:33.642ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48602025-04-30T06:26:33.642ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48612025-04-30T06:26:33.642ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48622025-04-30T06:26:33.643ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48632025-04-30T06:26:33.647ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48642025-04-30T06:26:33.652ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48652025-04-30T06:26:33.652ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48662025-04-30T06:26:33.652ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48672025-04-30T06:26:33.653ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48682025-04-30T06:26:33.679ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48692025-04-30T06:26:33.679ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48702025-04-30T06:26:33.679ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48712025-04-30T06:26:33.679ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48722025-04-30T06:26:33.680ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48732025-04-30T06:26:33.750ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
48742025-04-30T06:26:33.750ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
48752025-04-30T06:26:33.751ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:40480 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
48762025-04-30T06:26:33.752ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:40480 req_id = 942d9954-f5e6-4335-8f9e-3554755f48ec sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48772025-04-30T06:26:33.752ZINFOtest_disk_metrics (dropshot): request completed latency_us = 744 local_addr = [::1]:50989 method = GET remote_addr = [::1]:40480 req_id = 942d9954-f5e6-4335-8f9e-3554755f48ec response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48782025-04-30T06:26:33.753ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48792025-04-30T06:26:33.760ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48802025-04-30T06:26:33.760ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48812025-04-30T06:26:33.760ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48822025-04-30T06:26:33.760ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48832025-04-30T06:26:33.761ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "942d9954-f5e6-4335-8f9e-3554755f48ec", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
48842025-04-30T06:26:33.763ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48852025-04-30T06:26:33.763ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48862025-04-30T06:26:33.763ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
48872025-04-30T06:26:33.765ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
48882025-04-30T06:26:33.766ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48892025-04-30T06:26:33.766ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48902025-04-30T06:26:33.766ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48912025-04-30T06:26:33.766ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48922025-04-30T06:26:33.769ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48932025-04-30T06:26:33.769ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48942025-04-30T06:26:33.770ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48952025-04-30T06:26:33.771ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48962025-04-30T06:26:33.776ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
48972025-04-30T06:26:33.776ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48982025-04-30T06:26:33.776ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48992025-04-30T06:26:33.776ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49002025-04-30T06:26:33.813ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
49012025-04-30T06:26:33.814ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49022025-04-30T06:26:33.815ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49032025-04-30T06:26:33.815ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 412.737483ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49042025-04-30T06:26:33.815ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49052025-04-30T06:26:33.815ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49062025-04-30T06:26:33.815ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49072025-04-30T06:26:33.815ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49082025-04-30T06:26:33.815ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49092025-04-30T06:26:33.824ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49102025-04-30T06:26:33.829ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49112025-04-30T06:26:33.837ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49122025-04-30T06:26:33.837ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49132025-04-30T06:26:33.838ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49142025-04-30T06:26:33.838ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49152025-04-30T06:26:33.843ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
49162025-04-30T06:26:33.843ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49172025-04-30T06:26:33.844ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e8 2d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .-..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61879 req_id = f145a044-286c-4191-9b05-2af6588becd2
49182025-04-30T06:26:33.844ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59437,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61879 req_id = f145a044-286c-4191-9b05-2af6588becd2
49192025-04-30T06:26:33.845ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
49202025-04-30T06:26:33.845ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49212025-04-30T06:26:33.846ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61879 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f145a044-286c-4191-9b05-2af6588becd2
49222025-04-30T06:26:33.846ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3258, tv_nsec: 645888664 } })
49232025-04-30T06:26:33.847ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
49242025-04-30T06:26:33.847ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 d5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63163 req_id = 352f746c-1966-4881-b4b3-f4261b97d493
49252025-04-30T06:26:33.850ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26069,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63163 req_id = 352f746c-1966-4881-b4b3-f4261b97d493
49262025-04-30T06:26:33.850ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49272025-04-30T06:26:33.850ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63163 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 352f746c-1966-4881-b4b3-f4261b97d493
49282025-04-30T06:26:33.948ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49292025-04-30T06:26:33.949ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49302025-04-30T06:26:33.949ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49312025-04-30T06:26:33.949ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49322025-04-30T06:26:33.949ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
49332025-04-30T06:26:33.950ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49342025-04-30T06:26:33.957ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 56 f7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c V............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50729 req_id = 9038eed6-1c1e-48cd-928a-abcdebfd6d0a
49352025-04-30T06:26:33.958ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22263,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50729 req_id = 9038eed6-1c1e-48cd-928a-abcdebfd6d0a
49362025-04-30T06:26:33.958ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
49372025-04-30T06:26:33.958ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22263, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50729 req_id = 9038eed6-1c1e-48cd-928a-abcdebfd6d0a
49382025-04-30T06:26:33.959ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
49392025-04-30T06:26:33.959ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
49402025-04-30T06:26:33.959ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
49412025-04-30T06:26:33.961ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49422025-04-30T06:26:33.968ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49432025-04-30T06:26:33.968ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49442025-04-30T06:26:33.969ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
49452025-04-30T06:26:33.969ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49462025-04-30T06:26:33.982ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49472025-04-30T06:26:33.982ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
49482025-04-30T06:26:33.982ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49492025-04-30T06:26:33.988ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49502025-04-30T06:26:33.988ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e0dd24c-2cbd-4658-ac92-1a3d78fd8173", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49512025-04-30T06:26:33.988ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
49522025-04-30T06:26:33.988ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49532025-04-30T06:26:33.994ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49542025-04-30T06:26:33.995ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49552025-04-30T06:26:33.995ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49562025-04-30T06:26:33.995ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49572025-04-30T06:26:34.011ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ea570c-8fb5-4580-a245-a94c4fe3a7fb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49582025-04-30T06:26:34.011ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
49592025-04-30T06:26:34.025ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c96f979-be0d-4ca0-9d9b-155f41cd4692", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49602025-04-30T06:26:34.025ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
49612025-04-30T06:26:34.028ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45f6d0c7-36ff-4c06-afde-04f40ad0a9fa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49622025-04-30T06:26:34.036ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
49632025-04-30T06:26:34.040ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b2360e-8031-4434-a6d8-190f9b9f86f5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49642025-04-30T06:26:34.040ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
49652025-04-30T06:26:34.065ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
49662025-04-30T06:26:34.066ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e2e7a1c-0299-4ced-8dac-45c4335996fa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49672025-04-30T06:26:34.067ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
49682025-04-30T06:26:34.069ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc6787c3-9f4d-4685-803d-aaa227a03452", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49692025-04-30T06:26:34.069ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
49702025-04-30T06:26:34.071ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c658c84-6bce-4630-8f14-c7156a1275c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49712025-04-30T06:26:34.072ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
49722025-04-30T06:26:34.073ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "067160d9-ba0f-48d5-87fb-d7289b4808a9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49732025-04-30T06:26:34.073ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
49742025-04-30T06:26:34.074ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56288a2f-ca30-4ccd-a243-77a0f9c986e7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49752025-04-30T06:26:34.074ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
49762025-04-30T06:26:34.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95424165-1968-4717-b3ba-f778c77e7ea4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49772025-04-30T06:26:34.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
49782025-04-30T06:26:34.076ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2548aa59-b5b7-40a0-867c-af10fce9fd34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49792025-04-30T06:26:34.076ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
49802025-04-30T06:26:34.077ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f898bef-acf4-467b-ba81-26514dd9df39", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49812025-04-30T06:26:34.078ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
49822025-04-30T06:26:34.079ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c856f3d0-92c6-4fff-b5fb-10b1c8c9995e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49832025-04-30T06:26:34.079ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
49842025-04-30T06:26:34.082ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b23a05-356c-40ea-836d-9916f6e7bbd7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49852025-04-30T06:26:34.082ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
49862025-04-30T06:26:34.084ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd242520-fe96-49be-8ad3-fcde120b2b32", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49872025-04-30T06:26:34.084ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
49882025-04-30T06:26:34.087ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fcff39e-6d2e-43c0-9ec9-cbcd1f794fcb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49892025-04-30T06:26:34.088ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
49902025-04-30T06:26:34.091ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af8f2dcf-7744-4677-92c1-3594b2666455", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49912025-04-30T06:26:34.091ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
49922025-04-30T06:26:34.093ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4953b7b-a099-482b-8835-4f217ecc3f86", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49932025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
49942025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ad56e40-d579-4c4a-a712-a39050a6cda3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49952025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
49962025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad3d1c2-2b88-4786-b688-198d73bd389d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49972025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
49982025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63c8d606-24ed-42e2-acc3-86fb787770f3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
49992025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
50002025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "00870485-8520-46ce-980b-145f2100fa09", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50012025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
50022025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92df7c56-e294-4880-9fd9-076db52c025e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50032025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
50042025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7e37eb5-44f8-4a6e-9503-52bd0a40b843", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50052025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
50062025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56de5637-38b5-42e1-9eac-e186e71fcc99", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50072025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
50082025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2233da38-7281-4458-9bcb-a7b7494dd26c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50092025-04-30T06:26:34.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
50102025-04-30T06:26:34.109ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83b1c24a-bd46-4bff-854c-3bcfbf210a1a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50112025-04-30T06:26:34.109ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
50122025-04-30T06:26:34.111ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93bb9e66-85b4-4cfc-965d-412cb11dfd4a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50132025-04-30T06:26:34.112ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
50142025-04-30T06:26:34.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a45230bd-bc0c-4726-9e4a-c8e0230c4366", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50152025-04-30T06:26:34.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
50162025-04-30T06:26:34.115ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50172025-04-30T06:26:34.115ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50182025-04-30T06:26:34.115ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50192025-04-30T06:26:34.115ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50202025-04-30T06:26:34.117ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50212025-04-30T06:26:34.118ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a12a14e9-0de8-4e18-b7a8-014442dd69e0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50222025-04-30T06:26:34.119ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
50232025-04-30T06:26:34.120ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42dc0d07-7425-436a-bda4-7cba2ce2a37a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50242025-04-30T06:26:34.121ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
50252025-04-30T06:26:34.121ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50262025-04-30T06:26:34.133ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50272025-04-30T06:26:34.133ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50282025-04-30T06:26:34.133ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50292025-04-30T06:26:34.134ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50302025-04-30T06:26:34.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bb6207c-26a5-4b13-8c2b-cc83b4403487", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50312025-04-30T06:26:34.142ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
50322025-04-30T06:26:34.142ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50332025-04-30T06:26:34.142ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50342025-04-30T06:26:34.143ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50352025-04-30T06:26:34.145ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ae4eb9d-8f27-40ab-b96d-b27339b1531d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50362025-04-30T06:26:34.145ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
50372025-04-30T06:26:34.150ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8abc9b04-22ba-43b2-971f-3424cbc16034", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50382025-04-30T06:26:34.150ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
50392025-04-30T06:26:34.153ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "430b7d00-dec1-4857-84f6-a9997204b1ba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50402025-04-30T06:26:34.153ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
50412025-04-30T06:26:34.155ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c1c4a2c-ff94-4a90-b42c-7a4cd1e64c0f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50422025-04-30T06:26:34.155ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
50432025-04-30T06:26:34.159ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f1b4ab1-f540-4b97-a79c-47f7c16565d4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50442025-04-30T06:26:34.160ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
50452025-04-30T06:26:34.165ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77dfc015-562b-4858-b13d-c319c7d76ba8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50462025-04-30T06:26:34.165ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
50472025-04-30T06:26:34.167ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98155a89-876a-40fb-a541-7376431eaa81", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50482025-04-30T06:26:34.167ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
50492025-04-30T06:26:34.170ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7390f7fa-e91f-415b-b823-fb7a636c1498", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50502025-04-30T06:26:34.170ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
50512025-04-30T06:26:34.171ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad6d682-69b1-4f93-ba4f-1917f2382593", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50522025-04-30T06:26:34.171ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
50532025-04-30T06:26:34.173ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e30b270-4493-42e2-aca4-fbe131b07c53", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50542025-04-30T06:26:34.174ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
50552025-04-30T06:26:34.175ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
50562025-04-30T06:26:34.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50572025-04-30T06:26:34.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50582025-04-30T06:26:34.175ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
50592025-04-30T06:26:34.176ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50602025-04-30T06:26:34.177ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fb985ae-89f4-407c-b979-de338bd8ccb0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50612025-04-30T06:26:34.177ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
50622025-04-30T06:26:34.177ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3a8183-461e-4743-86a1-b6f32203066b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
50632025-04-30T06:26:34.178ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50642025-04-30T06:26:34.178ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50652025-04-30T06:26:34.178ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50662025-04-30T06:26:34.178ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50672025-04-30T06:26:34.178ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50682025-04-30T06:26:34.182ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
50692025-04-30T06:26:34.182ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50702025-04-30T06:26:34.185ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50712025-04-30T06:26:34.192ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50722025-04-30T06:26:34.192ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50732025-04-30T06:26:34.192ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50742025-04-30T06:26:34.192ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50752025-04-30T06:26:34.192ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50762025-04-30T06:26:34.192ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50772025-04-30T06:26:34.193ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
50782025-04-30T06:26:34.193ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50792025-04-30T06:26:34.198ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50802025-04-30T06:26:34.203ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50812025-04-30T06:26:34.203ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50822025-04-30T06:26:34.203ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50832025-04-30T06:26:34.204ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50842025-04-30T06:26:34.331ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
50852025-04-30T06:26:34.331ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
50862025-04-30T06:26:34.332ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50872025-04-30T06:26:34.332ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50882025-04-30T06:26:34.332ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50892025-04-30T06:26:34.332ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50902025-04-30T06:26:34.333ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50912025-04-30T06:26:34.333ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:42545 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
50922025-04-30T06:26:34.334ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50932025-04-30T06:26:34.336ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:42545 req_id = 1a46c67d-4eea-4b10-a8f0-615c7f9cdf79 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50942025-04-30T06:26:34.337ZINFOtest_disk_metrics (dropshot): request completed latency_us = 579 local_addr = [::1]:50989 method = GET remote_addr = [::1]:42545 req_id = 1a46c67d-4eea-4b10-a8f0-615c7f9cdf79 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50952025-04-30T06:26:34.337ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50962025-04-30T06:26:34.344ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50972025-04-30T06:26:34.344ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50982025-04-30T06:26:34.345ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50992025-04-30T06:26:34.345ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51002025-04-30T06:26:34.346ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a46c67d-4eea-4b10-a8f0-615c7f9cdf79", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51012025-04-30T06:26:34.348ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51022025-04-30T06:26:34.348ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51032025-04-30T06:26:34.348ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
51042025-04-30T06:26:34.351ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51052025-04-30T06:26:34.351ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51062025-04-30T06:26:34.351ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51072025-04-30T06:26:34.351ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51082025-04-30T06:26:34.351ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51092025-04-30T06:26:34.354ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51102025-04-30T06:26:34.354ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51112025-04-30T06:26:34.356ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51122025-04-30T06:26:34.356ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51132025-04-30T06:26:34.360ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51142025-04-30T06:26:34.360ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51152025-04-30T06:26:34.360ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51162025-04-30T06:26:34.360ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51172025-04-30T06:26:34.407ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51182025-04-30T06:26:34.408ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51192025-04-30T06:26:34.408ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 592.813011ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51202025-04-30T06:26:34.408ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51212025-04-30T06:26:34.408ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51222025-04-30T06:26:34.408ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51232025-04-30T06:26:34.408ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51242025-04-30T06:26:34.408ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51252025-04-30T06:26:34.410ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51262025-04-30T06:26:34.429ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
51272025-04-30T06:26:34.434ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51282025-04-30T06:26:34.439ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51292025-04-30T06:26:34.439ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51302025-04-30T06:26:34.439ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51312025-04-30T06:26:34.440ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51322025-04-30T06:26:34.444ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
51332025-04-30T06:26:34.444ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51342025-04-30T06:26:34.445ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 34 05 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 4............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63658 req_id = 22fefdeb-e3f0-4fa6-9542-7ef994d5435a
51352025-04-30T06:26:34.445ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13317,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63658 req_id = 22fefdeb-e3f0-4fa6-9542-7ef994d5435a
51362025-04-30T06:26:34.445ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
51372025-04-30T06:26:34.445ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51382025-04-30T06:26:34.445ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63658 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 22fefdeb-e3f0-4fa6-9542-7ef994d5435a
51392025-04-30T06:26:34.447ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3259, tv_nsec: 245289020 } })
51402025-04-30T06:26:34.447ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
51412025-04-30T06:26:34.447ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 59 34 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Y4..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60224 req_id = 966c32e6-f53b-4fc6-a9a2-4f267bb829a0
51422025-04-30T06:26:34.447ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22836,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60224 req_id = 966c32e6-f53b-4fc6-a9a2-4f267bb829a0
51432025-04-30T06:26:34.447ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51442025-04-30T06:26:34.447ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60224 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 966c32e6-f53b-4fc6-a9a2-4f267bb829a0
51452025-04-30T06:26:34.498ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
51462025-04-30T06:26:34.499ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51472025-04-30T06:26:34.499ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51482025-04-30T06:26:34.499ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51492025-04-30T06:26:34.499ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51502025-04-30T06:26:34.507ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51512025-04-30T06:26:34.508ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d6 ea 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55007 req_id = 3fa1dd63-b677-468f-9cc8-805b858dbd59
51522025-04-30T06:26:34.508ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55018,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55007 req_id = 3fa1dd63-b677-468f-9cc8-805b858dbd59
51532025-04-30T06:26:34.508ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
51542025-04-30T06:26:34.508ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55018, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55007 req_id = 3fa1dd63-b677-468f-9cc8-805b858dbd59
51552025-04-30T06:26:34.508ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
51562025-04-30T06:26:34.508ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
51572025-04-30T06:26:34.508ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
51582025-04-30T06:26:34.510ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d48735a3-4c51-4a48-a107-0fce6ec6ffcd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51592025-04-30T06:26:34.510ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
51602025-04-30T06:26:34.510ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51612025-04-30T06:26:34.515ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51622025-04-30T06:26:34.515ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51632025-04-30T06:26:34.515ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51642025-04-30T06:26:34.516ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51652025-04-30T06:26:34.517ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b76fb09-9db6-4726-8b64-2fa359b9faf2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51662025-04-30T06:26:34.518ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
51672025-04-30T06:26:34.518ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51682025-04-30T06:26:34.518ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51692025-04-30T06:26:34.519ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51702025-04-30T06:26:34.522ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dea0778-37b4-43c0-ac5c-0f265b6fb344", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51712025-04-30T06:26:34.523ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
51722025-04-30T06:26:34.525ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aadf631c-d10e-4416-a795-d75bd0eaf3fc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51732025-04-30T06:26:34.525ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
51742025-04-30T06:26:34.529ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f593c4d-d4b8-485a-8a2e-5a9c93521ff4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51752025-04-30T06:26:34.529ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
51762025-04-30T06:26:34.531ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43b9246e-0273-47cf-b8de-dc4cd5777c66", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51772025-04-30T06:26:34.532ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
51782025-04-30T06:26:34.536ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "314cf0f4-7a3d-4704-a1ba-62fb12b4421c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51792025-04-30T06:26:34.536ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
51802025-04-30T06:26:34.540ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "980d38e2-7c58-437c-ae9f-1c3492706470", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51812025-04-30T06:26:34.540ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
51822025-04-30T06:26:34.551ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17c9784c-28d3-4037-ac50-f1f35053a44f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51832025-04-30T06:26:34.551ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
51842025-04-30T06:26:34.554ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1b48d40-f4c1-4383-9156-23ff22b4e938", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51852025-04-30T06:26:34.554ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
51862025-04-30T06:26:34.557ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "651eacc2-793f-4832-af9d-ca61661c4485", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51872025-04-30T06:26:34.557ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
51882025-04-30T06:26:34.561ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
51892025-04-30T06:26:34.563ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "552d279c-1fb9-4f5e-b54b-359925acd512", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51902025-04-30T06:26:34.564ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
51912025-04-30T06:26:34.568ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfdb9e94-1739-4952-ba09-bea3f6f7a8b0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51922025-04-30T06:26:34.568ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
51932025-04-30T06:26:34.573ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5fc3ab5-0c65-4112-aec5-36dca1d15a8b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51942025-04-30T06:26:34.574ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
51952025-04-30T06:26:34.580ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "507f2e77-c842-4d47-968c-3353b2915c52", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51962025-04-30T06:26:34.580ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
51972025-04-30T06:26:34.583ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d42002f-86d9-44df-8d80-3bfeff7bd53a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
51982025-04-30T06:26:34.583ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
51992025-04-30T06:26:34.587ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd3b234-d98e-4ed5-a8c0-5bd199d56c47", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52002025-04-30T06:26:34.587ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
52012025-04-30T06:26:34.588ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d3e96c7-7983-4571-bedf-943d4594dea1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52022025-04-30T06:26:34.589ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
52032025-04-30T06:26:34.591ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93cf3bd4-4e8c-4812-a621-f71a3b920a9a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52042025-04-30T06:26:34.592ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
52052025-04-30T06:26:34.593ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e84a94e-95ce-4cc4-ba4b-bc648e271f3b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52062025-04-30T06:26:34.594ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
52072025-04-30T06:26:34.598ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06fba418-1762-4605-a280-f44f83c2ec9e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52082025-04-30T06:26:34.598ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
52092025-04-30T06:26:34.600ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebc71df0-1c3f-4498-afa7-616291cda4b6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52102025-04-30T06:26:34.600ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
52112025-04-30T06:26:34.602ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe1c38ef-46f6-45a3-8002-be1bbb2896a2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52122025-04-30T06:26:34.602ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
52132025-04-30T06:26:34.604ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e4855e1-1d0c-4d94-b20d-b7759c6e3e0f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52142025-04-30T06:26:34.604ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
52152025-04-30T06:26:34.608ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61bbb898-7fd4-4702-9064-1820b75e1228", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52162025-04-30T06:26:34.609ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
52172025-04-30T06:26:34.612ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52182025-04-30T06:26:34.612ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52192025-04-30T06:26:34.612ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52202025-04-30T06:26:34.612ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
52212025-04-30T06:26:34.615ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52222025-04-30T06:26:34.616ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8695ce0a-c2ad-4995-8635-c380b494fb7f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52232025-04-30T06:26:34.616ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
52242025-04-30T06:26:34.618ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52252025-04-30T06:26:34.627ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52262025-04-30T06:26:34.627ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52272025-04-30T06:26:34.627ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
52282025-04-30T06:26:34.628ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52292025-04-30T06:26:34.630ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44d7c92f-6c49-4547-b3b4-b954d52be428", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52302025-04-30T06:26:34.630ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
52312025-04-30T06:26:34.631ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52322025-04-30T06:26:34.631ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
52332025-04-30T06:26:34.638ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52342025-04-30T06:26:34.643ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed9cd29d-5114-4a16-9b93-e23b4ebef35e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52352025-04-30T06:26:34.644ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
52362025-04-30T06:26:34.652ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfc1450a-70ce-488d-ad9b-8ff03cf1dfdc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52372025-04-30T06:26:34.653ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
52382025-04-30T06:26:34.659ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6155867-af23-4e18-a5c1-8855df332b14", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52392025-04-30T06:26:34.660ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
52402025-04-30T06:26:34.664ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd666890-702e-4fc4-bf70-8e7861f90654", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52412025-04-30T06:26:34.664ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
52422025-04-30T06:26:34.666ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f8de90-5b1b-4b8f-8ac9-a2e5c9814ab6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52432025-04-30T06:26:34.666ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
52442025-04-30T06:26:34.671ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
52452025-04-30T06:26:34.672ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a91b865-ec8a-48db-9b94-7e9daa873eb9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52462025-04-30T06:26:34.672ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
52472025-04-30T06:26:34.674ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f8e38e9-02dd-47d6-b6e8-bf852804a5ec", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52482025-04-30T06:26:34.675ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
52492025-04-30T06:26:34.675ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4d088f9-32dd-49aa-818d-0bd3bb0ac312", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52502025-04-30T06:26:34.675ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
52512025-04-30T06:26:34.678ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8014a4f5-f037-431b-a1b4-c0e0a36341a4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52522025-04-30T06:26:34.678ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
52532025-04-30T06:26:34.680ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8691b18e-62b1-406d-9d91-115221afa992", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52542025-04-30T06:26:34.680ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
52552025-04-30T06:26:34.682ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8f5a1d1-481f-470b-94f3-5cde9ca02c70", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52562025-04-30T06:26:34.682ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
52572025-04-30T06:26:34.686ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a065842-a1e4-4b34-91e0-da022e80ba83", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52582025-04-30T06:26:34.686ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
52592025-04-30T06:26:34.688ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad35475a-769b-4c99-893f-e189bc692732", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52602025-04-30T06:26:34.688ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
52612025-04-30T06:26:34.690ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12688075-ccb1-4729-aafe-914d29efae34", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52622025-04-30T06:26:34.690ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
52632025-04-30T06:26:34.691ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70426f06-e6e6-48ee-867b-e0df75340450", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52642025-04-30T06:26:34.692ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
52652025-04-30T06:26:34.692ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ec2da86-09bc-46ec-b5a4-e81a194d4496", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52662025-04-30T06:26:34.693ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
52672025-04-30T06:26:34.694ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cf24d94-c45a-4d69-adfa-cdb99177313e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52682025-04-30T06:26:34.694ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
52692025-04-30T06:26:34.695ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "433cba41-f8bd-4c74-b0b3-a2bccf1caefa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
52702025-04-30T06:26:34.695ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52712025-04-30T06:26:34.695ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52722025-04-30T06:26:34.695ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52732025-04-30T06:26:34.695ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52742025-04-30T06:26:34.696ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52752025-04-30T06:26:34.700ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52762025-04-30T06:26:34.706ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52772025-04-30T06:26:34.706ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52782025-04-30T06:26:34.706ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52792025-04-30T06:26:34.706ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52802025-04-30T06:26:34.706ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52812025-04-30T06:26:34.707ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52822025-04-30T06:26:34.710ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52832025-04-30T06:26:34.715ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52842025-04-30T06:26:34.715ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52852025-04-30T06:26:34.715ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52862025-04-30T06:26:34.715ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52872025-04-30T06:26:34.791ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
52882025-04-30T06:26:34.791ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
52892025-04-30T06:26:34.792ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52902025-04-30T06:26:34.792ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52912025-04-30T06:26:34.792ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52922025-04-30T06:26:34.792ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
52932025-04-30T06:26:34.793ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52942025-04-30T06:26:34.793ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:48794 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
52952025-04-30T06:26:34.796ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:48794 req_id = a49cc739-d705-4ffd-94cc-ba9ea2d117d9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
52962025-04-30T06:26:34.797ZINFOtest_disk_metrics (dropshot): request completed latency_us = 699 local_addr = [::1]:50989 method = GET remote_addr = [::1]:48794 req_id = a49cc739-d705-4ffd-94cc-ba9ea2d117d9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
52972025-04-30T06:26:34.797ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52982025-04-30T06:26:34.804ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52992025-04-30T06:26:34.804ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53002025-04-30T06:26:34.804ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53012025-04-30T06:26:34.805ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53022025-04-30T06:26:34.806ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a49cc739-d705-4ffd-94cc-ba9ea2d117d9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
53032025-04-30T06:26:34.808ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53042025-04-30T06:26:34.808ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53052025-04-30T06:26:34.808ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
53062025-04-30T06:26:34.810ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53072025-04-30T06:26:34.810ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53082025-04-30T06:26:34.810ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53092025-04-30T06:26:34.810ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53102025-04-30T06:26:34.810ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53112025-04-30T06:26:34.813ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53122025-04-30T06:26:34.815ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53132025-04-30T06:26:34.815ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53142025-04-30T06:26:34.815ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53152025-04-30T06:26:34.819ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53162025-04-30T06:26:34.819ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53172025-04-30T06:26:34.820ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53182025-04-30T06:26:34.820ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53192025-04-30T06:26:34.858ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53202025-04-30T06:26:34.860ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53212025-04-30T06:26:34.860ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 452.178465ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53222025-04-30T06:26:34.861ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
53232025-04-30T06:26:34.861ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53242025-04-30T06:26:34.861ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53252025-04-30T06:26:34.862ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53262025-04-30T06:26:34.862ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53272025-04-30T06:26:34.862ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53282025-04-30T06:26:34.864ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
53292025-04-30T06:26:34.864ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
53302025-04-30T06:26:34.864ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53312025-04-30T06:26:34.864ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 26afdb92-1e9b-405f-b801-ae3d57c17794 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53322025-04-30T06:26:34.865ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 26afdb92-1e9b-405f-b801-ae3d57c17794 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53332025-04-30T06:26:34.865ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 26afdb92-1e9b-405f-b801-ae3d57c17794 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53342025-04-30T06:26:34.865ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 26afdb92-1e9b-405f-b801-ae3d57c17794 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53352025-04-30T06:26:34.869ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53362025-04-30T06:26:34.876ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53372025-04-30T06:26:34.876ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53382025-04-30T06:26:34.876ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53392025-04-30T06:26:34.877ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53402025-04-30T06:26:34.878ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 13397 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 26afdb92-1e9b-405f-b801-ae3d57c17794 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53412025-04-30T06:26:34.883ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "26afdb92-1e9b-405f-b801-ae3d57c17794", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
53422025-04-30T06:26:34.883ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53432025-04-30T06:26:34.884ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
53442025-04-30T06:26:34.884ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53452025-04-30T06:26:34.885ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 2ba27b67-57ee-450c-aafb-623782573ee7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53462025-04-30T06:26:34.886ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 2ba27b67-57ee-450c-aafb-623782573ee7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53472025-04-30T06:26:34.887ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 2ba27b67-57ee-450c-aafb-623782573ee7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53482025-04-30T06:26:34.888ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 2ba27b67-57ee-450c-aafb-623782573ee7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53492025-04-30T06:26:34.889ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 22 db 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 "............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58821 req_id = ef4ed6a2-1de7-4546-8e18-6356dbd2d031
53502025-04-30T06:26:34.889ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8923,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58821 req_id = ef4ed6a2-1de7-4546-8e18-6356dbd2d031
53512025-04-30T06:26:34.890ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
53522025-04-30T06:26:34.890ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53532025-04-30T06:26:34.891ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58821 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ef4ed6a2-1de7-4546-8e18-6356dbd2d031
53542025-04-30T06:26:34.895ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3259, tv_nsec: 691350007 } })
53552025-04-30T06:26:34.895ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
53562025-04-30T06:26:34.896ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 10158 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 2ba27b67-57ee-450c-aafb-623782573ee7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53572025-04-30T06:26:34.896ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ba27b67-57ee-450c-aafb-623782573ee7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
53582025-04-30T06:26:34.896ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
53592025-04-30T06:26:34.896ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53602025-04-30T06:26:34.897ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
53612025-04-30T06:26:34.897ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 21 24 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e !$..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52305 req_id = 4440069d-4732-4ced-bf57-b437776b31e1
53622025-04-30T06:26:34.897ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8484,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52305 req_id = 4440069d-4732-4ced-bf57-b437776b31e1
53632025-04-30T06:26:34.899ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53642025-04-30T06:26:34.899ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52305 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4440069d-4732-4ced-bf57-b437776b31e1
53652025-04-30T06:26:34.965ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
53662025-04-30T06:26:34.965ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53672025-04-30T06:26:34.966ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53682025-04-30T06:26:34.966ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53692025-04-30T06:26:34.966ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53702025-04-30T06:26:34.967ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53712025-04-30T06:26:34.968ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: be 17 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50097 req_id = 9e1ad76c-07ef-45ce-a4ec-44715143ad15
53722025-04-30T06:26:34.969ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48663,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50097 req_id = 9e1ad76c-07ef-45ce-a4ec-44715143ad15
53732025-04-30T06:26:34.969ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
53742025-04-30T06:26:34.969ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48663, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50097 req_id = 9e1ad76c-07ef-45ce-a4ec-44715143ad15
53752025-04-30T06:26:34.970ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
53762025-04-30T06:26:34.970ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
53772025-04-30T06:26:34.970ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
53782025-04-30T06:26:34.972ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53792025-04-30T06:26:34.982ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53802025-04-30T06:26:34.983ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53812025-04-30T06:26:34.983ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53822025-04-30T06:26:34.983ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53832025-04-30T06:26:34.989ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53842025-04-30T06:26:34.989ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53852025-04-30T06:26:34.995ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53862025-04-30T06:26:34.996ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53872025-04-30T06:26:34.999ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53882025-04-30T06:26:35.001ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53892025-04-30T06:26:35.002ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53902025-04-30T06:26:35.002ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53912025-04-30T06:26:35.002ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53922025-04-30T06:26:35.009ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "70093c7a-ccdb-4e43-9587-571976ee2aca", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
53932025-04-30T06:26:35.009ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
53942025-04-30T06:26:35.029ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b1e2d6a-541c-4b47-b157-1db7f7b7136b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
53952025-04-30T06:26:35.030ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
53962025-04-30T06:26:35.039ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0498d166-13cf-4d4e-961f-c4c77817b2c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
53972025-04-30T06:26:35.040ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
53982025-04-30T06:26:35.045ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5756daa5-bff3-447b-bb56-546c3cd7a67b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
53992025-04-30T06:26:35.046ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
54002025-04-30T06:26:35.059ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8271e8c9-dc3b-4cd5-a744-e2fb7cfe9b5c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54012025-04-30T06:26:35.060ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
54022025-04-30T06:26:35.068ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
54032025-04-30T06:26:35.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15c94cb2-a9ee-4377-98bb-b68cea6b8aeb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54042025-04-30T06:26:35.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
54052025-04-30T06:26:35.077ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a5c15e3-afdb-42d6-86d7-4ef58c910045", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54062025-04-30T06:26:35.077ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
54072025-04-30T06:26:35.080ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f6d3494-1ccd-4ec2-9882-96e8d474df14", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54082025-04-30T06:26:35.080ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
54092025-04-30T06:26:35.081ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "00b6764e-29b3-476a-97d9-50fcf60285d9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54102025-04-30T06:26:35.082ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
54112025-04-30T06:26:35.083ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aaab2c1-931a-4297-bc3f-ab67b27eefae", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54122025-04-30T06:26:35.083ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
54132025-04-30T06:26:35.084ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19852e58-347a-4e08-973b-bae52a795cd8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54142025-04-30T06:26:35.084ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
54152025-04-30T06:26:35.085ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b42fdbc7-3329-4fdf-9a59-d7241cb1d566", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54162025-04-30T06:26:35.086ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
54172025-04-30T06:26:35.087ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "299d36b6-f72d-4414-a552-82635797336a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54182025-04-30T06:26:35.087ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
54192025-04-30T06:26:35.089ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c86cd8eb-7288-4f69-bb40-576077b95c77", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54202025-04-30T06:26:35.090ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
54212025-04-30T06:26:35.094ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b63790a4-28a9-4ebe-b1e1-15af93e29214", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54222025-04-30T06:26:35.094ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
54232025-04-30T06:26:35.096ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38907b47-4832-4a75-a712-51be9510903d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54242025-04-30T06:26:35.097ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
54252025-04-30T06:26:35.098ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd24b218-6311-4d83-b89d-13cd179b4eac", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54262025-04-30T06:26:35.098ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
54272025-04-30T06:26:35.100ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "719d6ff6-dd78-486f-9541-aa50dd40d9f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54282025-04-30T06:26:35.100ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
54292025-04-30T06:26:35.102ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a1611fc-a848-4bf0-b0fe-436655e4a7c9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54302025-04-30T06:26:35.103ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
54312025-04-30T06:26:35.105ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5e7d3e5-60c4-469f-a96d-73d9a7f45c9e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54322025-04-30T06:26:35.105ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
54332025-04-30T06:26:35.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3b9885-006a-4a55-8087-750168423e17", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54342025-04-30T06:26:35.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
54352025-04-30T06:26:35.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85825c5f-0e5c-4c3f-9283-6376e165da84", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54362025-04-30T06:26:35.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
54372025-04-30T06:26:35.108ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ca17dc3-ca0a-4727-9561-47b6d51dfe7f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54382025-04-30T06:26:35.108ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
54392025-04-30T06:26:35.112ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb8256f8-3387-410b-9f6e-c2ce77409ae4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54402025-04-30T06:26:35.112ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
54412025-04-30T06:26:35.113ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f08941a1-5e17-4326-93d1-863552e4632d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54422025-04-30T06:26:35.113ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
54432025-04-30T06:26:35.115ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "828f0595-9b4a-4207-9f8f-c1cb25a8b8c0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54442025-04-30T06:26:35.115ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
54452025-04-30T06:26:35.118ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "624c2878-c6c8-4189-ba64-ce24683c6f0d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54462025-04-30T06:26:35.118ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
54472025-04-30T06:26:35.119ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54482025-04-30T06:26:35.119ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54492025-04-30T06:26:35.119ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54502025-04-30T06:26:35.119ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54512025-04-30T06:26:35.120ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54522025-04-30T06:26:35.122ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54532025-04-30T06:26:35.129ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54542025-04-30T06:26:35.129ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54552025-04-30T06:26:35.129ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54562025-04-30T06:26:35.130ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54572025-04-30T06:26:35.132ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd1f1732-aead-47bb-8188-29b9ddcc15c7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54582025-04-30T06:26:35.132ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
54592025-04-30T06:26:35.132ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54602025-04-30T06:26:35.132ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54612025-04-30T06:26:35.133ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54622025-04-30T06:26:35.137ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed814f67-91f6-43ae-82dc-8fabda104159", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54632025-04-30T06:26:35.138ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
54642025-04-30T06:26:35.147ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78df8fc8-bd0d-413b-b6f1-ed663ec71267", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54652025-04-30T06:26:35.148ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
54662025-04-30T06:26:35.149ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "169f9930-a82b-4358-8ccb-4d8922cf217b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54672025-04-30T06:26:35.149ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
54682025-04-30T06:26:35.151ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1eb0ae1-9998-48fd-a5c5-ba11d541f22e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54692025-04-30T06:26:35.151ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
54702025-04-30T06:26:35.153ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17186c3e-8c11-486f-a1bb-5b43a4ce20fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54712025-04-30T06:26:35.154ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
54722025-04-30T06:26:35.158ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a58fb2-ac3c-41d0-ae2a-d2748f916a12", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54732025-04-30T06:26:35.159ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
54742025-04-30T06:26:35.161ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8669f4ec-65c0-4ce1-a90f-055154aca5d3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54752025-04-30T06:26:35.161ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
54762025-04-30T06:26:35.164ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94a93bc6-1919-4cca-9ab0-696f98641f3e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54772025-04-30T06:26:35.165ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
54782025-04-30T06:26:35.170ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8bd2cd3-602d-4db1-9b54-c10bda658ab9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54792025-04-30T06:26:35.170ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
54802025-04-30T06:26:35.172ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5a423de-b74c-4d35-8991-7f5e32bc1871", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54812025-04-30T06:26:35.172ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
54822025-04-30T06:26:35.176ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
54832025-04-30T06:26:35.176ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54842025-04-30T06:26:35.176ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54852025-04-30T06:26:35.177ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
54862025-04-30T06:26:35.178ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
54872025-04-30T06:26:35.183ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
54882025-04-30T06:26:35.183ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
54892025-04-30T06:26:35.184ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "474a8a04-3737-4a71-94de-c74229b70521", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54902025-04-30T06:26:35.185ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
54912025-04-30T06:26:35.190ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
54922025-04-30T06:26:35.190ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54932025-04-30T06:26:35.197ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33cc5701-a31a-4595-9e96-7117307e27cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54942025-04-30T06:26:35.199ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
54952025-04-30T06:26:35.201ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0df887bd-3f69-4aae-8e5e-afe3d0668cb8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54962025-04-30T06:26:35.202ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
54972025-04-30T06:26:35.203ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b346a049-f1a1-448e-b564-35ba627d7231", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
54982025-04-30T06:26:35.203ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
54992025-04-30T06:26:35.204ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9ae6dd3-c08d-4b36-91e5-d091d6ebf6df", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
55002025-04-30T06:26:35.204ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
55012025-04-30T06:26:35.205ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceb92e05-ea69-4f89-95ab-574f099b7085", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
55022025-04-30T06:26:35.205ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
55032025-04-30T06:26:35.206ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48cfca83-31b2-4413-bef2-9bed6a7af6be", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
55042025-04-30T06:26:35.206ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55052025-04-30T06:26:35.206ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55062025-04-30T06:26:35.206ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55072025-04-30T06:26:35.206ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55082025-04-30T06:26:35.207ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55092025-04-30T06:26:35.210ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55102025-04-30T06:26:35.214ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55112025-04-30T06:26:35.214ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55122025-04-30T06:26:35.214ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55132025-04-30T06:26:35.214ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55142025-04-30T06:26:35.214ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55152025-04-30T06:26:35.214ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55162025-04-30T06:26:35.218ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55172025-04-30T06:26:35.223ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55182025-04-30T06:26:35.223ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55192025-04-30T06:26:35.223ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55202025-04-30T06:26:35.224ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55212025-04-30T06:26:35.228ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55222025-04-30T06:26:35.228ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55232025-04-30T06:26:35.228ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55242025-04-30T06:26:35.228ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55252025-04-30T06:26:35.229ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55262025-04-30T06:26:35.282ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
55272025-04-30T06:26:35.282ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
55282025-04-30T06:26:35.283ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:56690 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55292025-04-30T06:26:35.284ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
55302025-04-30T06:26:35.284ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:56690 req_id = c6cb4616-0bc8-435a-a27b-379415f98866 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55312025-04-30T06:26:35.285ZINFOtest_disk_metrics (dropshot): request completed latency_us = 663 local_addr = [::1]:50989 method = GET remote_addr = [::1]:56690 req_id = c6cb4616-0bc8-435a-a27b-379415f98866 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55322025-04-30T06:26:35.285ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55332025-04-30T06:26:35.295ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55342025-04-30T06:26:35.295ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55352025-04-30T06:26:35.295ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55362025-04-30T06:26:35.295ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55372025-04-30T06:26:35.297ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6cb4616-0bc8-435a-a27b-379415f98866", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
55382025-04-30T06:26:35.300ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55392025-04-30T06:26:35.301ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55402025-04-30T06:26:35.302ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
55412025-04-30T06:26:35.309ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55422025-04-30T06:26:35.309ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55432025-04-30T06:26:35.311ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55442025-04-30T06:26:35.311ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55452025-04-30T06:26:35.314ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55462025-04-30T06:26:35.317ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55472025-04-30T06:26:35.319ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55482025-04-30T06:26:35.319ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55492025-04-30T06:26:35.320ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55502025-04-30T06:26:35.397ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55512025-04-30T06:26:35.397ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55522025-04-30T06:26:35.397ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55532025-04-30T06:26:35.400ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55542025-04-30T06:26:35.486ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
55552025-04-30T06:26:35.487ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
55562025-04-30T06:26:35.487ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
55572025-04-30T06:26:35.487ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 626.844036ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
55582025-04-30T06:26:35.487ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
55592025-04-30T06:26:35.487ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55602025-04-30T06:26:35.488ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55612025-04-30T06:26:35.488ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55622025-04-30T06:26:35.488ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55632025-04-30T06:26:35.488ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55642025-04-30T06:26:35.491ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55652025-04-30T06:26:35.494ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55662025-04-30T06:26:35.494ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55672025-04-30T06:26:35.494ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55682025-04-30T06:26:35.494ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55692025-04-30T06:26:35.498ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
55702025-04-30T06:26:35.498ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
55712025-04-30T06:26:35.498ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 31 87 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 1............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51446 req_id = a59ea18c-d198-4307-9363-dd4362e2d1f2
55722025-04-30T06:26:35.498ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12679,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51446 req_id = a59ea18c-d198-4307-9363-dd4362e2d1f2
55732025-04-30T06:26:35.499ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
55742025-04-30T06:26:35.499ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55752025-04-30T06:26:35.499ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51446 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a59ea18c-d198-4307-9363-dd4362e2d1f2
55762025-04-30T06:26:35.499ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3260, tv_nsec: 298534613 } })
55772025-04-30T06:26:35.499ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
55782025-04-30T06:26:35.500ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6e 61 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e na..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49962 req_id = e67d7af2-070f-4df6-8e81-9234a7f30987
55792025-04-30T06:26:35.500ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28257,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49962 req_id = e67d7af2-070f-4df6-8e81-9234a7f30987
55802025-04-30T06:26:35.500ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55812025-04-30T06:26:35.500ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49962 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e67d7af2-070f-4df6-8e81-9234a7f30987
55822025-04-30T06:26:35.643ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
55832025-04-30T06:26:35.643ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55842025-04-30T06:26:35.643ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55852025-04-30T06:26:35.643ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55862025-04-30T06:26:35.644ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55872025-04-30T06:26:35.644ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55882025-04-30T06:26:35.645ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a3 17 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61747 req_id = 83ef5691-fd73-4e79-a0ee-95c95857607a
55892025-04-30T06:26:35.645ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41751,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61747 req_id = 83ef5691-fd73-4e79-a0ee-95c95857607a
55902025-04-30T06:26:35.645ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
55912025-04-30T06:26:35.645ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41751, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61747 req_id = 83ef5691-fd73-4e79-a0ee-95c95857607a
55922025-04-30T06:26:35.645ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
55932025-04-30T06:26:35.645ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
55942025-04-30T06:26:35.645ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
55952025-04-30T06:26:35.648ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55962025-04-30T06:26:35.655ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55972025-04-30T06:26:35.655ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55982025-04-30T06:26:35.656ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55992025-04-30T06:26:35.656ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56002025-04-30T06:26:35.657ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "362d662a-9eec-42b0-a20d-21adbd200b55", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56012025-04-30T06:26:35.657ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
56022025-04-30T06:26:35.660ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56032025-04-30T06:26:35.660ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56042025-04-30T06:26:35.660ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56052025-04-30T06:26:35.660ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40dcd205-5f43-49cf-b6be-ba0dbcd40ca2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56062025-04-30T06:26:35.661ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
56072025-04-30T06:26:35.663ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "522e088c-906f-43de-86f3-b3dd17752619", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56082025-04-30T06:26:35.663ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
56092025-04-30T06:26:35.665ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecb79121-c841-468d-8e80-4c1f9330ee83", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56102025-04-30T06:26:35.665ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
56112025-04-30T06:26:35.668ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a418470a-2d1e-4157-80e4-d274bb8fa105", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56122025-04-30T06:26:35.669ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
56132025-04-30T06:26:35.671ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "adfdcbdb-5256-497e-b60b-66acc2c9d092", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56142025-04-30T06:26:35.671ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
56152025-04-30T06:26:35.675ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "973a1b44-c534-4bc6-8266-daaaa4f63171", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56162025-04-30T06:26:35.691ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
56172025-04-30T06:26:35.698ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39c4f638-a69e-4ae6-9946-a0f484aa5329", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56182025-04-30T06:26:35.699ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
56192025-04-30T06:26:35.701ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "408cc35f-3a54-4ef8-bb3b-ce1b83eed2aa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56202025-04-30T06:26:35.701ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
56212025-04-30T06:26:35.706ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cbe64e2-821c-4a40-9c64-0949eb1dfca1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56222025-04-30T06:26:35.706ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
56232025-04-30T06:26:35.709ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "763cf0cf-de7e-4204-8d1b-473590f74871", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56242025-04-30T06:26:35.710ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
56252025-04-30T06:26:35.711ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2cbc78d-de11-4c4d-a17b-d6b5a2d62960", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56262025-04-30T06:26:35.712ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
56272025-04-30T06:26:35.713ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8502bd66-39d6-4b90-84ea-9b8c34e28e70", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56282025-04-30T06:26:35.720ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
56292025-04-30T06:26:35.720ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
56302025-04-30T06:26:35.723ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eb6edb6-4e59-4fb5-b1c0-e606e82dd392", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56312025-04-30T06:26:35.724ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
56322025-04-30T06:26:35.727ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c741361-45f5-41d2-995b-4a395e86342b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56332025-04-30T06:26:35.728ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
56342025-04-30T06:26:35.733ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4252bf50-ddfe-4adb-a926-62352dd0730d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56352025-04-30T06:26:35.733ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
56362025-04-30T06:26:35.736ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6945249-f6bf-46ad-915d-c33e029cf78e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56372025-04-30T06:26:35.736ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
56382025-04-30T06:26:35.739ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef797573-6fce-4d84-9af5-e8fb173eae9e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56392025-04-30T06:26:35.739ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
56402025-04-30T06:26:35.744ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c1129b-20d8-4d15-92aa-3696a4b99549", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56412025-04-30T06:26:35.744ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
56422025-04-30T06:26:35.746ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e9522fb-2e0b-45fc-8dbd-b5281e28534e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56432025-04-30T06:26:35.747ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
56442025-04-30T06:26:35.748ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70e02acc-f494-4337-93f0-191a37f18e10", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56452025-04-30T06:26:35.748ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
56462025-04-30T06:26:35.749ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c301374-8ae9-4d1a-ba1f-a063b5b176a4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56472025-04-30T06:26:35.749ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
56482025-04-30T06:26:35.750ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "feb9386f-d852-434b-b513-fee26ad5743f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56492025-04-30T06:26:35.750ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
56502025-04-30T06:26:35.752ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aacefdcb-8851-432a-a09f-fd3c89d64354", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56512025-04-30T06:26:35.752ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
56522025-04-30T06:26:35.757ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7963ab0a-cded-4bc0-bc56-c51c75362f5d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56532025-04-30T06:26:35.757ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
56542025-04-30T06:26:35.759ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a2ce5e-ce21-4e11-ad03-44b4594e8bbd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56552025-04-30T06:26:35.760ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
56562025-04-30T06:26:35.762ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3024fb0a-c42f-4230-881f-070fab7708d5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56572025-04-30T06:26:35.762ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
56582025-04-30T06:26:35.764ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a47e7c72-7694-4abf-8acd-1426502d9a5e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56592025-04-30T06:26:35.764ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
56602025-04-30T06:26:35.767ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "150b5c0c-7bd8-443b-8012-d32949399a84", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56612025-04-30T06:26:35.768ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
56622025-04-30T06:26:35.770ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af8512cd-de3f-4f9b-a6b2-0f08c4eb5c41", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56632025-04-30T06:26:35.770ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
56642025-04-30T06:26:35.771ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56652025-04-30T06:26:35.771ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56662025-04-30T06:26:35.771ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56672025-04-30T06:26:35.771ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56682025-04-30T06:26:35.772ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56692025-04-30T06:26:35.772ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a552f10-d97a-458a-ae52-621a6d6ef5cd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56702025-04-30T06:26:35.773ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
56712025-04-30T06:26:35.777ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e46577f-b6a4-4635-a204-cd38efcfd85c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56722025-04-30T06:26:35.778ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
56732025-04-30T06:26:35.778ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56742025-04-30T06:26:35.798ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56752025-04-30T06:26:35.798ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56762025-04-30T06:26:35.798ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56772025-04-30T06:26:35.799ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56782025-04-30T06:26:35.825ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b0caccc-de39-430b-be75-151809d30e45", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56792025-04-30T06:26:35.825ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
56802025-04-30T06:26:35.827ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56812025-04-30T06:26:35.827ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56822025-04-30T06:26:35.828ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56832025-04-30T06:26:35.831ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3cc69f3-1b56-4147-b6b7-89872dcaefba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56842025-04-30T06:26:35.831ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
56852025-04-30T06:26:35.834ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62f6a398-fd80-4705-9728-4119cece01b5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56862025-04-30T06:26:35.835ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
56872025-04-30T06:26:35.841ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88099d43-de06-45bb-92dd-260f8e972cc4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56882025-04-30T06:26:35.841ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
56892025-04-30T06:26:35.846ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "488d26fe-71bb-4269-8ab1-49144a5f21f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56902025-04-30T06:26:35.846ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
56912025-04-30T06:26:35.852ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0cd237c-191c-40f4-bde6-59fb2621768e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56922025-04-30T06:26:35.852ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
56932025-04-30T06:26:35.856ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "814db2f9-4e7c-4b7e-8c8f-cdd1369af74a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56942025-04-30T06:26:35.856ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
56952025-04-30T06:26:35.858ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c34e6f92-ef99-4be9-b513-44ac56c55bf6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56962025-04-30T06:26:35.858ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
56972025-04-30T06:26:35.865ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
56982025-04-30T06:26:35.870ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51352165-4d53-4cc4-8d0a-91793f79f373", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
56992025-04-30T06:26:35.870ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
57002025-04-30T06:26:35.871ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2e5f672-5225-494a-a571-8e1c5d79d7bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
57012025-04-30T06:26:35.871ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
57022025-04-30T06:26:35.873ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "568f807d-e85d-477e-b444-357f7a529172", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
57032025-04-30T06:26:35.873ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
57042025-04-30T06:26:35.874ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24771ab4-32d4-4c24-b127-b9aa2b3b3ae1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
57052025-04-30T06:26:35.874ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
57062025-04-30T06:26:35.876ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96f66920-6f8d-4be1-b6b4-925d07b5a1c7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
57072025-04-30T06:26:35.876ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57082025-04-30T06:26:35.876ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57092025-04-30T06:26:35.876ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57102025-04-30T06:26:35.876ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57112025-04-30T06:26:35.878ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57122025-04-30T06:26:35.883ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57132025-04-30T06:26:35.888ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57142025-04-30T06:26:35.888ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57152025-04-30T06:26:35.888ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57162025-04-30T06:26:35.888ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57172025-04-30T06:26:35.888ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57182025-04-30T06:26:35.888ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57192025-04-30T06:26:35.900ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57202025-04-30T06:26:35.905ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57212025-04-30T06:26:35.906ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57222025-04-30T06:26:35.906ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57232025-04-30T06:26:35.906ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57242025-04-30T06:26:35.962ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
57252025-04-30T06:26:35.962ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
57262025-04-30T06:26:35.962ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57272025-04-30T06:26:35.962ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57282025-04-30T06:26:35.962ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57292025-04-30T06:26:35.962ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57302025-04-30T06:26:35.963ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57312025-04-30T06:26:35.963ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:61871 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57322025-04-30T06:26:35.967ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:61871 req_id = 3b7a2217-97d0-45e1-b20c-76c2dab82566 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57332025-04-30T06:26:35.988ZINFOtest_disk_metrics (dropshot): request completed latency_us = 21806 local_addr = [::1]:50989 method = GET remote_addr = [::1]:61871 req_id = 3b7a2217-97d0-45e1-b20c-76c2dab82566 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57342025-04-30T06:26:35.990ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57352025-04-30T06:26:35.990ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b7a2217-97d0-45e1-b20c-76c2dab82566", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
57362025-04-30T06:26:35.999ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57372025-04-30T06:26:35.999ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57382025-04-30T06:26:35.999ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
57392025-04-30T06:26:36.005ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57402025-04-30T06:26:36.005ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57412025-04-30T06:26:36.005ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57422025-04-30T06:26:36.005ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57432025-04-30T06:26:36.005ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57442025-04-30T06:26:36.006ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57452025-04-30T06:26:36.008ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57462025-04-30T06:26:36.019ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57472025-04-30T06:26:36.019ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57482025-04-30T06:26:36.019ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57492025-04-30T06:26:36.019ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57502025-04-30T06:26:36.028ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57512025-04-30T06:26:36.030ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57522025-04-30T06:26:36.030ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57532025-04-30T06:26:36.030ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57542025-04-30T06:26:36.030ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57552025-04-30T06:26:36.037ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57562025-04-30T06:26:36.037ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57572025-04-30T06:26:36.037ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57582025-04-30T06:26:36.038ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57592025-04-30T06:26:36.040ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57602025-04-30T06:26:36.040ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57612025-04-30T06:26:36.041ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57622025-04-30T06:26:36.094ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
57632025-04-30T06:26:36.094ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
57642025-04-30T06:26:36.095ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 607.239224ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
57652025-04-30T06:26:36.095ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
57662025-04-30T06:26:36.095ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57672025-04-30T06:26:36.095ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57682025-04-30T06:26:36.095ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57692025-04-30T06:26:36.095ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57702025-04-30T06:26:36.096ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57712025-04-30T06:26:36.108ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57722025-04-30T06:26:36.112ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57732025-04-30T06:26:36.112ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57742025-04-30T06:26:36.112ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57752025-04-30T06:26:36.112ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57762025-04-30T06:26:36.113ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
57772025-04-30T06:26:36.118ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
57782025-04-30T06:26:36.118ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
57792025-04-30T06:26:36.119ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5e 7f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ^............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62179 req_id = 530228aa-29d5-4c6b-a470-58b18ef2bb4a
57802025-04-30T06:26:36.119ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24191,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62179 req_id = 530228aa-29d5-4c6b-a470-58b18ef2bb4a
57812025-04-30T06:26:36.119ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
57822025-04-30T06:26:36.119ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57832025-04-30T06:26:36.119ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62179 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 530228aa-29d5-4c6b-a470-58b18ef2bb4a
57842025-04-30T06:26:36.121ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3260, tv_nsec: 920107707 } })
57852025-04-30T06:26:36.121ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
57862025-04-30T06:26:36.121ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0f 93 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49294 req_id = dc22a529-8940-49db-9e5b-bb29099872da
57872025-04-30T06:26:36.121ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3987,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49294 req_id = dc22a529-8940-49db-9e5b-bb29099872da
57882025-04-30T06:26:36.121ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57892025-04-30T06:26:36.121ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49294 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dc22a529-8940-49db-9e5b-bb29099872da
57902025-04-30T06:26:36.188ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
57912025-04-30T06:26:36.188ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57922025-04-30T06:26:36.188ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57932025-04-30T06:26:36.188ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57942025-04-30T06:26:36.188ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57952025-04-30T06:26:36.189ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57962025-04-30T06:26:36.189ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57972025-04-30T06:26:36.189ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57982025-04-30T06:26:36.189ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
57992025-04-30T06:26:36.190ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
58002025-04-30T06:26:36.191ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 69 ab 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c i............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64042 req_id = 8f12c757-1d44-44cb-a4b9-86fd4326e23a
58012025-04-30T06:26:36.191ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27051,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64042 req_id = 8f12c757-1d44-44cb-a4b9-86fd4326e23a
58022025-04-30T06:26:36.192ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
58032025-04-30T06:26:36.192ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27051, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64042 req_id = 8f12c757-1d44-44cb-a4b9-86fd4326e23a
58042025-04-30T06:26:36.194ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
58052025-04-30T06:26:36.194ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
58062025-04-30T06:26:36.194ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
58072025-04-30T06:26:36.195ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
58082025-04-30T06:26:36.195ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58092025-04-30T06:26:36.197ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58102025-04-30T06:26:36.207ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58112025-04-30T06:26:36.207ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58122025-04-30T06:26:36.207ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
58132025-04-30T06:26:36.208ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58142025-04-30T06:26:36.212ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
58152025-04-30T06:26:36.212ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58162025-04-30T06:26:36.213ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "102e3b48-6180-43a0-abd4-ef47468597ff", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58172025-04-30T06:26:36.213ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
58182025-04-30T06:26:36.215ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58192025-04-30T06:26:36.215ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
58202025-04-30T06:26:36.218ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58212025-04-30T06:26:36.220ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed7d70b-05d2-437a-a0ad-4710d91a495b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58222025-04-30T06:26:36.220ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
58232025-04-30T06:26:36.223ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90e398fd-ca3d-4bf5-a62d-cf048909727f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58242025-04-30T06:26:36.223ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
58252025-04-30T06:26:36.225ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "256acc50-5eac-4ab6-8301-fd24899f42f1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58262025-04-30T06:26:36.226ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
58272025-04-30T06:26:36.231ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f685a29a-c211-44e9-a41a-f775d25335b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58282025-04-30T06:26:36.231ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
58292025-04-30T06:26:36.233ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10670538-f950-4bbd-aa11-d5ae97378a0c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58302025-04-30T06:26:36.233ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
58312025-04-30T06:26:36.237ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cc7ba18-03a7-46ef-ac0b-f5c3afc78562", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58322025-04-30T06:26:36.237ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
58332025-04-30T06:26:36.241ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98efd4f6-9ea2-4efc-ad06-db2e68bf1912", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58342025-04-30T06:26:36.241ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
58352025-04-30T06:26:36.242ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b6d0523-fff7-4a40-8d43-f7eded1774d1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58362025-04-30T06:26:36.242ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
58372025-04-30T06:26:36.244ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40e1ee8c-5581-4a69-b387-b1b99f918512", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58382025-04-30T06:26:36.245ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
58392025-04-30T06:26:36.246ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8c88ff7-aa0b-4beb-9efd-5ba0739ffea7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58402025-04-30T06:26:36.246ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
58412025-04-30T06:26:36.252ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
58422025-04-30T06:26:36.256ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5b77669-950b-43b6-804a-9089cb12fe67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58432025-04-30T06:26:36.258ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
58442025-04-30T06:26:36.261ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "961135fb-92ea-4c7b-9b15-83405a37715f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58452025-04-30T06:26:36.261ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
58462025-04-30T06:26:36.264ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4448dca-3a18-46b9-9a79-c460ddf96968", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58472025-04-30T06:26:36.264ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
58482025-04-30T06:26:36.268ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "212c7e3a-1f6d-4909-9b13-fd5716f11635", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58492025-04-30T06:26:36.268ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
58502025-04-30T06:26:36.271ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da799fda-3182-4928-bf63-c625b0720c18", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58512025-04-30T06:26:36.271ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
58522025-04-30T06:26:36.276ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95601eb4-7d54-4d08-bcdb-cd1eb2ec68ba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58532025-04-30T06:26:36.276ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
58542025-04-30T06:26:36.279ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80a4571c-604f-43ae-9407-9f189459a719", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58552025-04-30T06:26:36.279ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
58562025-04-30T06:26:36.281ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65f05567-a893-42e3-a025-996c4a09f200", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58572025-04-30T06:26:36.284ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
58582025-04-30T06:26:36.286ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40aeb76b-92ee-4c1e-8cc0-9243a17a2437", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58592025-04-30T06:26:36.287ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
58602025-04-30T06:26:36.288ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45216add-2e6f-485b-9ba9-a018ebd4acb1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58612025-04-30T06:26:36.288ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
58622025-04-30T06:26:36.289ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1845d92b-7394-40ef-a5aa-990b12525d81", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58632025-04-30T06:26:36.289ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
58642025-04-30T06:26:36.290ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58652025-04-30T06:26:36.290ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eefc7736-77ac-4a06-a499-213b73d0df96", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58662025-04-30T06:26:36.290ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
58672025-04-30T06:26:36.292ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81886aff-bdfd-44c6-8dda-23c47ec09119", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58682025-04-30T06:26:36.292ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
58692025-04-30T06:26:36.294ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "760bdbe2-99fd-47e3-8f2a-71af91479ef5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58702025-04-30T06:26:36.294ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
58712025-04-30T06:26:36.300ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cda3f9d-c457-40e5-beb4-44f88e5c2db9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58722025-04-30T06:26:36.300ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
58732025-04-30T06:26:36.302ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfdaaaa6-d1a3-4c3c-81f5-c7c933b91e9d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58742025-04-30T06:26:36.302ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
58752025-04-30T06:26:36.303ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58762025-04-30T06:26:36.303ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58772025-04-30T06:26:36.303ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58782025-04-30T06:26:36.303ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
58792025-04-30T06:26:36.304ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58802025-04-30T06:26:36.311ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66514b00-c976-453c-ab0a-5f61ad5501e4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58812025-04-30T06:26:36.311ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
58822025-04-30T06:26:36.313ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d4345c3-5e5c-43da-af1f-86ee3b0e4d78", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58832025-04-30T06:26:36.313ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
58842025-04-30T06:26:36.316ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad5232a0-e941-427f-a231-02a89572831c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58852025-04-30T06:26:36.316ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
58862025-04-30T06:26:36.318ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ba79455-55c5-4cec-acf7-8193af588c1a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58872025-04-30T06:26:36.318ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
58882025-04-30T06:26:36.321ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e989a840-3ab2-4d06-b343-0882c70e4ae7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58892025-04-30T06:26:36.321ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
58902025-04-30T06:26:36.322ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e980e624-0576-4af4-9aa6-8f7fc21d5923", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58912025-04-30T06:26:36.322ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
58922025-04-30T06:26:36.324ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d07389f-a845-4a2f-8938-111977e5241e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58932025-04-30T06:26:36.324ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
58942025-04-30T06:26:36.325ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3837cb94-4bd3-4e59-814f-9480fce7ea51", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58952025-04-30T06:26:36.325ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
58962025-04-30T06:26:36.327ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22fcc68a-e4fc-4d04-a9d3-347475075e4a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
58972025-04-30T06:26:36.328ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
58982025-04-30T06:26:36.328ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58992025-04-30T06:26:36.337ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59002025-04-30T06:26:36.337ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59012025-04-30T06:26:36.337ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59022025-04-30T06:26:36.340ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59032025-04-30T06:26:36.345ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59042025-04-30T06:26:36.345ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59052025-04-30T06:26:36.345ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59062025-04-30T06:26:36.350ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fe259fc-bfb0-45e9-a573-74f6cba36bea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59072025-04-30T06:26:36.350ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
59082025-04-30T06:26:36.359ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83097912-948d-441f-80f3-af36978d43af", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59092025-04-30T06:26:36.360ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
59102025-04-30T06:26:36.364ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75aa0c6f-6a63-4c57-88c6-2c31e415961b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59112025-04-30T06:26:36.364ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
59122025-04-30T06:26:36.370ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81c21f1c-f269-4e67-9902-e9055a097d9d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59132025-04-30T06:26:36.370ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
59142025-04-30T06:26:36.373ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb896702-61ab-4184-8fda-bfe16cffed4c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59152025-04-30T06:26:36.373ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
59162025-04-30T06:26:36.377ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95577c47-1a18-467f-b634-bd948cf97235", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59172025-04-30T06:26:36.377ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
59182025-04-30T06:26:36.381ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eaa9785-8259-4000-8eb5-a00a9b9b0f91", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59192025-04-30T06:26:36.381ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
59202025-04-30T06:26:36.385ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
59212025-04-30T06:26:36.386ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb2c4e2-fe61-48e9-85b1-0e3b9c429c12", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59222025-04-30T06:26:36.386ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
59232025-04-30T06:26:36.388ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9b9585-2ca0-4a37-9267-c71e0113d2c4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59242025-04-30T06:26:36.388ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59252025-04-30T06:26:36.389ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59262025-04-30T06:26:36.389ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59272025-04-30T06:26:36.389ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59282025-04-30T06:26:36.390ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59292025-04-30T06:26:36.393ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59302025-04-30T06:26:36.400ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59312025-04-30T06:26:36.400ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59322025-04-30T06:26:36.400ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59332025-04-30T06:26:36.400ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59342025-04-30T06:26:36.400ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59352025-04-30T06:26:36.401ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59362025-04-30T06:26:36.404ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59372025-04-30T06:26:36.409ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59382025-04-30T06:26:36.409ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59392025-04-30T06:26:36.410ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59402025-04-30T06:26:36.411ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59412025-04-30T06:26:36.600ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
59422025-04-30T06:26:36.600ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
59432025-04-30T06:26:36.600ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59442025-04-30T06:26:36.600ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59452025-04-30T06:26:36.600ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59462025-04-30T06:26:36.601ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59472025-04-30T06:26:36.601ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59482025-04-30T06:26:36.601ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:36545 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
59492025-04-30T06:26:36.602ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:36545 req_id = f10622b0-fc84-4840-a67b-7c4400ade017 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59502025-04-30T06:26:36.602ZINFOtest_disk_metrics (dropshot): request completed latency_us = 461 local_addr = [::1]:50989 method = GET remote_addr = [::1]:36545 req_id = f10622b0-fc84-4840-a67b-7c4400ade017 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59512025-04-30T06:26:36.604ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10622b0-fc84-4840-a67b-7c4400ade017", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
59522025-04-30T06:26:36.610ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59532025-04-30T06:26:36.610ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59542025-04-30T06:26:36.610ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
59552025-04-30T06:26:36.612ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59562025-04-30T06:26:36.612ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59572025-04-30T06:26:36.612ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59582025-04-30T06:26:36.612ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59592025-04-30T06:26:36.613ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59602025-04-30T06:26:36.614ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59612025-04-30T06:26:36.657ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59622025-04-30T06:26:36.657ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59632025-04-30T06:26:36.657ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59642025-04-30T06:26:36.658ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59652025-04-30T06:26:36.664ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59662025-04-30T06:26:36.669ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59672025-04-30T06:26:36.670ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59682025-04-30T06:26:36.670ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59692025-04-30T06:26:36.670ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59702025-04-30T06:26:36.671ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59712025-04-30T06:26:36.671ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59722025-04-30T06:26:36.671ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59732025-04-30T06:26:36.749ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
59742025-04-30T06:26:36.749ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
59752025-04-30T06:26:36.749ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 654.559979ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
59762025-04-30T06:26:36.749ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
59772025-04-30T06:26:36.749ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59782025-04-30T06:26:36.749ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59792025-04-30T06:26:36.750ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59802025-04-30T06:26:36.750ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59812025-04-30T06:26:36.752ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59822025-04-30T06:26:36.752ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
59832025-04-30T06:26:36.755ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59842025-04-30T06:26:36.762ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59852025-04-30T06:26:36.762ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59862025-04-30T06:26:36.762ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59872025-04-30T06:26:36.764ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59882025-04-30T06:26:36.767ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
59892025-04-30T06:26:36.767ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
59902025-04-30T06:26:36.768ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f9 0a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56938 req_id = 5fad9421-dbff-45ae-86f1-0cf302f61aff
59912025-04-30T06:26:36.768ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63754,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56938 req_id = 5fad9421-dbff-45ae-86f1-0cf302f61aff
59922025-04-30T06:26:36.769ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
59932025-04-30T06:26:36.796ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
59942025-04-30T06:26:36.796ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56938 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5fad9421-dbff-45ae-86f1-0cf302f61aff
59952025-04-30T06:26:36.797ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3261, tv_nsec: 596643069 } })
59962025-04-30T06:26:36.802ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
59972025-04-30T06:26:36.803ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59982025-04-30T06:26:36.803ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59992025-04-30T06:26:36.803ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60002025-04-30T06:26:36.803ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60012025-04-30T06:26:36.804ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60022025-04-30T06:26:36.804ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c3 96 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56964 req_id = 84214f2e-b2fa-43a3-982c-785f9a472607
60032025-04-30T06:26:36.805ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50070,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56964 req_id = 84214f2e-b2fa-43a3-982c-785f9a472607
60042025-04-30T06:26:36.805ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60052025-04-30T06:26:36.806ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56964 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 84214f2e-b2fa-43a3-982c-785f9a472607
60062025-04-30T06:26:36.865ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60072025-04-30T06:26:36.867ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60082025-04-30T06:26:36.872ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60092025-04-30T06:26:36.872ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60102025-04-30T06:26:36.872ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60112025-04-30T06:26:36.872ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60122025-04-30T06:26:36.873ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
60132025-04-30T06:26:36.875ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60142025-04-30T06:26:36.877ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 1c61f5ba-ee38-4413-82ee-f8e0958697dc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60152025-04-30T06:26:36.877ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a3 cf 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56065 req_id = f05e5bc9-7e4a-4aa9-b777-9618c2034268
60162025-04-30T06:26:36.877ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41935,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56065 req_id = f05e5bc9-7e4a-4aa9-b777-9618c2034268
60172025-04-30T06:26:36.877ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
60182025-04-30T06:26:36.877ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41935, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56065 req_id = f05e5bc9-7e4a-4aa9-b777-9618c2034268
60192025-04-30T06:26:36.877ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 1c61f5ba-ee38-4413-82ee-f8e0958697dc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60202025-04-30T06:26:36.877ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 1c61f5ba-ee38-4413-82ee-f8e0958697dc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60212025-04-30T06:26:36.878ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 1c61f5ba-ee38-4413-82ee-f8e0958697dc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60222025-04-30T06:26:36.878ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60232025-04-30T06:26:36.878ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60242025-04-30T06:26:36.878ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60252025-04-30T06:26:36.879ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
60262025-04-30T06:26:36.879ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
60272025-04-30T06:26:36.879ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
60282025-04-30T06:26:36.881ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2479996b-e2ad-43ca-9130-96d3144cc314", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60292025-04-30T06:26:36.881ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
60302025-04-30T06:26:36.882ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 5461 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 1c61f5ba-ee38-4413-82ee-f8e0958697dc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60312025-04-30T06:26:36.883ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c61f5ba-ee38-4413-82ee-f8e0958697dc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60322025-04-30T06:26:36.883ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60332025-04-30T06:26:36.884ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f3fbb388-2bdf-4b23-86c2-bd9411f5298b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60342025-04-30T06:26:36.884ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f3fbb388-2bdf-4b23-86c2-bd9411f5298b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60352025-04-30T06:26:36.884ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f3fbb388-2bdf-4b23-86c2-bd9411f5298b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60362025-04-30T06:26:36.884ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f3fbb388-2bdf-4b23-86c2-bd9411f5298b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60372025-04-30T06:26:36.885ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9599dac-8ded-46a2-97b6-0596a30b9d24", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60382025-04-30T06:26:36.885ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
60392025-04-30T06:26:36.887ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 3338 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f3fbb388-2bdf-4b23-86c2-bd9411f5298b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60402025-04-30T06:26:36.887ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3fbb388-2bdf-4b23-86c2-bd9411f5298b", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60412025-04-30T06:26:36.887ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
60422025-04-30T06:26:36.887ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60432025-04-30T06:26:36.887ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
60442025-04-30T06:26:36.888ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70f5194b-c695-449b-b2f3-8eb8ac13fc80", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60452025-04-30T06:26:36.889ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
60462025-04-30T06:26:36.891ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e2731ae-6de2-4420-bd20-7918118dc054", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60472025-04-30T06:26:36.891ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
60482025-04-30T06:26:36.894ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceb400b1-4284-467b-b991-5c366dc2f614", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60492025-04-30T06:26:36.894ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
60502025-04-30T06:26:36.896ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d29218f-dd43-465a-aba4-0ec5e50d8fe9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60512025-04-30T06:26:36.896ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
60522025-04-30T06:26:36.898ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "525bfa59-0535-4545-b299-fd582c11a8ad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60532025-04-30T06:26:36.898ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
60542025-04-30T06:26:36.900ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0180690-0d4e-4d91-96c0-a7f8b7b8a29b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60552025-04-30T06:26:36.900ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
60562025-04-30T06:26:36.901ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "309e7c75-6348-4bc3-8c07-5c0cf93e3749", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60572025-04-30T06:26:36.901ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
60582025-04-30T06:26:36.903ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "641ebb4b-c9a2-465b-aa5a-d350cb8b0be1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60592025-04-30T06:26:36.903ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
60602025-04-30T06:26:36.905ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c33dcca-3898-4d2e-96e1-27e1f70b2c18", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60612025-04-30T06:26:36.905ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
60622025-04-30T06:26:36.908ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7b7a65f-2b0a-4221-81e0-02e44e5f083f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60632025-04-30T06:26:36.908ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
60642025-04-30T06:26:36.908ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
60652025-04-30T06:26:36.927ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "481493f1-3549-4002-829d-a763833b9714", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60662025-04-30T06:26:36.927ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
60672025-04-30T06:26:36.960ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60682025-04-30T06:26:36.964ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60692025-04-30T06:26:36.965ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60702025-04-30T06:26:36.965ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60712025-04-30T06:26:36.966ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60722025-04-30T06:26:36.986ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60732025-04-30T06:26:36.992ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60742025-04-30T06:26:36.992ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60752025-04-30T06:26:36.992ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60762025-04-30T06:26:36.992ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60772025-04-30T06:26:36.994ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60782025-04-30T06:26:36.994ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d88af5e-1b37-4d8f-84be-5a6a9c2e716a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
60792025-04-30T06:26:36.995ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
60802025-04-30T06:26:36.995ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60812025-04-30T06:26:37.015ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60822025-04-30T06:26:37.016ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60832025-04-30T06:26:37.017ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60842025-04-30T06:26:37.017ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60852025-04-30T06:26:37.017ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60862025-04-30T06:26:37.017ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60872025-04-30T06:26:37.019ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60882025-04-30T06:26:37.030ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2b88623-01d3-4ed9-8dc8-a1ef3aab5bfb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
60892025-04-30T06:26:37.030ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
60902025-04-30T06:26:37.034ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53beb4e5-385f-4e3a-92fe-473bf0090a9d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
60912025-04-30T06:26:37.035ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
60922025-04-30T06:26:37.043ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92465373-cb27-4e89-a021-a91bf942e990", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
60932025-04-30T06:26:37.043ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
60942025-04-30T06:26:37.054ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95d30e40-7501-4c28-b9fb-9d6cc4ee4c2b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
60952025-04-30T06:26:37.055ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
60962025-04-30T06:26:37.059ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7c8c99-0b7a-4486-ab02-8e6ba3d4cb93", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
60972025-04-30T06:26:37.060ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
60982025-04-30T06:26:37.061ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2f2ef7a-6c90-4c29-a913-cbd2e1461ba4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
60992025-04-30T06:26:37.061ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
61002025-04-30T06:26:37.063ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59121cbe-0eac-4d15-ad4e-6d4455a113b8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61012025-04-30T06:26:37.063ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
61022025-04-30T06:26:37.067ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "237dd184-b8e6-4c15-9a0d-ce997c8460f3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61032025-04-30T06:26:37.067ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
61042025-04-30T06:26:37.068ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
61052025-04-30T06:26:37.069ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5dd96ff-d309-494e-91a4-abccd5501fab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61062025-04-30T06:26:37.069ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
61072025-04-30T06:26:37.071ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f13ff823-cbfc-449d-8a55-9a7bae783eb6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61082025-04-30T06:26:37.071ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
61092025-04-30T06:26:37.073ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aade0cb-cc49-4625-9752-2ad0b8dfc66d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61102025-04-30T06:26:37.073ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
61112025-04-30T06:26:37.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d0dbab2-4029-4494-b990-7e2858d0886d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61122025-04-30T06:26:37.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
61132025-04-30T06:26:37.077ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f62f4b64-1ff3-4b1e-9228-94fa42449c28", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61142025-04-30T06:26:37.077ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
61152025-04-30T06:26:37.080ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6442826-b532-4e15-9562-6408851bfefb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61162025-04-30T06:26:37.082ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
61172025-04-30T06:26:37.085ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "162f247b-ec5a-4b1c-bd2a-ffd5cda93e3a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61182025-04-30T06:26:37.085ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
61192025-04-30T06:26:37.112ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "448a1dcb-7dbb-4fcc-ad10-2069c56431fe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61202025-04-30T06:26:37.113ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
61212025-04-30T06:26:37.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51fb918b-cafd-4418-a15a-f4ea691d3bca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61222025-04-30T06:26:37.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
61232025-04-30T06:26:37.117ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cd5101e-2c92-4199-bef4-afcf48261cfc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61242025-04-30T06:26:37.117ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
61252025-04-30T06:26:37.118ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe21b5e-f0c5-4223-93ed-9296e8b82762", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61262025-04-30T06:26:37.118ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
61272025-04-30T06:26:37.119ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61282025-04-30T06:26:37.119ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61292025-04-30T06:26:37.119ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61302025-04-30T06:26:37.119ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61312025-04-30T06:26:37.121ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61322025-04-30T06:26:37.122ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d69ade5-dcac-4030-928d-fc357dda88d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61332025-04-30T06:26:37.122ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
61342025-04-30T06:26:37.124ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61352025-04-30T06:26:37.136ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61362025-04-30T06:26:37.137ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61372025-04-30T06:26:37.137ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61382025-04-30T06:26:37.139ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61392025-04-30T06:26:37.142ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "138f0fc0-70d1-45af-b09f-9c5d55c8c0c7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61402025-04-30T06:26:37.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
61412025-04-30T06:26:37.144ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61422025-04-30T06:26:37.145ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61432025-04-30T06:26:37.145ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61442025-04-30T06:26:37.195ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61452025-04-30T06:26:37.195ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61462025-04-30T06:26:37.201ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
61472025-04-30T06:26:37.202ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
61482025-04-30T06:26:37.207ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31c1676a-e3f5-45d8-a8ba-e9edea0d8900", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61492025-04-30T06:26:37.207ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
61502025-04-30T06:26:37.209ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
61512025-04-30T06:26:37.209ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
61522025-04-30T06:26:37.217ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
61532025-04-30T06:26:37.237ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61542025-04-30T06:26:37.243ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8adb5247-6364-4775-9451-8d4cb358b81c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61552025-04-30T06:26:37.244ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
61562025-04-30T06:26:37.249ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f778c77-4484-4561-b5c4-88ade2964a09", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61572025-04-30T06:26:37.249ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
61582025-04-30T06:26:37.251ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d1eaa19-4fe4-442b-9ce0-d186b11328e0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61592025-04-30T06:26:37.251ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
61602025-04-30T06:26:37.254ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f1636c5-2ea9-45b8-969d-48fbb34eb759", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61612025-04-30T06:26:37.254ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
61622025-04-30T06:26:37.258ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
61632025-04-30T06:26:37.261ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acbdbb4d-bb24-4bbf-a17e-e4749bb3598c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61642025-04-30T06:26:37.263ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
61652025-04-30T06:26:37.264ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8602a24d-eaaf-42fa-b1ff-1adf54eadd52", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61662025-04-30T06:26:37.265ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
61672025-04-30T06:26:37.269ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8188fd90-2835-4713-84df-caf370ae60cb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61682025-04-30T06:26:37.269ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
61692025-04-30T06:26:37.271ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aef9831-d07f-478e-a37a-c83ceedee0c9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61702025-04-30T06:26:37.271ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
61712025-04-30T06:26:37.274ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22554481-5d9a-466f-b7d6-9922d841b77e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
61722025-04-30T06:26:37.274ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61732025-04-30T06:26:37.275ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61742025-04-30T06:26:37.275ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61752025-04-30T06:26:37.275ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61762025-04-30T06:26:37.276ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61772025-04-30T06:26:37.284ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61782025-04-30T06:26:37.289ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61792025-04-30T06:26:37.289ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61802025-04-30T06:26:37.289ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61812025-04-30T06:26:37.289ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61822025-04-30T06:26:37.290ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61832025-04-30T06:26:37.290ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61842025-04-30T06:26:37.295ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61852025-04-30T06:26:37.301ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61862025-04-30T06:26:37.301ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61872025-04-30T06:26:37.301ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61882025-04-30T06:26:37.302ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61892025-04-30T06:26:37.302ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
61902025-04-30T06:26:37.309ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61912025-04-30T06:26:37.309ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61922025-04-30T06:26:37.309ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61932025-04-30T06:26:37.309ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61942025-04-30T06:26:37.309ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61952025-04-30T06:26:37.402ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
61962025-04-30T06:26:37.402ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
61972025-04-30T06:26:37.402ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:33930 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61982025-04-30T06:26:37.405ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61992025-04-30T06:26:37.414ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62002025-04-30T06:26:37.414ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62012025-04-30T06:26:37.414ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62022025-04-30T06:26:37.416ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62032025-04-30T06:26:37.416ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:33930 req_id = f96cb394-f063-4761-bccc-fc3df48ed638 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62042025-04-30T06:26:37.419ZINFOtest_disk_metrics (dropshot): request completed latency_us = 3838 local_addr = [::1]:50989 method = GET remote_addr = [::1]:33930 req_id = f96cb394-f063-4761-bccc-fc3df48ed638 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62052025-04-30T06:26:37.420ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f96cb394-f063-4761-bccc-fc3df48ed638", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62062025-04-30T06:26:37.424ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62072025-04-30T06:26:37.425ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62082025-04-30T06:26:37.425ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
62092025-04-30T06:26:37.432ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62102025-04-30T06:26:37.432ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62112025-04-30T06:26:37.432ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62122025-04-30T06:26:37.432ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62132025-04-30T06:26:37.435ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62142025-04-30T06:26:37.438ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62152025-04-30T06:26:37.443ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62162025-04-30T06:26:37.444ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62172025-04-30T06:26:37.444ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62182025-04-30T06:26:37.444ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62192025-04-30T06:26:37.445ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62202025-04-30T06:26:37.445ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62212025-04-30T06:26:37.446ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62222025-04-30T06:26:37.514ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62232025-04-30T06:26:37.515ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62242025-04-30T06:26:37.515ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 765.441832ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62252025-04-30T06:26:37.515ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
62262025-04-30T06:26:37.515ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62272025-04-30T06:26:37.515ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62282025-04-30T06:26:37.515ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62292025-04-30T06:26:37.515ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62302025-04-30T06:26:37.519ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62312025-04-30T06:26:37.520ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
62322025-04-30T06:26:37.522ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62332025-04-30T06:26:37.526ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62342025-04-30T06:26:37.527ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62352025-04-30T06:26:37.527ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62362025-04-30T06:26:37.528ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62372025-04-30T06:26:37.533ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
62382025-04-30T06:26:37.533ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62392025-04-30T06:26:37.534ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0a 3f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .?..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49860 req_id = 49ca76f4-ed30-4a72-8545-a3200bebc304
62402025-04-30T06:26:37.534ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2623,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49860 req_id = 49ca76f4-ed30-4a72-8545-a3200bebc304
62412025-04-30T06:26:37.534ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
62422025-04-30T06:26:37.534ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62432025-04-30T06:26:37.535ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49860 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 49ca76f4-ed30-4a72-8545-a3200bebc304
62442025-04-30T06:26:37.535ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3262, tv_nsec: 334380753 } })
62452025-04-30T06:26:37.535ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
62462025-04-30T06:26:37.535ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 05 5f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ._..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55597 req_id = a4c490cb-65d9-44cd-85b0-691059faf14c
62472025-04-30T06:26:37.535ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1375,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55597 req_id = a4c490cb-65d9-44cd-85b0-691059faf14c
62482025-04-30T06:26:37.535ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62492025-04-30T06:26:37.536ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55597 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a4c490cb-65d9-44cd-85b0-691059faf14c
62502025-04-30T06:26:37.607ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62512025-04-30T06:26:37.608ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62522025-04-30T06:26:37.608ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62532025-04-30T06:26:37.608ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62542025-04-30T06:26:37.608ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62552025-04-30T06:26:37.608ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62562025-04-30T06:26:37.609ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c0 cb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50747 req_id = cb007742-a7b6-4a20-bbaf-14834b77e2af
62572025-04-30T06:26:37.609ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49355,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50747 req_id = cb007742-a7b6-4a20-bbaf-14834b77e2af
62582025-04-30T06:26:37.609ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
62592025-04-30T06:26:37.609ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49355, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50747 req_id = cb007742-a7b6-4a20-bbaf-14834b77e2af
62602025-04-30T06:26:37.610ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
62612025-04-30T06:26:37.610ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
62622025-04-30T06:26:37.610ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
62632025-04-30T06:26:37.611ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62642025-04-30T06:26:37.618ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62652025-04-30T06:26:37.619ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62662025-04-30T06:26:37.619ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62672025-04-30T06:26:37.619ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62682025-04-30T06:26:37.630ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62692025-04-30T06:26:37.630ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62702025-04-30T06:26:37.631ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62712025-04-30T06:26:37.632ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed5d6773-a37a-4ba3-ace4-902859445d8f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62722025-04-30T06:26:37.632ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
62732025-04-30T06:26:37.635ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10ef9952-cd8c-45f4-a656-bb81d6fd2c59", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62742025-04-30T06:26:37.635ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
62752025-04-30T06:26:37.638ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "350a8662-4219-4431-b974-aee4b7d15193", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62762025-04-30T06:26:37.638ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
62772025-04-30T06:26:37.643ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6477c14a-4610-4f44-beb0-17945dcbd7eb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62782025-04-30T06:26:37.643ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
62792025-04-30T06:26:37.647ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "301b7397-dfc8-4bfa-9622-7821df9242a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62802025-04-30T06:26:37.647ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
62812025-04-30T06:26:37.649ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7293cb9-aff1-492b-8226-ee40a8b99622", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62822025-04-30T06:26:37.649ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
62832025-04-30T06:26:37.651ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35e99fd3-9af9-4740-8261-b3bfb79c4c9c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62842025-04-30T06:26:37.651ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
62852025-04-30T06:26:37.654ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88dda8d1-ac8d-4f55-9939-86ef7ee6f00c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62862025-04-30T06:26:37.654ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
62872025-04-30T06:26:37.655ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "def67e9c-00c0-490f-835d-c798fd42c2dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62882025-04-30T06:26:37.656ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
62892025-04-30T06:26:37.657ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36ffe01f-dad7-4262-bf4e-5ab931098a07", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62902025-04-30T06:26:37.659ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
62912025-04-30T06:26:37.664ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d365ad7-4a52-44dc-b7c6-49c68d30be45", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62922025-04-30T06:26:37.664ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
62932025-04-30T06:26:37.669ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a186028a-1e5c-47ac-9f83-9f6b3fa9f927", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62942025-04-30T06:26:37.669ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
62952025-04-30T06:26:37.671ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
62962025-04-30T06:26:37.675ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14a6390b-09cf-423b-9e91-782f2c75a5f3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62972025-04-30T06:26:37.675ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
62982025-04-30T06:26:37.680ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae728106-349c-45ab-927f-5332ac03222b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
62992025-04-30T06:26:37.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
63002025-04-30T06:26:37.688ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b166e127-8136-49dc-b2c3-5ae788ca6291", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63012025-04-30T06:26:37.688ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
63022025-04-30T06:26:37.694ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebb8fc54-f8b1-4767-bde6-3010acb5b051", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63032025-04-30T06:26:37.704ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
63042025-04-30T06:26:37.710ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b45ca139-cde1-4279-81ff-d2f2a3b090af", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63052025-04-30T06:26:37.710ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
63062025-04-30T06:26:37.714ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "532a1208-eced-4755-8f0f-81453aa93c0c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63072025-04-30T06:26:37.714ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
63082025-04-30T06:26:37.718ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5052fde2-618a-4b7a-8b2b-e2a105a5ec91", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63092025-04-30T06:26:37.718ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
63102025-04-30T06:26:37.719ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e78f6b4-90eb-413f-ace7-4bbd76c6e557", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63112025-04-30T06:26:37.719ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
63122025-04-30T06:26:37.721ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03d31995-10e0-4721-91ba-6f4775234733", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63132025-04-30T06:26:37.721ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
63142025-04-30T06:26:37.722ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63152025-04-30T06:26:37.722ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63162025-04-30T06:26:37.722ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63172025-04-30T06:26:37.722ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63182025-04-30T06:26:37.723ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63192025-04-30T06:26:37.724ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66fbb65a-4d14-4a32-b742-745c6cedbb84", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63202025-04-30T06:26:37.724ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
63212025-04-30T06:26:37.726ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63222025-04-30T06:26:37.739ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63232025-04-30T06:26:37.739ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63242025-04-30T06:26:37.739ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63252025-04-30T06:26:37.740ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63262025-04-30T06:26:37.741ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89c6d230-5f6c-4ccf-b592-a5672e9032ab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63272025-04-30T06:26:37.741ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
63282025-04-30T06:26:37.745ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63292025-04-30T06:26:37.745ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63302025-04-30T06:26:37.745ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63312025-04-30T06:26:37.745ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f72a752d-bfe5-4b7d-b621-d117061b6059", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63322025-04-30T06:26:37.746ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
63332025-04-30T06:26:37.749ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc1a1c9-773d-44a3-9807-51afbc672a8e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63342025-04-30T06:26:37.749ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
63352025-04-30T06:26:37.753ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "548ff72f-2443-450c-9c56-aed8774704f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63362025-04-30T06:26:37.754ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
63372025-04-30T06:26:37.759ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10e435c-3f5d-43c1-a6b8-0a8d05d1477a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63382025-04-30T06:26:37.759ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
63392025-04-30T06:26:37.766ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "788f79c4-d7c9-40d4-aade-4b2dc2bd3c10", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63402025-04-30T06:26:37.766ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
63412025-04-30T06:26:37.776ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c16159c0-a04f-4eaf-8b1a-c125cd949463", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63422025-04-30T06:26:37.776ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
63432025-04-30T06:26:37.785ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
63442025-04-30T06:26:37.796ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c87205db-1bee-4b84-8010-539fec72edce", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63452025-04-30T06:26:37.797ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
63462025-04-30T06:26:37.797ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a924d7a4-e7a0-4c92-86d5-c7431e415806", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63472025-04-30T06:26:37.798ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
63482025-04-30T06:26:37.799ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a87c402-a58d-4fce-86e0-67dd690a5a7c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63492025-04-30T06:26:37.799ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
63502025-04-30T06:26:37.800ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d01402dc-9199-43bf-8b98-419d8b56751f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63512025-04-30T06:26:37.801ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
63522025-04-30T06:26:37.802ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dec74e56-fdf9-4e48-a862-6ac97078d95e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63532025-04-30T06:26:37.802ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
63542025-04-30T06:26:37.804ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3881e1-4c20-4de7-bd9c-aa03f4680aa9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63552025-04-30T06:26:37.804ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
63562025-04-30T06:26:37.807ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48274a5f-a848-43e8-bd02-22008745495a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63572025-04-30T06:26:37.807ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
63582025-04-30T06:26:37.809ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47555ece-4b72-4ee2-b3c4-f6f9d5315de3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63592025-04-30T06:26:37.809ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
63602025-04-30T06:26:37.813ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bd34f79-f5d4-4440-a068-b85d9e1f3dfb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63612025-04-30T06:26:37.814ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
63622025-04-30T06:26:37.816ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "647b8667-b1ab-4ec4-b042-1e7bb9663ce7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63632025-04-30T06:26:37.817ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
63642025-04-30T06:26:37.819ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6350b3f4-14e7-469f-af47-a14267e60fdd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63652025-04-30T06:26:37.819ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
63662025-04-30T06:26:37.822ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5428673b-637c-4f21-bdcb-3c6ffc27837f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63672025-04-30T06:26:37.822ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
63682025-04-30T06:26:37.827ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9787421-0502-463c-be09-6ef7e3478a78", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63692025-04-30T06:26:37.827ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
63702025-04-30T06:26:37.828ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "830d76fb-500d-46ab-bfc5-3b20677dadb3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63712025-04-30T06:26:37.829ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
63722025-04-30T06:26:37.830ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31d00f5e-69e9-4b93-9a48-be100c6a0430", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63732025-04-30T06:26:37.830ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
63742025-04-30T06:26:37.831ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f983923-de32-4a85-b6b6-447baa1cca86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
63752025-04-30T06:26:37.832ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63762025-04-30T06:26:37.832ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63772025-04-30T06:26:37.832ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63782025-04-30T06:26:37.832ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63792025-04-30T06:26:37.833ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63802025-04-30T06:26:37.840ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63812025-04-30T06:26:37.843ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63822025-04-30T06:26:37.843ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63832025-04-30T06:26:37.843ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63842025-04-30T06:26:37.843ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63852025-04-30T06:26:37.843ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63862025-04-30T06:26:37.844ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63872025-04-30T06:26:37.844ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63882025-04-30T06:26:37.845ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63892025-04-30T06:26:37.845ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63902025-04-30T06:26:37.845ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63912025-04-30T06:26:37.846ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63922025-04-30T06:26:37.848ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63932025-04-30T06:26:37.856ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63942025-04-30T06:26:37.857ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63952025-04-30T06:26:37.857ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63962025-04-30T06:26:37.858ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63972025-04-30T06:26:37.871ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63982025-04-30T06:26:37.876ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63992025-04-30T06:26:37.876ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64002025-04-30T06:26:37.876ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64012025-04-30T06:26:37.877ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64022025-04-30T06:26:37.955ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
64032025-04-30T06:26:37.955ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
64042025-04-30T06:26:37.956ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:64496 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
64052025-04-30T06:26:37.959ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64062025-04-30T06:26:37.959ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64072025-04-30T06:26:37.960ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64082025-04-30T06:26:37.963ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:64496 req_id = c2c7b79e-3fad-4f34-b51e-c77e7c1a1c3f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64092025-04-30T06:26:37.965ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1883 local_addr = [::1]:50989 method = GET remote_addr = [::1]:64496 req_id = c2c7b79e-3fad-4f34-b51e-c77e7c1a1c3f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64102025-04-30T06:26:37.967ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2c7b79e-3fad-4f34-b51e-c77e7c1a1c3f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
64112025-04-30T06:26:37.969ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64122025-04-30T06:26:37.969ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64132025-04-30T06:26:37.969ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
64142025-04-30T06:26:37.973ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64152025-04-30T06:26:37.974ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64162025-04-30T06:26:37.974ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64172025-04-30T06:26:37.974ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64182025-04-30T06:26:37.975ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64192025-04-30T06:26:37.983ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64202025-04-30T06:26:37.987ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64212025-04-30T06:26:37.987ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64222025-04-30T06:26:37.987ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64232025-04-30T06:26:37.988ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64242025-04-30T06:26:37.990ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64252025-04-30T06:26:37.991ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64262025-04-30T06:26:37.997ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64272025-04-30T06:26:37.998ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64282025-04-30T06:26:37.998ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64292025-04-30T06:26:37.998ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64302025-04-30T06:26:38.040ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64312025-04-30T06:26:38.041ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64322025-04-30T06:26:38.041ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 525.978999ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64332025-04-30T06:26:38.041ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64342025-04-30T06:26:38.041ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64352025-04-30T06:26:38.041ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64362025-04-30T06:26:38.041ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64372025-04-30T06:26:38.041ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64382025-04-30T06:26:38.042ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64392025-04-30T06:26:38.045ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64402025-04-30T06:26:38.068ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64412025-04-30T06:26:38.068ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64422025-04-30T06:26:38.068ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64432025-04-30T06:26:38.068ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64442025-04-30T06:26:38.071ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
64452025-04-30T06:26:38.074ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
64462025-04-30T06:26:38.074ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64472025-04-30T06:26:38.076ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 21 f2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 !............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51356 req_id = c6856bf3-2211-40e7-9d74-b370489881ff
64482025-04-30T06:26:38.077ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8690,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51356 req_id = c6856bf3-2211-40e7-9d74-b370489881ff
64492025-04-30T06:26:38.077ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
64502025-04-30T06:26:38.077ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64512025-04-30T06:26:38.077ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51356 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c6856bf3-2211-40e7-9d74-b370489881ff
64522025-04-30T06:26:38.077ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3262, tv_nsec: 876647674 } })
64532025-04-30T06:26:38.077ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
64542025-04-30T06:26:38.078ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6d 2c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e m,..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61191 req_id = fdfcd71f-9186-462c-90a1-f11cd6177a79
64552025-04-30T06:26:38.078ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27948,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61191 req_id = fdfcd71f-9186-462c-90a1-f11cd6177a79
64562025-04-30T06:26:38.078ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64572025-04-30T06:26:38.078ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61191 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fdfcd71f-9186-462c-90a1-f11cd6177a79
64582025-04-30T06:26:38.149ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64592025-04-30T06:26:38.150ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64602025-04-30T06:26:38.150ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64612025-04-30T06:26:38.150ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64622025-04-30T06:26:38.150ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64632025-04-30T06:26:38.152ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64642025-04-30T06:26:38.152ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f8 1a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50244 req_id = 94f74918-a9ce-4e69-b646-56b6a8c06605
64652025-04-30T06:26:38.152ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63514,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50244 req_id = 94f74918-a9ce-4e69-b646-56b6a8c06605
64662025-04-30T06:26:38.152ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
64672025-04-30T06:26:38.152ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63514, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50244 req_id = 94f74918-a9ce-4e69-b646-56b6a8c06605
64682025-04-30T06:26:38.153ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
64692025-04-30T06:26:38.153ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
64702025-04-30T06:26:38.153ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
64712025-04-30T06:26:38.156ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64722025-04-30T06:26:38.165ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64732025-04-30T06:26:38.165ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64742025-04-30T06:26:38.167ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64752025-04-30T06:26:38.169ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64762025-04-30T06:26:38.169ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dede494-0cc7-4ffe-a8ed-714f75bfdc71", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
64772025-04-30T06:26:38.170ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
64782025-04-30T06:26:38.174ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64792025-04-30T06:26:38.174ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64802025-04-30T06:26:38.175ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64812025-04-30T06:26:38.176ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
64822025-04-30T06:26:38.176ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
64832025-04-30T06:26:38.176ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
64842025-04-30T06:26:38.178ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
64852025-04-30T06:26:38.179ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08c485f3-abef-4c93-9bea-84850c529270", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
64862025-04-30T06:26:38.179ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
64872025-04-30T06:26:38.183ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
64882025-04-30T06:26:38.183ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
64892025-04-30T06:26:38.190ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
64902025-04-30T06:26:38.190ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
64912025-04-30T06:26:38.193ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0be2fccb-0c93-4d5a-8249-b54180d19ccd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
64922025-04-30T06:26:38.193ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
64932025-04-30T06:26:38.198ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25ac7714-7141-4455-bd13-17cd090609b1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
64942025-04-30T06:26:38.198ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
64952025-04-30T06:26:38.209ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bca94ca-861d-484c-a17a-1817f0d8df22", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
64962025-04-30T06:26:38.209ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
64972025-04-30T06:26:38.213ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
64982025-04-30T06:26:38.216ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e16222b0-796e-48f3-a198-307667713224", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
64992025-04-30T06:26:38.217ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
65002025-04-30T06:26:38.219ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ef414e-d1aa-4acc-9c77-c49b069d2328", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65012025-04-30T06:26:38.219ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
65022025-04-30T06:26:38.221ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "678cd3aa-ba41-4368-8fcd-00da968488f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65032025-04-30T06:26:38.221ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
65042025-04-30T06:26:38.223ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e6be763-216b-4e53-b718-ab787d3c7684", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65052025-04-30T06:26:38.223ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
65062025-04-30T06:26:38.224ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea3c6809-e222-4b73-8c99-59e79adc65d5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65072025-04-30T06:26:38.225ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
65082025-04-30T06:26:38.226ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3a99212-ab3c-4d14-875e-fb15a6b1ad50", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65092025-04-30T06:26:38.227ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
65102025-04-30T06:26:38.229ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cc3154c-2581-4e7b-a287-e337442ff2a7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65112025-04-30T06:26:38.229ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
65122025-04-30T06:26:38.236ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d18be60-3f9f-450d-8c64-24abcbb4348f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65132025-04-30T06:26:38.236ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
65142025-04-30T06:26:38.244ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8091f8d-0657-4efa-ba30-60575d144e97", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65152025-04-30T06:26:38.244ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
65162025-04-30T06:26:38.248ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d95f302a-6f89-46d0-b724-189bfd6bad81", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65172025-04-30T06:26:38.248ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
65182025-04-30T06:26:38.251ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2937a89-9bc9-4e8a-bd58-b515099b881e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65192025-04-30T06:26:38.251ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
65202025-04-30T06:26:38.258ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eae9e0e-0f8e-4b3e-a307-6041cf07b1a0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65212025-04-30T06:26:38.258ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
65222025-04-30T06:26:38.262ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aaf15c5-05aa-456f-a851-c14a83fc3ae0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65232025-04-30T06:26:38.262ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
65242025-04-30T06:26:38.265ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65252025-04-30T06:26:38.265ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65262025-04-30T06:26:38.265ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65272025-04-30T06:26:38.265ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
65282025-04-30T06:26:38.266ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65292025-04-30T06:26:38.269ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5d80af9-656d-4f60-88f9-8bf896d22885", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65302025-04-30T06:26:38.269ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
65312025-04-30T06:26:38.269ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65322025-04-30T06:26:38.280ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65332025-04-30T06:26:38.280ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65342025-04-30T06:26:38.281ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
65352025-04-30T06:26:38.286ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65362025-04-30T06:26:38.289ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
65372025-04-30T06:26:38.289ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a40f4b9-c08e-4f9d-9959-79b8192697ed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65382025-04-30T06:26:38.290ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
65392025-04-30T06:26:38.293ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65402025-04-30T06:26:38.293ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
65412025-04-30T06:26:38.294ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65422025-04-30T06:26:38.295ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf6764df-678d-4422-a905-d71d8525f2a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65432025-04-30T06:26:38.295ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
65442025-04-30T06:26:38.296ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1eaa9120-8d41-4e81-af74-e5541417d433", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65452025-04-30T06:26:38.297ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
65462025-04-30T06:26:38.298ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a7fd3c1-67da-4c46-9893-ef1bfc058a4b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65472025-04-30T06:26:38.298ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
65482025-04-30T06:26:38.300ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ec7f805-381b-4156-85a7-e149267df6e5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65492025-04-30T06:26:38.300ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
65502025-04-30T06:26:38.302ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df45061c-d2ce-4540-b28f-cc54b340b919", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65512025-04-30T06:26:38.303ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
65522025-04-30T06:26:38.306ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4179dc55-1e4d-40a3-b53e-bc161c4fdb98", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65532025-04-30T06:26:38.306ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
65542025-04-30T06:26:38.310ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1ad138e-4b4d-43bb-bb38-b20f550af2ea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65552025-04-30T06:26:38.310ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
65562025-04-30T06:26:38.315ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b536a82-31e6-4fa6-bbc0-a75fb5a48564", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65572025-04-30T06:26:38.316ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
65582025-04-30T06:26:38.318ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec543d86-c387-4612-92bf-e8fe0ad6433f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65592025-04-30T06:26:38.318ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
65602025-04-30T06:26:38.320ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8aa754c-2e94-4dd8-82d0-be831b500184", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65612025-04-30T06:26:38.321ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
65622025-04-30T06:26:38.322ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be7c908e-2a31-4e77-9126-2327d8eb6906", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65632025-04-30T06:26:38.322ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
65642025-04-30T06:26:38.322ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b6aa695-6f7b-4e55-a821-b529efc9eea0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65652025-04-30T06:26:38.323ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
65662025-04-30T06:26:38.324ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea92804d-1488-4f78-be25-7e06917ef496", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65672025-04-30T06:26:38.324ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
65682025-04-30T06:26:38.325ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3a643f7-ad78-41be-829d-5ed33d472596", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65692025-04-30T06:26:38.325ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
65702025-04-30T06:26:38.327ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ff32dac-fcb3-4174-b2ac-08c7d6abe0d6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65712025-04-30T06:26:38.327ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
65722025-04-30T06:26:38.330ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60c1e057-83ca-4209-b4aa-3d8ca53f5051", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65732025-04-30T06:26:38.330ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
65742025-04-30T06:26:38.333ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
65752025-04-30T06:26:38.335ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52703832-6f56-4f51-b4fb-c2f44985536f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65762025-04-30T06:26:38.335ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
65772025-04-30T06:26:38.339ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f43a82f-046b-41ec-9a59-f0e670c93e2c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65782025-04-30T06:26:38.339ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
65792025-04-30T06:26:38.341ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32c3e29f-ae80-4c0f-892c-742f853369e1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65802025-04-30T06:26:38.341ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
65812025-04-30T06:26:38.344ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc378715-623d-4a8b-9824-0308118a2e76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65822025-04-30T06:26:38.344ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
65832025-04-30T06:26:38.346ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53137cc1-d948-42db-b80d-f38dc5df1191", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65842025-04-30T06:26:38.346ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
65852025-04-30T06:26:38.347ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5302dbb2-4685-40d2-8299-5c7aa5ee91c3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65862025-04-30T06:26:38.347ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
65872025-04-30T06:26:38.348ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05010e3b-9df9-41e1-9ad9-54cce90f2c32", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65882025-04-30T06:26:38.348ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
65892025-04-30T06:26:38.349ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ccd15c4-ea1c-4503-861f-65d2c71d3fe9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65902025-04-30T06:26:38.349ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
65912025-04-30T06:26:38.350ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e562c6cf-59b6-4dbc-b125-a15e18e79b83", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
65922025-04-30T06:26:38.350ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65932025-04-30T06:26:38.350ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65942025-04-30T06:26:38.350ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65952025-04-30T06:26:38.350ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65962025-04-30T06:26:38.351ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65972025-04-30T06:26:38.354ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65982025-04-30T06:26:38.364ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65992025-04-30T06:26:38.365ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66002025-04-30T06:26:38.365ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66012025-04-30T06:26:38.365ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66022025-04-30T06:26:38.365ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66032025-04-30T06:26:38.365ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66042025-04-30T06:26:38.369ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66052025-04-30T06:26:38.377ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66062025-04-30T06:26:38.377ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66072025-04-30T06:26:38.377ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66082025-04-30T06:26:38.379ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66092025-04-30T06:26:38.451ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
66102025-04-30T06:26:38.451ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
66112025-04-30T06:26:38.452ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66122025-04-30T06:26:38.452ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66132025-04-30T06:26:38.452ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66142025-04-30T06:26:38.452ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66152025-04-30T06:26:38.453ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66162025-04-30T06:26:38.453ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:51780 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
66172025-04-30T06:26:38.456ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:51780 req_id = 9fb483cd-5658-4123-bfd4-3cf84343189c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66182025-04-30T06:26:38.457ZINFOtest_disk_metrics (dropshot): request completed latency_us = 480 local_addr = [::1]:50989 method = GET remote_addr = [::1]:51780 req_id = 9fb483cd-5658-4123-bfd4-3cf84343189c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66192025-04-30T06:26:38.457ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fb483cd-5658-4123-bfd4-3cf84343189c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
66202025-04-30T06:26:38.461ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66212025-04-30T06:26:38.461ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66222025-04-30T06:26:38.461ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
66232025-04-30T06:26:38.466ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66242025-04-30T06:26:38.466ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66252025-04-30T06:26:38.466ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66262025-04-30T06:26:38.466ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66272025-04-30T06:26:38.466ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66282025-04-30T06:26:38.467ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66292025-04-30T06:26:38.473ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66302025-04-30T06:26:38.473ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66312025-04-30T06:26:38.474ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66322025-04-30T06:26:38.475ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66332025-04-30T06:26:38.477ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66342025-04-30T06:26:38.488ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66352025-04-30T06:26:38.488ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66362025-04-30T06:26:38.488ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66372025-04-30T06:26:38.490ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66382025-04-30T06:26:38.496ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66392025-04-30T06:26:38.497ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66402025-04-30T06:26:38.498ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66412025-04-30T06:26:38.552ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66422025-04-30T06:26:38.552ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66432025-04-30T06:26:38.552ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 511.212989ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66442025-04-30T06:26:38.552ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66452025-04-30T06:26:38.552ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66462025-04-30T06:26:38.552ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66472025-04-30T06:26:38.552ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66482025-04-30T06:26:38.552ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66492025-04-30T06:26:38.554ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66502025-04-30T06:26:38.561ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66512025-04-30T06:26:38.565ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66522025-04-30T06:26:38.565ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66532025-04-30T06:26:38.565ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66542025-04-30T06:26:38.565ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66552025-04-30T06:26:38.566ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
66562025-04-30T06:26:38.570ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
66572025-04-30T06:26:38.570ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66582025-04-30T06:26:38.571ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 39 e3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 9............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53074 req_id = 9b460161-5ede-4d29-a0de-2810263015fd
66592025-04-30T06:26:38.571ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14819,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53074 req_id = 9b460161-5ede-4d29-a0de-2810263015fd
66602025-04-30T06:26:38.571ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
66612025-04-30T06:26:38.571ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66622025-04-30T06:26:38.571ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53074 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9b460161-5ede-4d29-a0de-2810263015fd
66632025-04-30T06:26:38.572ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3263, tv_nsec: 371173365 } })
66642025-04-30T06:26:38.572ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
66652025-04-30T06:26:38.572ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 91 c0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57739 req_id = a7457fd0-4a8d-4ac1-9571-0e15c8048454
66662025-04-30T06:26:38.572ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37312,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57739 req_id = a7457fd0-4a8d-4ac1-9571-0e15c8048454
66672025-04-30T06:26:38.573ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66682025-04-30T06:26:38.573ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57739 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a7457fd0-4a8d-4ac1-9571-0e15c8048454
66692025-04-30T06:26:38.637ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66702025-04-30T06:26:38.638ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66712025-04-30T06:26:38.638ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66722025-04-30T06:26:38.638ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66732025-04-30T06:26:38.638ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66742025-04-30T06:26:38.639ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66752025-04-30T06:26:38.641ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 54 e9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c T............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63331 req_id = 305b92f2-d3ae-426b-9f16-404fa7fdf1ea
66762025-04-30T06:26:38.641ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21737,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63331 req_id = 305b92f2-d3ae-426b-9f16-404fa7fdf1ea
66772025-04-30T06:26:38.641ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
66782025-04-30T06:26:38.641ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21737, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63331 req_id = 305b92f2-d3ae-426b-9f16-404fa7fdf1ea
66792025-04-30T06:26:38.643ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
66802025-04-30T06:26:38.643ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
66812025-04-30T06:26:38.643ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
66822025-04-30T06:26:38.645ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66832025-04-30T06:26:38.658ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66842025-04-30T06:26:38.658ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66852025-04-30T06:26:38.658ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66862025-04-30T06:26:38.659ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66872025-04-30T06:26:38.663ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f37bccd5-e425-4d55-bc6d-900558f18f45", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
66882025-04-30T06:26:38.663ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
66892025-04-30T06:26:38.664ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66902025-04-30T06:26:38.664ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66912025-04-30T06:26:38.665ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66922025-04-30T06:26:38.667ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04be0241-b8a7-4a7e-8e40-b84fa6c0db9a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
66932025-04-30T06:26:38.667ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
66942025-04-30T06:26:38.672ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9e22a3d-d598-4447-8965-fd02542c13e9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
66952025-04-30T06:26:38.672ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
66962025-04-30T06:26:38.675ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "713edb5d-b12d-4078-b346-ddab840050ac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
66972025-04-30T06:26:38.675ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
66982025-04-30T06:26:38.680ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e571f8c4-6e28-4a37-97eb-decfd3938ca0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
66992025-04-30T06:26:38.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
67002025-04-30T06:26:38.685ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85c2c0fa-39f3-499a-80c2-242957d4fab2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67012025-04-30T06:26:38.685ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
67022025-04-30T06:26:38.689ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4a54314-310d-46b3-bca9-dfcd3b7201b2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67032025-04-30T06:26:38.689ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
67042025-04-30T06:26:38.694ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ef91b98-bf9c-46bd-a688-f2bc8fda22ca", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67052025-04-30T06:26:38.694ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
67062025-04-30T06:26:38.698ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bb75651-7727-4353-84f1-ea8154e275de", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67072025-04-30T06:26:38.698ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
67082025-04-30T06:26:38.699ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
67092025-04-30T06:26:38.701ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50a7b766-bf2a-45b4-8a72-279c0890858e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67102025-04-30T06:26:38.701ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
67112025-04-30T06:26:38.703ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d58b00a5-12ea-419f-9056-3c0075eafd1c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67122025-04-30T06:26:38.703ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
67132025-04-30T06:26:38.705ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6100e87f-76db-46ea-8942-5aa03dca2a75", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67142025-04-30T06:26:38.705ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
67152025-04-30T06:26:38.708ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aedfc75-bc69-4766-ad27-76746b55fbc7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67162025-04-30T06:26:38.708ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
67172025-04-30T06:26:38.711ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c5bdf6e-6ab9-45e5-9b7b-94a43e7b4db8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67182025-04-30T06:26:38.712ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
67192025-04-30T06:26:38.715ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "923609e0-66b6-4559-ab8a-a461224f2c94", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67202025-04-30T06:26:38.715ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
67212025-04-30T06:26:38.720ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2babfb40-5d1d-4835-bca6-bc5c39f1e2c9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67222025-04-30T06:26:38.720ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
67232025-04-30T06:26:38.724ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa900011-b89d-4a6d-9fd4-e7cd790813b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67242025-04-30T06:26:38.724ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
67252025-04-30T06:26:38.735ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48baa344-2924-48ca-835f-d342f83ce070", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67262025-04-30T06:26:38.736ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
67272025-04-30T06:26:38.742ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4a6927e-e4e8-4f13-9842-19d35655b56c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67282025-04-30T06:26:38.742ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
67292025-04-30T06:26:38.744ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f20eeff8-3066-4c9b-b9b6-db7544f1eef7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67302025-04-30T06:26:38.745ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
67312025-04-30T06:26:38.746ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a875a1a-d239-4f24-b571-6affe193ae81", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67322025-04-30T06:26:38.747ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
67332025-04-30T06:26:38.748ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc6954ff-57c5-4c86-8943-c9342335bb3a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67342025-04-30T06:26:38.748ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
67352025-04-30T06:26:38.750ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67362025-04-30T06:26:38.750ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67372025-04-30T06:26:38.750ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67382025-04-30T06:26:38.750ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
67392025-04-30T06:26:38.751ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67402025-04-30T06:26:38.751ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a76beee-cc9a-4418-9164-2c1e97a77083", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67412025-04-30T06:26:38.751ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
67422025-04-30T06:26:38.754ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b9a1722-63f9-4c32-8104-34e94ac80ee7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67432025-04-30T06:26:38.754ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
67442025-04-30T06:26:38.755ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67452025-04-30T06:26:38.764ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67462025-04-30T06:26:38.764ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67472025-04-30T06:26:38.764ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
67482025-04-30T06:26:38.765ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67492025-04-30T06:26:38.768ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67502025-04-30T06:26:38.768ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
67512025-04-30T06:26:38.769ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67522025-04-30T06:26:38.769ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e6ac386-7989-4377-b159-3ff7070277a0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67532025-04-30T06:26:38.769ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
67542025-04-30T06:26:38.771ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bbcb71f-729e-4729-8c7b-4e1a1095e9df", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67552025-04-30T06:26:38.772ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
67562025-04-30T06:26:38.776ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cff09d4d-23ad-4370-9b95-07892f4a4eb0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67572025-04-30T06:26:38.776ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
67582025-04-30T06:26:38.779ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10314479-5a59-44fb-97ea-317fb3905b10", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67592025-04-30T06:26:38.779ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
67602025-04-30T06:26:38.788ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9d3667e-6543-40d4-a1ee-0afd3534db74", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67612025-04-30T06:26:38.788ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
67622025-04-30T06:26:38.790ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d04f69a-4eac-4ac0-85aa-1c32baf2fb2f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67632025-04-30T06:26:38.790ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
67642025-04-30T06:26:38.793ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f5fc39a-4201-41b2-9203-3543dfc18c7e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67652025-04-30T06:26:38.793ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
67662025-04-30T06:26:38.794ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0af723ca-1348-46b2-9f11-de486860ca33", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67672025-04-30T06:26:38.794ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
67682025-04-30T06:26:38.795ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0497659c-e50f-4049-9588-d2665c0d211c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67692025-04-30T06:26:38.796ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
67702025-04-30T06:26:38.797ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0fc5c28-af62-4e45-8596-7f1ff8c07a64", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67712025-04-30T06:26:38.797ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
67722025-04-30T06:26:38.798ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "461a502c-4946-4b3f-bd79-2bfa8bbb613f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67732025-04-30T06:26:38.799ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
67742025-04-30T06:26:38.802ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
67752025-04-30T06:26:38.802ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9508e086-fcf9-448f-9e75-07356b9285b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67762025-04-30T06:26:38.803ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
67772025-04-30T06:26:38.807ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a398611-32eb-48b7-8d59-0455c9902c76", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67782025-04-30T06:26:38.807ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
67792025-04-30T06:26:38.810ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "179ef11b-b798-4bad-8859-a4c61a1faa7f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67802025-04-30T06:26:38.810ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
67812025-04-30T06:26:38.813ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b020fd4-cb8e-4316-bf1d-3ef646c8993a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67822025-04-30T06:26:38.813ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
67832025-04-30T06:26:38.816ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f71f15af-b016-42e2-8991-f119c6115968", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67842025-04-30T06:26:38.816ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
67852025-04-30T06:26:38.818ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7197ae9f-aca1-4a8f-b247-584438f7d48d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67862025-04-30T06:26:38.818ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
67872025-04-30T06:26:38.819ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cf0fd78-df9e-4135-8a53-bbea4bb8d9f3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67882025-04-30T06:26:38.819ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
67892025-04-30T06:26:38.820ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67d6c004-9787-4a71-89af-360c4207075f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67902025-04-30T06:26:38.820ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
67912025-04-30T06:26:38.822ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "551f25b9-918e-4150-9a95-6a27fa0880b6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67922025-04-30T06:26:38.822ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
67932025-04-30T06:26:38.823ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6f72196-a2dc-463d-a776-efbbb73f4b94", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
67942025-04-30T06:26:38.823ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67952025-04-30T06:26:38.823ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67962025-04-30T06:26:38.823ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67972025-04-30T06:26:38.823ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67982025-04-30T06:26:38.824ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67992025-04-30T06:26:38.826ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68002025-04-30T06:26:38.830ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68012025-04-30T06:26:38.830ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68022025-04-30T06:26:38.830ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68032025-04-30T06:26:38.830ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68042025-04-30T06:26:38.830ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68052025-04-30T06:26:38.830ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68062025-04-30T06:26:38.835ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68072025-04-30T06:26:38.839ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68082025-04-30T06:26:38.840ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68092025-04-30T06:26:38.840ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68102025-04-30T06:26:38.840ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68112025-04-30T06:26:38.904ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
68122025-04-30T06:26:38.904ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
68132025-04-30T06:26:38.906ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68142025-04-30T06:26:38.906ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68152025-04-30T06:26:38.906ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68162025-04-30T06:26:38.906ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
68172025-04-30T06:26:38.906ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68182025-04-30T06:26:38.906ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:45513 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
68192025-04-30T06:26:38.907ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
68202025-04-30T06:26:38.908ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68212025-04-30T06:26:38.908ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 3a055204-df38-4a16-be59-642977de7cd2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68222025-04-30T06:26:38.908ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:45513 req_id = 04c9ec3f-f1b2-47f4-a529-cc0108c36a48 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68232025-04-30T06:26:38.909ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 3a055204-df38-4a16-be59-642977de7cd2 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68242025-04-30T06:26:38.909ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 3a055204-df38-4a16-be59-642977de7cd2 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68252025-04-30T06:26:38.910ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 3a055204-df38-4a16-be59-642977de7cd2 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68262025-04-30T06:26:38.910ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1957 local_addr = [::1]:50989 method = GET remote_addr = [::1]:45513 req_id = 04c9ec3f-f1b2-47f4-a529-cc0108c36a48 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68272025-04-30T06:26:38.912ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "04c9ec3f-f1b2-47f4-a529-cc0108c36a48", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
68282025-04-30T06:26:38.914ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68292025-04-30T06:26:38.914ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68302025-04-30T06:26:38.914ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
68312025-04-30T06:26:38.916ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68322025-04-30T06:26:38.916ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68332025-04-30T06:26:38.916ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68342025-04-30T06:26:38.916ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68352025-04-30T06:26:38.918ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68362025-04-30T06:26:38.918ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68372025-04-30T06:26:38.929ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68382025-04-30T06:26:38.929ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68392025-04-30T06:26:38.929ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
68402025-04-30T06:26:38.930ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68412025-04-30T06:26:38.932ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 23614 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 3a055204-df38-4a16-be59-642977de7cd2 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
68422025-04-30T06:26:38.933ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68432025-04-30T06:26:38.940ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68442025-04-30T06:26:38.940ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68452025-04-30T06:26:38.940ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68462025-04-30T06:26:38.940ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68472025-04-30T06:26:38.943ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a055204-df38-4a16-be59-642977de7cd2", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
68482025-04-30T06:26:38.945ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
68492025-04-30T06:26:38.947ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68502025-04-30T06:26:38.947ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
68512025-04-30T06:26:38.948ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68522025-04-30T06:26:38.949ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 944d3eec-76e5-4883-96e7-0a2185622649 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
68532025-04-30T06:26:38.950ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 944d3eec-76e5-4883-96e7-0a2185622649 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
68542025-04-30T06:26:38.950ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 944d3eec-76e5-4883-96e7-0a2185622649 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
68552025-04-30T06:26:38.952ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 944d3eec-76e5-4883-96e7-0a2185622649 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
68562025-04-30T06:26:38.964ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 14914 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 944d3eec-76e5-4883-96e7-0a2185622649 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
68572025-04-30T06:26:38.966ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "944d3eec-76e5-4883-96e7-0a2185622649", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
68582025-04-30T06:26:38.966ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
68592025-04-30T06:26:38.966ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68602025-04-30T06:26:38.966ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
68612025-04-30T06:26:38.993ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68622025-04-30T06:26:38.994ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68632025-04-30T06:26:39.007ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68642025-04-30T06:26:39.019ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68652025-04-30T06:26:39.019ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68662025-04-30T06:26:39.020ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68672025-04-30T06:26:39.025ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68682025-04-30T06:26:39.026ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68692025-04-30T06:26:39.026ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 473.420649ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68702025-04-30T06:26:39.026ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
68712025-04-30T06:26:39.026ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68722025-04-30T06:26:39.026ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68732025-04-30T06:26:39.026ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68742025-04-30T06:26:39.026ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68752025-04-30T06:26:39.041ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68762025-04-30T06:26:39.044ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68772025-04-30T06:26:39.050ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68782025-04-30T06:26:39.050ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68792025-04-30T06:26:39.050ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68802025-04-30T06:26:39.051ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68812025-04-30T06:26:39.060ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
68822025-04-30T06:26:39.060ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
68832025-04-30T06:26:39.062ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 17 72 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .r..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62510 req_id = e0a720e8-e25f-4d66-b283-04bdca3691e0
68842025-04-30T06:26:39.062ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6002,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62510 req_id = e0a720e8-e25f-4d66-b283-04bdca3691e0
68852025-04-30T06:26:39.062ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
68862025-04-30T06:26:39.062ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68872025-04-30T06:26:39.062ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62510 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e0a720e8-e25f-4d66-b283-04bdca3691e0
68882025-04-30T06:26:39.064ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3263, tv_nsec: 863652307 } })
68892025-04-30T06:26:39.064ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
68902025-04-30T06:26:39.065ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 17 24 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .$..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50771 req_id = 7bab4570-ad40-4057-aaaf-abe84a076e9b
68912025-04-30T06:26:39.065ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5924,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50771 req_id = 7bab4570-ad40-4057-aaaf-abe84a076e9b
68922025-04-30T06:26:39.065ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68932025-04-30T06:26:39.065ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50771 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7bab4570-ad40-4057-aaaf-abe84a076e9b
68942025-04-30T06:26:39.138ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
68952025-04-30T06:26:39.140ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c2 84 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54589 req_id = bb6087aa-f293-417a-ba4a-beade3aeb5d8
68962025-04-30T06:26:39.140ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49796,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54589 req_id = bb6087aa-f293-417a-ba4a-beade3aeb5d8
68972025-04-30T06:26:39.140ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
68982025-04-30T06:26:39.140ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49796, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54589 req_id = bb6087aa-f293-417a-ba4a-beade3aeb5d8
68992025-04-30T06:26:39.140ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
69002025-04-30T06:26:39.140ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
69012025-04-30T06:26:39.140ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
69022025-04-30T06:26:39.141ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
69032025-04-30T06:26:39.143ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6da3e886-748f-4c2a-9cca-d16417cc99bf", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
69042025-04-30T06:26:39.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
69052025-04-30T06:26:39.145ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2ddbe27-6631-41de-9903-aa4f0d19b7b4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
69062025-04-30T06:26:39.146ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
69072025-04-30T06:26:39.147ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c69de23-d0bf-427a-806a-5c2e7909a722", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
69082025-04-30T06:26:39.147ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
69092025-04-30T06:26:39.148ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f436f4c-c836-4244-9b0e-6d613b0b3f23", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
69102025-04-30T06:26:39.149ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
69112025-04-30T06:26:39.150ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d27938-870f-4ba5-8df7-895ebdc38359", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
69122025-04-30T06:26:39.151ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
69132025-04-30T06:26:39.153ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad27053d-e451-43fc-9f42-199c5ae63fa1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
69142025-04-30T06:26:39.154ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
69152025-04-30T06:26:39.156ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df475a36-cfe6-4b87-8733-83167bb05705", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69162025-04-30T06:26:39.156ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
69172025-04-30T06:26:39.165ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5edd736-869a-4ac6-a5e9-4cf6919ac6f7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69182025-04-30T06:26:39.165ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
69192025-04-30T06:26:39.168ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b02f762-a96d-4061-b0dd-44b41a250246", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69202025-04-30T06:26:39.171ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
69212025-04-30T06:26:39.175ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a0bd9d6-75b9-4c13-9557-3c6b45142e89", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69222025-04-30T06:26:39.175ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
69232025-04-30T06:26:39.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69242025-04-30T06:26:39.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69252025-04-30T06:26:39.176ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
69262025-04-30T06:26:39.177ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
69272025-04-30T06:26:39.178ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86358eeb-a0d7-4b87-81f9-e449604142f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69282025-04-30T06:26:39.178ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
69292025-04-30T06:26:39.180ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c654fce8-6903-4a04-8f1b-a3eea68fc3c7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69302025-04-30T06:26:39.180ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
69312025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "354e04c2-a1c7-4c9d-a548-e295ae1d0608", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69322025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
69332025-04-30T06:26:39.195ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
69342025-04-30T06:26:39.195ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
69352025-04-30T06:26:39.195ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
69362025-04-30T06:26:39.195ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69372025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a4a845a-c636-49cb-94b9-1f8c8066da16", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69382025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
69392025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e595bd6-b354-42a4-b05d-6fd89119e7e9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69402025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
69412025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4aed206d-3d55-4220-acc6-a0e777f720cb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69422025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
69432025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65f8db95-7893-46ff-9ac4-70ea8fc55f89", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69442025-04-30T06:26:39.195ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
69452025-04-30T06:26:39.195ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69462025-04-30T06:26:39.195ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69472025-04-30T06:26:39.195ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69482025-04-30T06:26:39.195ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69492025-04-30T06:26:39.195ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69502025-04-30T06:26:39.196ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33db9ea5-3925-45a4-a851-a6fe105f30ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69512025-04-30T06:26:39.197ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
69522025-04-30T06:26:39.197ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69532025-04-30T06:26:39.202ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69542025-04-30T06:26:39.202ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69552025-04-30T06:26:39.203ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69562025-04-30T06:26:39.203ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69572025-04-30T06:26:39.207ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69582025-04-30T06:26:39.207ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69592025-04-30T06:26:39.208ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69602025-04-30T06:26:39.208ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af468cd7-100a-4668-9bf0-65bfe30b2f97", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69612025-04-30T06:26:39.208ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
69622025-04-30T06:26:39.209ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "887e6839-b83e-43c4-bfec-6d431026433e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69632025-04-30T06:26:39.209ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
69642025-04-30T06:26:39.211ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db3ca11b-c822-45e1-aac1-4b757faadd58", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69652025-04-30T06:26:39.211ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
69662025-04-30T06:26:39.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5c86cd-33f2-4c48-b467-137ed5dace7e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69672025-04-30T06:26:39.212ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
69682025-04-30T06:26:39.214ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dbf35af-8aa8-4bb3-ac85-453f733a8c8e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69692025-04-30T06:26:39.214ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
69702025-04-30T06:26:39.218ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c31a5ae-c8ce-4d8b-b482-2df63bed0d6d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69712025-04-30T06:26:39.220ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
69722025-04-30T06:26:39.222ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc4d4d83-0f96-4956-abde-ce3e673b990e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69732025-04-30T06:26:39.222ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
69742025-04-30T06:26:39.226ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f172c3d-3cb0-4057-a9e7-e42c56dfa5d2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69752025-04-30T06:26:39.226ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
69762025-04-30T06:26:39.231ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "819a009b-49f2-4d9f-be0b-1b5945846dd0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69772025-04-30T06:26:39.231ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
69782025-04-30T06:26:39.234ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa97cb0b-35f1-4f0d-9940-f9286e660b47", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69792025-04-30T06:26:39.235ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
69802025-04-30T06:26:39.242ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cdf0b15-9c72-4e51-8504-1c197011fa6d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69812025-04-30T06:26:39.242ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
69822025-04-30T06:26:39.243ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
69832025-04-30T06:26:39.248ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d26a3f9e-3cc8-40b0-9788-48844e38c488", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69842025-04-30T06:26:39.248ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
69852025-04-30T06:26:39.249ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86a04ec6-107a-407c-b660-cec605d9a2d3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69862025-04-30T06:26:39.249ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
69872025-04-30T06:26:39.250ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "542fd420-adda-4742-b612-ccd36014e01d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69882025-04-30T06:26:39.250ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
69892025-04-30T06:26:39.251ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c454f28-fb9e-4487-8300-7635643bda42", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69902025-04-30T06:26:39.251ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
69912025-04-30T06:26:39.252ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9105f7c-0f18-41ea-ab38-00b5cac3fc3f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69922025-04-30T06:26:39.254ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
69932025-04-30T06:26:39.255ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a490c31e-88dc-458d-99cb-a9f05591bb9e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69942025-04-30T06:26:39.257ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
69952025-04-30T06:26:39.259ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a82af70-faac-496d-baa1-67114f83217f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69962025-04-30T06:26:39.259ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
69972025-04-30T06:26:39.262ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62e38e6b-7280-468d-ab96-4193263d730b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
69982025-04-30T06:26:39.262ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
69992025-04-30T06:26:39.264ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "037bc0eb-3ac1-490d-9edd-6d47bafe34f5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70002025-04-30T06:26:39.264ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
70012025-04-30T06:26:39.268ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7bd8106-3e21-458c-9945-a6347f75781b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70022025-04-30T06:26:39.268ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
70032025-04-30T06:26:39.271ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ea9b358-14ac-4b9d-9e55-22b22480722a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70042025-04-30T06:26:39.271ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
70052025-04-30T06:26:39.273ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6a8dfcf-cab4-4e87-a140-3346e7175f92", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70062025-04-30T06:26:39.273ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
70072025-04-30T06:26:39.274ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d00de5c-9407-49dd-b3eb-416c0ee5ed25", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70082025-04-30T06:26:39.274ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
70092025-04-30T06:26:39.275ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1ef2b34-b443-4028-bd84-1f1bec3397a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70102025-04-30T06:26:39.275ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
70112025-04-30T06:26:39.276ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eab32d9-008c-41bc-983a-9c3b53ee1c32", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70122025-04-30T06:26:39.276ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
70132025-04-30T06:26:39.276ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
70142025-04-30T06:26:39.277ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d581ebc-1243-462f-8265-63d372168d2e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70152025-04-30T06:26:39.277ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70162025-04-30T06:26:39.277ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70172025-04-30T06:26:39.277ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70182025-04-30T06:26:39.277ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70192025-04-30T06:26:39.278ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70202025-04-30T06:26:39.289ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70212025-04-30T06:26:39.306ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70222025-04-30T06:26:39.306ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70232025-04-30T06:26:39.306ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70242025-04-30T06:26:39.306ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70252025-04-30T06:26:39.306ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70262025-04-30T06:26:39.307ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70272025-04-30T06:26:39.307ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70282025-04-30T06:26:39.307ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70292025-04-30T06:26:39.307ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70302025-04-30T06:26:39.307ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70312025-04-30T06:26:39.308ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70322025-04-30T06:26:39.314ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70332025-04-30T06:26:39.319ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70342025-04-30T06:26:39.319ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70352025-04-30T06:26:39.319ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70362025-04-30T06:26:39.320ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70372025-04-30T06:26:39.335ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70382025-04-30T06:26:39.340ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70392025-04-30T06:26:39.340ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70402025-04-30T06:26:39.340ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70412025-04-30T06:26:39.340ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70422025-04-30T06:26:39.387ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
70432025-04-30T06:26:39.387ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
70442025-04-30T06:26:39.389ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:61414 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70452025-04-30T06:26:39.389ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:61414 req_id = 8101796d-e850-4d65-8c27-363ef3919bab sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70462025-04-30T06:26:39.389ZINFOtest_disk_metrics (dropshot): request completed latency_us = 352 local_addr = [::1]:50989 method = GET remote_addr = [::1]:61414 req_id = 8101796d-e850-4d65-8c27-363ef3919bab response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70472025-04-30T06:26:39.389ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8101796d-e850-4d65-8c27-363ef3919bab", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
70482025-04-30T06:26:39.391ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70492025-04-30T06:26:39.391ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70502025-04-30T06:26:39.391ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
70512025-04-30T06:26:39.392ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70522025-04-30T06:26:39.392ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70532025-04-30T06:26:39.392ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70542025-04-30T06:26:39.392ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70552025-04-30T06:26:39.393ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70562025-04-30T06:26:39.393ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70572025-04-30T06:26:39.393ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70582025-04-30T06:26:39.394ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70592025-04-30T06:26:39.395ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70602025-04-30T06:26:39.399ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70612025-04-30T06:26:39.400ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70622025-04-30T06:26:39.400ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70632025-04-30T06:26:39.401ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70642025-04-30T06:26:39.450ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
70652025-04-30T06:26:39.450ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
70662025-04-30T06:26:39.451ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
70672025-04-30T06:26:39.451ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 425.200855ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
70682025-04-30T06:26:39.451ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
70692025-04-30T06:26:39.451ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70702025-04-30T06:26:39.451ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70712025-04-30T06:26:39.451ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70722025-04-30T06:26:39.451ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70732025-04-30T06:26:39.454ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70742025-04-30T06:26:39.456ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70752025-04-30T06:26:39.461ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70762025-04-30T06:26:39.461ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70772025-04-30T06:26:39.461ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70782025-04-30T06:26:39.461ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70792025-04-30T06:26:39.465ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
70802025-04-30T06:26:39.465ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
70812025-04-30T06:26:39.466ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e2 8e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55683 req_id = 692f362f-f3b6-4de5-b2a1-6d7d6581f2ff
70822025-04-30T06:26:39.467ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57998,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55683 req_id = 692f362f-f3b6-4de5-b2a1-6d7d6581f2ff
70832025-04-30T06:26:39.467ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
70842025-04-30T06:26:39.467ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70852025-04-30T06:26:39.467ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55683 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 692f362f-f3b6-4de5-b2a1-6d7d6581f2ff
70862025-04-30T06:26:39.467ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3264, tv_nsec: 266783079 } })
70872025-04-30T06:26:39.467ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
70882025-04-30T06:26:39.468ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 62 f5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e b...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55987 req_id = cf71d708-22a7-4d0f-a6d2-77d7608d7c99
70892025-04-30T06:26:39.468ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25333,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55987 req_id = cf71d708-22a7-4d0f-a6d2-77d7608d7c99
70902025-04-30T06:26:39.469ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70912025-04-30T06:26:39.469ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55987 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cf71d708-22a7-4d0f-a6d2-77d7608d7c99
70922025-04-30T06:26:39.527ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
70932025-04-30T06:26:39.528ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70942025-04-30T06:26:39.528ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70952025-04-30T06:26:39.528ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70962025-04-30T06:26:39.528ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70972025-04-30T06:26:39.529ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70982025-04-30T06:26:39.529ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: dc 69 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .i..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64455 req_id = 25f7048d-8008-47f6-a3c6-4bdb14b66c80
70992025-04-30T06:26:39.529ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56425,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64455 req_id = 25f7048d-8008-47f6-a3c6-4bdb14b66c80
71002025-04-30T06:26:39.529ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
71012025-04-30T06:26:39.529ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56425, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64455 req_id = 25f7048d-8008-47f6-a3c6-4bdb14b66c80
71022025-04-30T06:26:39.530ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
71032025-04-30T06:26:39.530ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
71042025-04-30T06:26:39.530ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
71052025-04-30T06:26:39.532ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71062025-04-30T06:26:39.537ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71072025-04-30T06:26:39.538ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71082025-04-30T06:26:39.538ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71092025-04-30T06:26:39.538ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71102025-04-30T06:26:39.539ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a05bb427-92a9-440d-b29f-fd8a3e32775d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71112025-04-30T06:26:39.539ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
71122025-04-30T06:26:39.543ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71132025-04-30T06:26:39.543ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71142025-04-30T06:26:39.543ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71152025-04-30T06:26:39.544ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "687a8d44-15b7-4078-8817-4dc455410479", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71162025-04-30T06:26:39.544ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
71172025-04-30T06:26:39.561ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b96147d1-42c0-4cb7-a4ee-dcf8e0c2d5dc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71182025-04-30T06:26:39.561ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
71192025-04-30T06:26:39.565ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc8e9d3-04ef-4243-829d-e33ba7fe2b5b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71202025-04-30T06:26:39.565ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
71212025-04-30T06:26:39.570ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5048fcc7-f80c-4af7-83e6-b8ce9b52fcce", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71222025-04-30T06:26:39.570ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
71232025-04-30T06:26:39.575ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49486f68-8afe-4d67-b204-2f063b8f659f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71242025-04-30T06:26:39.575ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
71252025-04-30T06:26:39.583ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
71262025-04-30T06:26:39.584ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27eb1c53-e2af-4d5c-8a6b-180acc43c9b2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71272025-04-30T06:26:39.585ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
71282025-04-30T06:26:39.588ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddf6c964-4b81-4ea9-88ad-01812b724077", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71292025-04-30T06:26:39.588ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
71302025-04-30T06:26:39.589ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14adfb88-93fd-4c99-9e81-b2cc49cb30e5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71312025-04-30T06:26:39.590ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
71322025-04-30T06:26:39.591ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fa4bf7a-e597-4045-8c1c-585232f7f2cc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71332025-04-30T06:26:39.591ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
71342025-04-30T06:26:39.593ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d76255ab-3583-481a-88c0-8f83c2cacd23", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71352025-04-30T06:26:39.593ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
71362025-04-30T06:26:39.595ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4657280-a52a-4bdb-aa41-008bde2f2616", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71372025-04-30T06:26:39.595ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
71382025-04-30T06:26:39.597ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91903ebc-9755-48b3-b975-ef83554059eb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71392025-04-30T06:26:39.597ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
71402025-04-30T06:26:39.601ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c0285c5-3dc1-45cd-8245-244e24df971d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71412025-04-30T06:26:39.601ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
71422025-04-30T06:26:39.605ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03ccf7d8-5df1-4097-89a7-4524877a52f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71432025-04-30T06:26:39.605ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
71442025-04-30T06:26:39.609ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0cca8d3-1a58-4014-8f01-18f1f04b2652", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71452025-04-30T06:26:39.609ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
71462025-04-30T06:26:39.612ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61ec3dce-c2bf-438c-8912-bb5b006d621f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71472025-04-30T06:26:39.612ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
71482025-04-30T06:26:39.615ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38b90890-150c-463e-968b-a6d08344c970", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71492025-04-30T06:26:39.616ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
71502025-04-30T06:26:39.619ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd86c79e-df04-4cf9-be4e-47989de6e8e2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71512025-04-30T06:26:39.620ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
71522025-04-30T06:26:39.622ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd44d6d7-76e5-451f-bbfe-266ad5becb76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71532025-04-30T06:26:39.622ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
71542025-04-30T06:26:39.623ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7460a8a-dc75-4e47-80bf-f7435b3d5bc1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71552025-04-30T06:26:39.623ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
71562025-04-30T06:26:39.625ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fb79a10-af95-444c-84f7-23004ef97dcb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71572025-04-30T06:26:39.625ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
71582025-04-30T06:26:39.626ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b085aed-f651-4362-999b-458e10504cbe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71592025-04-30T06:26:39.627ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
71602025-04-30T06:26:39.629ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3ae788f-660b-4e87-b9cd-3e48765b3f11", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71612025-04-30T06:26:39.629ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
71622025-04-30T06:26:39.632ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df051f85-bab3-4887-b6fb-70bd1b9af72f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71632025-04-30T06:26:39.632ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
71642025-04-30T06:26:39.634ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71652025-04-30T06:26:39.634ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71662025-04-30T06:26:39.634ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71672025-04-30T06:26:39.635ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71682025-04-30T06:26:39.636ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71692025-04-30T06:26:39.639ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71702025-04-30T06:26:39.647ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71712025-04-30T06:26:39.647ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71722025-04-30T06:26:39.647ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71732025-04-30T06:26:39.650ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71742025-04-30T06:26:39.652ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71752025-04-30T06:26:39.652ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71762025-04-30T06:26:39.662ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71772025-04-30T06:26:39.665ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d3c8c00-f3b1-4e64-be48-3ba22ecd23f1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71782025-04-30T06:26:39.665ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
71792025-04-30T06:26:39.670ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "714b318e-f316-4f10-8c6a-a314ccd1ef02", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71802025-04-30T06:26:39.670ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
71812025-04-30T06:26:39.675ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78318dcd-876c-4825-aa23-24769952b8ee", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71822025-04-30T06:26:39.675ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
71832025-04-30T06:26:39.679ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9230c35-a8ca-4833-af2b-351dfc9a2bf1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71842025-04-30T06:26:39.679ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
71852025-04-30T06:26:39.683ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "211175ab-94b9-446c-9011-54afd6459854", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71862025-04-30T06:26:39.683ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
71872025-04-30T06:26:39.685ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6a2c472-4269-4af3-9caf-86bd0fe42792", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71882025-04-30T06:26:39.685ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
71892025-04-30T06:26:39.687ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4f56ec7-2c4d-468f-8794-9ec073a6b3d7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71902025-04-30T06:26:39.687ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
71912025-04-30T06:26:39.689ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa0d9dde-2303-4d89-88b9-27c489a4d15d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71922025-04-30T06:26:39.689ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
71932025-04-30T06:26:39.691ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff3d02e-99f7-4320-9170-6ba9a08073fc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71942025-04-30T06:26:39.691ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
71952025-04-30T06:26:39.694ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
71962025-04-30T06:26:39.705ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43c74f4f-c09e-4976-95bd-349604bcdcbd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71972025-04-30T06:26:39.705ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
71982025-04-30T06:26:39.709ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf5074e0-8760-445b-9b41-b927fd40c7ec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
71992025-04-30T06:26:39.709ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
72002025-04-30T06:26:39.713ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "920c4725-2aed-48a2-bd4f-dd0e62acc03e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72012025-04-30T06:26:39.714ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
72022025-04-30T06:26:39.720ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b7f92c4-0cac-47a2-9a26-470211475bf1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72032025-04-30T06:26:39.720ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
72042025-04-30T06:26:39.724ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8912d4ec-4670-483d-a1a3-f0de098441c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72052025-04-30T06:26:39.724ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
72062025-04-30T06:26:39.728ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "633681b3-e4e4-4d49-b63c-93b04a51001a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72072025-04-30T06:26:39.729ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
72082025-04-30T06:26:39.731ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bda2378-e76f-47da-9a9d-38fedd671e27", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72092025-04-30T06:26:39.731ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
72102025-04-30T06:26:39.732ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df7595f7-03e7-49d7-8332-d9a5e51a9006", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72112025-04-30T06:26:39.733ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
72122025-04-30T06:26:39.734ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f635145-baad-44c6-aacf-75ca4200f222", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72132025-04-30T06:26:39.734ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
72142025-04-30T06:26:39.736ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "016940e5-fb74-46db-a959-c322c7aa588d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72152025-04-30T06:26:39.736ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
72162025-04-30T06:26:39.737ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "637db8e5-a47c-422e-9d8e-e2d3d921861b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72172025-04-30T06:26:39.737ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72182025-04-30T06:26:39.737ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72192025-04-30T06:26:39.737ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72202025-04-30T06:26:39.737ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72212025-04-30T06:26:39.745ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72222025-04-30T06:26:39.749ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72232025-04-30T06:26:39.749ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72242025-04-30T06:26:39.749ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72252025-04-30T06:26:39.749ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72262025-04-30T06:26:39.749ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72272025-04-30T06:26:39.751ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72282025-04-30T06:26:39.760ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72292025-04-30T06:26:39.760ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72302025-04-30T06:26:39.760ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72312025-04-30T06:26:39.760ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72322025-04-30T06:26:39.760ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72332025-04-30T06:26:39.762ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72342025-04-30T06:26:39.764ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72352025-04-30T06:26:39.772ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72362025-04-30T06:26:39.772ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72372025-04-30T06:26:39.772ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72382025-04-30T06:26:39.773ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72392025-04-30T06:26:39.776ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72402025-04-30T06:26:39.782ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72412025-04-30T06:26:39.782ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72422025-04-30T06:26:39.782ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72432025-04-30T06:26:39.783ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72442025-04-30T06:26:39.784ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72452025-04-30T06:26:39.784ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72462025-04-30T06:26:39.785ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72472025-04-30T06:26:39.851ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
72482025-04-30T06:26:39.851ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
72492025-04-30T06:26:39.852ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:53716 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
72502025-04-30T06:26:39.853ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:53716 req_id = d6ce766a-a05a-4785-884f-24fc65f4424d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72512025-04-30T06:26:39.853ZINFOtest_disk_metrics (dropshot): request completed latency_us = 477 local_addr = [::1]:50989 method = GET remote_addr = [::1]:53716 req_id = d6ce766a-a05a-4785-884f-24fc65f4424d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72522025-04-30T06:26:39.854ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ce766a-a05a-4785-884f-24fc65f4424d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
72532025-04-30T06:26:39.856ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72542025-04-30T06:26:39.856ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72552025-04-30T06:26:39.856ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
72562025-04-30T06:26:39.858ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72572025-04-30T06:26:39.858ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72582025-04-30T06:26:39.859ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72592025-04-30T06:26:39.859ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72602025-04-30T06:26:39.859ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72612025-04-30T06:26:39.862ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72622025-04-30T06:26:39.866ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72632025-04-30T06:26:39.866ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72642025-04-30T06:26:39.866ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72652025-04-30T06:26:39.866ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72662025-04-30T06:26:39.898ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
72672025-04-30T06:26:39.900ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
72682025-04-30T06:26:39.900ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
72692025-04-30T06:26:39.900ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 449.385501ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
72702025-04-30T06:26:39.901ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
72712025-04-30T06:26:39.901ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72722025-04-30T06:26:39.901ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72732025-04-30T06:26:39.901ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72742025-04-30T06:26:39.901ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72752025-04-30T06:26:39.904ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72762025-04-30T06:26:39.906ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72772025-04-30T06:26:39.913ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72782025-04-30T06:26:39.913ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72792025-04-30T06:26:39.913ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72802025-04-30T06:26:39.913ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72812025-04-30T06:26:39.917ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
72822025-04-30T06:26:39.917ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
72832025-04-30T06:26:39.919ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5e c4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ^............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53119 req_id = 000af46f-167f-400b-913a-f5acae6e02d8
72842025-04-30T06:26:39.919ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24260,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53119 req_id = 000af46f-167f-400b-913a-f5acae6e02d8
72852025-04-30T06:26:39.919ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
72862025-04-30T06:26:39.920ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72872025-04-30T06:26:39.921ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53119 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 000af46f-167f-400b-913a-f5acae6e02d8
72882025-04-30T06:26:39.923ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3264, tv_nsec: 722528621 } })
72892025-04-30T06:26:39.923ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
72902025-04-30T06:26:39.924ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 71 b4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e q...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64906 req_id = ffdc73fb-d067-4066-8a95-f76c654326f5
72912025-04-30T06:26:39.924ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29108,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64906 req_id = ffdc73fb-d067-4066-8a95-f76c654326f5
72922025-04-30T06:26:39.925ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72932025-04-30T06:26:39.925ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64906 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ffdc73fb-d067-4066-8a95-f76c654326f5
72942025-04-30T06:26:39.984ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
72952025-04-30T06:26:39.985ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72962025-04-30T06:26:39.985ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72972025-04-30T06:26:39.985ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72982025-04-30T06:26:39.986ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72992025-04-30T06:26:39.988ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73002025-04-30T06:26:39.989ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73012025-04-30T06:26:39.989ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73022025-04-30T06:26:39.991ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d1 bb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50730 req_id = 82d35236-5ada-4d47-9f32-bfc400e25fa3
73032025-04-30T06:26:39.991ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53691,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50730 req_id = 82d35236-5ada-4d47-9f32-bfc400e25fa3
73042025-04-30T06:26:39.991ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
73052025-04-30T06:26:39.992ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53691, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50730 req_id = 82d35236-5ada-4d47-9f32-bfc400e25fa3
73062025-04-30T06:26:39.994ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
73072025-04-30T06:26:39.994ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
73082025-04-30T06:26:39.994ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
73092025-04-30T06:26:39.995ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73102025-04-30T06:26:39.995ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73112025-04-30T06:26:39.995ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73122025-04-30T06:26:39.996ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73132025-04-30T06:26:39.996ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73142025-04-30T06:26:40.003ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73152025-04-30T06:26:40.003ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73162025-04-30T06:26:40.003ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73172025-04-30T06:26:40.004ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73182025-04-30T06:26:40.020ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73192025-04-30T06:26:40.020ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73202025-04-30T06:26:40.021ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73212025-04-30T06:26:40.025ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c140e36e-f3d9-4bdf-8f2d-6f9b15c6674b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73222025-04-30T06:26:40.026ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
73232025-04-30T06:26:40.042ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28a97745-ed32-44dc-804b-ad0ea4d66ec7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73242025-04-30T06:26:40.042ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
73252025-04-30T06:26:40.074ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a0b2d1b-4295-4d28-8b65-27a09d64bb8a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73262025-04-30T06:26:40.076ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
73272025-04-30T06:26:40.088ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "970a3d76-bc54-4980-8fca-2df47d1fadb9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73282025-04-30T06:26:40.088ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
73292025-04-30T06:26:40.093ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
73302025-04-30T06:26:40.102ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "708fab9e-5fc5-410a-89af-07b1e0ffb19a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73312025-04-30T06:26:40.102ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
73322025-04-30T06:26:40.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51607af3-d16e-4970-a739-f3f20d6ff57d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73332025-04-30T06:26:40.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
73342025-04-30T06:26:40.111ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbe30584-8e6f-472f-afab-6f6c51dcd755", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73352025-04-30T06:26:40.111ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
73362025-04-30T06:26:40.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8fb4a29-3d93-415b-8947-7ffc3c102a40", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73372025-04-30T06:26:40.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
73382025-04-30T06:26:40.116ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "450bf05e-b316-4ac4-a601-5dd7de09ab1e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73392025-04-30T06:26:40.116ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
73402025-04-30T06:26:40.117ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "544a1371-749f-433c-910f-bd8b5d161ed4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73412025-04-30T06:26:40.118ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
73422025-04-30T06:26:40.119ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12c3363c-ccd7-4dd4-936c-7297f3eb18db", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73432025-04-30T06:26:40.119ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
73442025-04-30T06:26:40.120ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c103633f-3ca2-4b39-9ef3-590e1aa5a130", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73452025-04-30T06:26:40.121ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
73462025-04-30T06:26:40.123ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6362756-7fc1-4ac1-b7ec-3aeb0c91e933", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73472025-04-30T06:26:40.123ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
73482025-04-30T06:26:40.125ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed81509b-dfb7-4c26-afec-4962f2d1035d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73492025-04-30T06:26:40.126ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
73502025-04-30T06:26:40.128ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c408350-8d82-417b-a09b-184350a9b7a7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73512025-04-30T06:26:40.128ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
73522025-04-30T06:26:40.130ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbf17b32-975e-475b-b74a-f1f3a49a71ba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73532025-04-30T06:26:40.130ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
73542025-04-30T06:26:40.133ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d5cf07-d8c0-416b-b62f-cdc5c6fbf60a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73552025-04-30T06:26:40.133ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
73562025-04-30T06:26:40.135ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aee2842-e977-462b-9a83-abefdcf76a0a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73572025-04-30T06:26:40.135ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
73582025-04-30T06:26:40.139ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48aa405e-bc87-4f99-8bb4-bb2de287cf3a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73592025-04-30T06:26:40.139ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
73602025-04-30T06:26:40.140ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4401740-091a-46ef-8de3-1dfff736f3df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73612025-04-30T06:26:40.140ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
73622025-04-30T06:26:40.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfdccaec-e3db-4671-999c-6a3eafcc525e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73632025-04-30T06:26:40.141ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
73642025-04-30T06:26:40.142ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2807e995-e9f8-4493-8e56-5196baf15c40", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73652025-04-30T06:26:40.142ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
73662025-04-30T06:26:40.143ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a8cd869-9a81-4415-8486-9b1dfb6444b1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73672025-04-30T06:26:40.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
73682025-04-30T06:26:40.144ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73692025-04-30T06:26:40.144ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73702025-04-30T06:26:40.144ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73712025-04-30T06:26:40.144ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73722025-04-30T06:26:40.145ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73732025-04-30T06:26:40.147ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbe3f8b6-4761-4800-9a94-1ebf0d4a66fd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
73742025-04-30T06:26:40.147ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
73752025-04-30T06:26:40.147ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73762025-04-30T06:26:40.154ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73772025-04-30T06:26:40.154ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73782025-04-30T06:26:40.154ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73792025-04-30T06:26:40.154ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73802025-04-30T06:26:40.157ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73812025-04-30T06:26:40.157ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73822025-04-30T06:26:40.158ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73832025-04-30T06:26:40.159ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd7e2d8f-9c8d-4d07-8edd-6e7f4c017e2d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
73842025-04-30T06:26:40.159ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
73852025-04-30T06:26:40.161ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ada3b8f-450e-4b4f-b3ea-0969e7b8d03b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
73862025-04-30T06:26:40.161ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
73872025-04-30T06:26:40.165ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82a680b8-5282-4a3a-9f0f-15f1c0268d39", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
73882025-04-30T06:26:40.165ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
73892025-04-30T06:26:40.168ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba341300-7d42-4a5c-8336-ef4d032169bd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
73902025-04-30T06:26:40.169ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
73912025-04-30T06:26:40.171ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7450d1d-f793-40e8-b692-49c7308171f0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
73922025-04-30T06:26:40.171ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
73932025-04-30T06:26:40.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73942025-04-30T06:26:40.176ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73952025-04-30T06:26:40.176ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
73962025-04-30T06:26:40.177ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
73972025-04-30T06:26:40.178ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99f5d226-496f-4e38-9064-e2e83d42db48", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
73982025-04-30T06:26:40.178ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
73992025-04-30T06:26:40.182ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fc22927-78a2-4f9a-bcd3-5d1bba6f5ec0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74002025-04-30T06:26:40.182ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
74012025-04-30T06:26:40.182ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
74022025-04-30T06:26:40.182ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
74032025-04-30T06:26:40.182ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3405312e-5927-45c4-b431-cd248e8428c4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74042025-04-30T06:26:40.182ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
74052025-04-30T06:26:40.184ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4c425a6-a490-4f20-9f7c-815b1ef6e552", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74062025-04-30T06:26:40.184ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
74072025-04-30T06:26:40.186ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22c4f6b3-01e6-444a-9908-f7538417ee8e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74082025-04-30T06:26:40.186ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
74092025-04-30T06:26:40.188ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d7439f5-ef76-4f4e-a1ce-9c3e96e0897f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74102025-04-30T06:26:40.188ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
74112025-04-30T06:26:40.189ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
74122025-04-30T06:26:40.189ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74132025-04-30T06:26:40.189ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d7ef5f4-7411-4202-bcc9-f57bebb59870", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74142025-04-30T06:26:40.189ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
74152025-04-30T06:26:40.194ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
74162025-04-30T06:26:40.197ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ab67a01-50bb-4acd-bfcb-2654bef93224", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74172025-04-30T06:26:40.197ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
74182025-04-30T06:26:40.200ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a62fb9d4-eb6a-4325-8197-02626d154ed0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74192025-04-30T06:26:40.200ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
74202025-04-30T06:26:40.203ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c00955ce-d12f-4b2c-b2b7-678a38b2f64e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74212025-04-30T06:26:40.204ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
74222025-04-30T06:26:40.207ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6f17e47-5766-452c-96df-10a4add12aa3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74232025-04-30T06:26:40.207ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
74242025-04-30T06:26:40.209ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb480c18-c8fd-42a3-a030-f540e71a01ff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74252025-04-30T06:26:40.209ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
74262025-04-30T06:26:40.209ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8cc80aa-221d-4da6-8321-3cf43fd1e399", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74272025-04-30T06:26:40.210ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
74282025-04-30T06:26:40.210ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8608fee8-ebd5-43f0-985d-f38db066cc8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74292025-04-30T06:26:40.211ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
74302025-04-30T06:26:40.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63754e88-613a-450d-9235-a30c5b72ef4c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74312025-04-30T06:26:40.212ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
74322025-04-30T06:26:40.213ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dda62ec-adc1-4faf-97c5-18da00c09d03", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74332025-04-30T06:26:40.214ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74342025-04-30T06:26:40.214ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74352025-04-30T06:26:40.214ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74362025-04-30T06:26:40.214ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74372025-04-30T06:26:40.215ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74382025-04-30T06:26:40.217ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74392025-04-30T06:26:40.221ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74402025-04-30T06:26:40.221ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74412025-04-30T06:26:40.221ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74422025-04-30T06:26:40.221ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74432025-04-30T06:26:40.221ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74442025-04-30T06:26:40.222ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74452025-04-30T06:26:40.225ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74462025-04-30T06:26:40.229ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74472025-04-30T06:26:40.230ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74482025-04-30T06:26:40.230ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74492025-04-30T06:26:40.230ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74502025-04-30T06:26:40.281ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
74512025-04-30T06:26:40.281ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
74522025-04-30T06:26:40.283ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74532025-04-30T06:26:40.283ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74542025-04-30T06:26:40.283ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74552025-04-30T06:26:40.283ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
74562025-04-30T06:26:40.284ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74572025-04-30T06:26:40.284ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:32805 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
74582025-04-30T06:26:40.285ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
74592025-04-30T06:26:40.286ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:32805 req_id = 76ed283a-14b5-46fb-89cb-db13a7bb32b8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74602025-04-30T06:26:40.286ZINFOtest_disk_metrics (dropshot): request completed latency_us = 627 local_addr = [::1]:50989 method = GET remote_addr = [::1]:32805 req_id = 76ed283a-14b5-46fb-89cb-db13a7bb32b8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74612025-04-30T06:26:40.288ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "76ed283a-14b5-46fb-89cb-db13a7bb32b8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
74622025-04-30T06:26:40.293ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74632025-04-30T06:26:40.293ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74642025-04-30T06:26:40.293ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
74652025-04-30T06:26:40.297ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74662025-04-30T06:26:40.297ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74672025-04-30T06:26:40.297ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74682025-04-30T06:26:40.297ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74692025-04-30T06:26:40.298ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74702025-04-30T06:26:40.299ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74712025-04-30T06:26:40.306ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74722025-04-30T06:26:40.306ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74732025-04-30T06:26:40.306ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
74742025-04-30T06:26:40.307ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74752025-04-30T06:26:40.309ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74762025-04-30T06:26:40.315ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74772025-04-30T06:26:40.315ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74782025-04-30T06:26:40.315ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74792025-04-30T06:26:40.316ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74802025-04-30T06:26:40.317ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74812025-04-30T06:26:40.317ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
74822025-04-30T06:26:40.318ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74832025-04-30T06:26:40.347ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
74842025-04-30T06:26:40.347ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
74852025-04-30T06:26:40.347ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 446.407668ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
74862025-04-30T06:26:40.347ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
74872025-04-30T06:26:40.347ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74882025-04-30T06:26:40.347ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74892025-04-30T06:26:40.347ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74902025-04-30T06:26:40.347ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74912025-04-30T06:26:40.348ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74922025-04-30T06:26:40.350ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74932025-04-30T06:26:40.354ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74942025-04-30T06:26:40.354ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74952025-04-30T06:26:40.354ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74962025-04-30T06:26:40.354ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74972025-04-30T06:26:40.360ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
74982025-04-30T06:26:40.362ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
74992025-04-30T06:26:40.362ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
75002025-04-30T06:26:40.363ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fd ef 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55408 req_id = 7c925982-5f05-4082-a881-eea0385a1a7d
75012025-04-30T06:26:40.364ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65007,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55408 req_id = 7c925982-5f05-4082-a881-eea0385a1a7d
75022025-04-30T06:26:40.365ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
75032025-04-30T06:26:40.365ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75042025-04-30T06:26:40.365ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55408 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7c925982-5f05-4082-a881-eea0385a1a7d
75052025-04-30T06:26:40.366ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3265, tv_nsec: 164892552 } })
75062025-04-30T06:26:40.366ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
75072025-04-30T06:26:40.367ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 01 72 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .r..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63106 req_id = 9fe0d23f-e96d-41b8-86ff-90caac82dec6
75082025-04-30T06:26:40.367ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 370,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63106 req_id = 9fe0d23f-e96d-41b8-86ff-90caac82dec6
75092025-04-30T06:26:40.367ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75102025-04-30T06:26:40.367ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63106 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9fe0d23f-e96d-41b8-86ff-90caac82dec6
75112025-04-30T06:26:40.458ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
75122025-04-30T06:26:40.459ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75132025-04-30T06:26:40.459ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75142025-04-30T06:26:40.459ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75152025-04-30T06:26:40.459ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75162025-04-30T06:26:40.460ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75172025-04-30T06:26:40.461ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1b b8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57935 req_id = 7139a4e4-8f37-4a69-9839-619dd9c58085
75182025-04-30T06:26:40.461ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7096,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57935 req_id = 7139a4e4-8f37-4a69-9839-619dd9c58085
75192025-04-30T06:26:40.461ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
75202025-04-30T06:26:40.461ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7096, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57935 req_id = 7139a4e4-8f37-4a69-9839-619dd9c58085
75212025-04-30T06:26:40.462ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
75222025-04-30T06:26:40.462ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
75232025-04-30T06:26:40.462ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
75242025-04-30T06:26:40.465ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75252025-04-30T06:26:40.477ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75262025-04-30T06:26:40.477ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75272025-04-30T06:26:40.477ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75282025-04-30T06:26:40.477ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75292025-04-30T06:26:40.483ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea8e43db-a122-48be-aac5-6bb7085d520c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75302025-04-30T06:26:40.485ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
75312025-04-30T06:26:40.485ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75322025-04-30T06:26:40.485ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75332025-04-30T06:26:40.486ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75342025-04-30T06:26:40.490ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17325214-70dd-491f-aa0a-97fd574523dd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75352025-04-30T06:26:40.490ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
75362025-04-30T06:26:40.493ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4039b2c6-c32e-450b-a508-dcb682b15128", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75372025-04-30T06:26:40.493ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
75382025-04-30T06:26:40.495ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cbc4cc0-46e1-4cb8-8cec-cd8ac673efc1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75392025-04-30T06:26:40.496ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
75402025-04-30T06:26:40.498ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bebd21ea-9303-46c1-8ea3-f64deb26e7d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75412025-04-30T06:26:40.500ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
75422025-04-30T06:26:40.502ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c6ba96b-9b8c-4b4a-8b15-56d6292d7056", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75432025-04-30T06:26:40.503ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
75442025-04-30T06:26:40.506ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ff5bf31-7d60-415b-89db-b5d822aabe72", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75452025-04-30T06:26:40.506ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
75462025-04-30T06:26:40.508ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23ea458c-b59a-4e07-8911-7303ed1cf35d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75472025-04-30T06:26:40.509ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
75482025-04-30T06:26:40.510ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c2acfcc-a609-4a3d-91f9-8ac2c373fe82", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75492025-04-30T06:26:40.510ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
75502025-04-30T06:26:40.528ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9b913d2-fd39-4bb8-b1f5-d6e42f161408", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75512025-04-30T06:26:40.528ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
75522025-04-30T06:26:40.531ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3db4d8b7-8b30-44ed-bfe4-aff6100a06b4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75532025-04-30T06:26:40.531ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
75542025-04-30T06:26:40.534ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "caf6868d-af3c-4ff0-984f-03e7a9925d04", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75552025-04-30T06:26:40.534ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
75562025-04-30T06:26:40.537ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
75572025-04-30T06:26:40.542ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0505aa1f-0340-45c7-b9df-078e080b14db", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75582025-04-30T06:26:40.543ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
75592025-04-30T06:26:40.550ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b6ae86-09ad-48db-ad4a-73f2c978664f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75602025-04-30T06:26:40.551ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
75612025-04-30T06:26:40.561ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0b6f3eb-9238-4002-8ce9-ed5a9397fd04", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75622025-04-30T06:26:40.561ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
75632025-04-30T06:26:40.583ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca315e21-e192-4239-9d3c-dd2742fb8298", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75642025-04-30T06:26:40.583ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
75652025-04-30T06:26:40.587ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3073f9b-b2a1-4941-bcb2-9697a2a93ab2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75662025-04-30T06:26:40.588ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
75672025-04-30T06:26:40.589ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75682025-04-30T06:26:40.589ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75692025-04-30T06:26:40.589ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75702025-04-30T06:26:40.589ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75712025-04-30T06:26:40.592ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75722025-04-30T06:26:40.596ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75732025-04-30T06:26:40.606ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75742025-04-30T06:26:40.606ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75752025-04-30T06:26:40.606ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75762025-04-30T06:26:40.607ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75772025-04-30T06:26:40.610ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a62fdd84-6a92-45c7-bd81-30ff34658ce0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75782025-04-30T06:26:40.610ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
75792025-04-30T06:26:40.611ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75802025-04-30T06:26:40.611ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75812025-04-30T06:26:40.612ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75822025-04-30T06:26:40.622ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f4b8303-2c2c-4be8-a91d-d87ef78f62d2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75832025-04-30T06:26:40.623ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
75842025-04-30T06:26:40.628ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f9520fd-f85c-44d2-b12c-8f7e5ff32f1a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75852025-04-30T06:26:40.628ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
75862025-04-30T06:26:40.634ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c1f4dbd-a731-406e-a75d-472fdf03b863", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75872025-04-30T06:26:40.635ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
75882025-04-30T06:26:40.644ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25dc72c2-bd70-40ec-8858-c3344a0eb0fa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75892025-04-30T06:26:40.653ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
75902025-04-30T06:26:40.659ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03368719-a539-47ec-b570-8d31ad5a7443", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75912025-04-30T06:26:40.659ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
75922025-04-30T06:26:40.663ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d50f83-eef5-45fb-a6c3-4a87270b1815", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75932025-04-30T06:26:40.664ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
75942025-04-30T06:26:40.666ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
75952025-04-30T06:26:40.671ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a12dfd2-f610-4a01-9283-154f9e7bb53c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75962025-04-30T06:26:40.671ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
75972025-04-30T06:26:40.676ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c285baba-579d-438a-8705-35416e6ea51c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
75982025-04-30T06:26:40.676ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
75992025-04-30T06:26:40.679ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eb18b7b-59fb-43d3-8b8c-0678840c509f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76002025-04-30T06:26:40.679ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
76012025-04-30T06:26:40.682ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48611200-d650-422a-afda-b2b29595d4d5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76022025-04-30T06:26:40.682ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
76032025-04-30T06:26:40.686ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2011b2ba-d15d-422a-b98d-c46e87417032", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76042025-04-30T06:26:40.686ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
76052025-04-30T06:26:40.689ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf62c6aa-0263-4619-9240-22a0c6b5a8c8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76062025-04-30T06:26:40.689ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
76072025-04-30T06:26:40.691ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab530de7-39e9-45f7-8e49-c352f1ad535c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76082025-04-30T06:26:40.691ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
76092025-04-30T06:26:40.693ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ab018b4-2f76-4e2c-80fa-443b0e4cd85e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76102025-04-30T06:26:40.693ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
76112025-04-30T06:26:40.694ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b614b5c-d546-4df1-b872-f93e2fa46676", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76122025-04-30T06:26:40.694ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
76132025-04-30T06:26:40.705ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3460661-a860-4f5c-ad96-4033afbe9334", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76142025-04-30T06:26:40.706ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
76152025-04-30T06:26:40.715ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ab2e1ce-756b-403f-aa69-37042b5776a3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76162025-04-30T06:26:40.715ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
76172025-04-30T06:26:40.717ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76182025-04-30T06:26:40.725ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76192025-04-30T06:26:40.726ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76202025-04-30T06:26:40.726ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76212025-04-30T06:26:40.727ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76222025-04-30T06:26:40.731ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76232025-04-30T06:26:40.737ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76242025-04-30T06:26:40.739ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76252025-04-30T06:26:40.739ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76262025-04-30T06:26:40.739ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76272025-04-30T06:26:40.742ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76282025-04-30T06:26:40.742ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76292025-04-30T06:26:40.742ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76302025-04-30T06:26:40.743ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49c7fb23-6ec2-4bf2-8581-4edc8a7855c3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76312025-04-30T06:26:40.743ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
76322025-04-30T06:26:40.771ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b80c4f93-80ee-411a-89f7-14ef325b40db", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76332025-04-30T06:26:40.771ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
76342025-04-30T06:26:40.773ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc9d0068-d653-466c-9a46-03de7433605b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76352025-04-30T06:26:40.773ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
76362025-04-30T06:26:40.778ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "471ca494-097f-45fc-80b3-98e09ba2df02", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76372025-04-30T06:26:40.778ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
76382025-04-30T06:26:40.782ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8385e9b6-8633-4fe6-bb8a-15edec85e62e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76392025-04-30T06:26:40.782ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
76402025-04-30T06:26:40.785ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f8a032a-dfe4-486a-b11d-ba79cc065fef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76412025-04-30T06:26:40.793ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
76422025-04-30T06:26:40.793ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "055dfde2-1b8f-4777-9118-328b4db486dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76432025-04-30T06:26:40.793ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
76442025-04-30T06:26:40.793ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56da3425-8e12-4abd-a5ec-5bb0c31d5767", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76452025-04-30T06:26:40.793ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
76462025-04-30T06:26:40.794ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e851caa-751f-4660-a2b7-8b4ca3f65ac4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76472025-04-30T06:26:40.794ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
76482025-04-30T06:26:40.795ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74ce1158-1626-4267-a4d5-3b0ca295036c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76492025-04-30T06:26:40.795ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76502025-04-30T06:26:40.795ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76512025-04-30T06:26:40.796ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76522025-04-30T06:26:40.796ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76532025-04-30T06:26:40.799ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76542025-04-30T06:26:40.799ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
76552025-04-30T06:26:40.803ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76562025-04-30T06:26:40.818ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76572025-04-30T06:26:40.818ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76582025-04-30T06:26:40.818ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76592025-04-30T06:26:40.818ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76602025-04-30T06:26:40.818ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76612025-04-30T06:26:40.819ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76622025-04-30T06:26:40.835ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76632025-04-30T06:26:40.843ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76642025-04-30T06:26:40.843ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76652025-04-30T06:26:40.843ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76662025-04-30T06:26:40.844ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76672025-04-30T06:26:40.926ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
76682025-04-30T06:26:40.926ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
76692025-04-30T06:26:40.926ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76702025-04-30T06:26:40.926ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76712025-04-30T06:26:40.926ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76722025-04-30T06:26:40.927ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76732025-04-30T06:26:40.930ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76742025-04-30T06:26:40.930ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:52725 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
76752025-04-30T06:26:40.931ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
76762025-04-30T06:26:40.931ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
76772025-04-30T06:26:40.931ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
76782025-04-30T06:26:40.932ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76792025-04-30T06:26:40.934ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 4ed659bf-178a-4064-aa88-e864f53162d4 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76802025-04-30T06:26:40.934ZINFOtest_disk_metrics (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:39352 remote_addr = 127.0.0.1:51857
76812025-04-30T06:26:40.934ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:52725 req_id = d7ab160a-0adf-49ff-aee0-b4fed7b69642 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76822025-04-30T06:26:40.935ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 4ed659bf-178a-4064-aa88-e864f53162d4 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76832025-04-30T06:26:40.935ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 4ed659bf-178a-4064-aa88-e864f53162d4 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76842025-04-30T06:26:40.935ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 4ed659bf-178a-4064-aa88-e864f53162d4 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76852025-04-30T06:26:40.936ZINFOtest_disk_metrics (dropshot): request completed latency_us = 2802 local_addr = [::1]:50989 method = GET remote_addr = [::1]:52725 req_id = d7ab160a-0adf-49ff-aee0-b4fed7b69642 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76862025-04-30T06:26:40.937ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76872025-04-30T06:26:40.944ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76882025-04-30T06:26:40.944ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76892025-04-30T06:26:40.945ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76902025-04-30T06:26:40.948ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76912025-04-30T06:26:40.948ZTRACtest_disk_metrics (dropshot): incoming request local_addr = 127.0.0.1:39352 method = GET remote_addr = 127.0.0.1:51857 req_id = 663d8b17-f388-441a-8cf2-c94ea1699bbd uri = /06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
76922025-04-30T06:26:40.948ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ab160a-0adf-49ff-aee0-b4fed7b69642", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
76932025-04-30T06:26:40.952ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76942025-04-30T06:26:40.952ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76952025-04-30T06:26:40.952ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
76962025-04-30T06:26:40.955ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76972025-04-30T06:26:40.955ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76982025-04-30T06:26:40.955ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76992025-04-30T06:26:40.955ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77002025-04-30T06:26:40.956ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77012025-04-30T06:26:40.961ZINFOtest_disk_metrics (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 13235 local_addr = 127.0.0.1:39352 method = GET remote_addr = 127.0.0.1:51857 req_id = 663d8b17-f388-441a-8cf2-c94ea1699bbd response_code = 200 uri = /06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
77022025-04-30T06:26:40.964ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
77032025-04-30T06:26:40.988ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
77042025-04-30T06:26:40.988ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
77052025-04-30T06:26:40.988ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
77062025-04-30T06:26:40.988ZDEBGtest_disk_metrics (results-sink): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77072025-04-30T06:26:40.988ZDEBGtest_disk_metrics (results-sink-cluster): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77082025-04-30T06:26:40.988ZDEBGtest_disk_metrics (clickhouse-client): unrolling 17 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77092025-04-30T06:26:40.989ZDEBGtest_disk_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77102025-04-30T06:26:40.989ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
77112025-04-30T06:26:40.998ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 64288 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 4ed659bf-178a-4064-aa88-e864f53162d4 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
77122025-04-30T06:26:41.000ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77132025-04-30T06:26:41.000ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
77142025-04-30T06:26:41.001ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77152025-04-30T06:26:41.002ZTRACtest_disk_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77162025-04-30T06:26:41.002ZDEBGtest_disk_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:40.989274073Z } timeseries_name = http_service:request_latency_histogram
77172025-04-30T06:26:41.009ZDEBGtest_disk_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77182025-04-30T06:26:41.009ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77192025-04-30T06:26:41.009ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
77202025-04-30T06:26:41.010ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ed659bf-178a-4064-aa88-e864f53162d4", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
77212025-04-30T06:26:41.010ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
77222025-04-30T06:26:41.012ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
77232025-04-30T06:26:41.012ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77242025-04-30T06:26:41.015ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77252025-04-30T06:26:41.015ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77262025-04-30T06:26:41.015ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77272025-04-30T06:26:41.016ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77282025-04-30T06:26:41.027ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 47f087ab-46fb-4eaa-8e7c-67dee952cbb6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
77292025-04-30T06:26:41.028ZTRACtest_disk_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77302025-04-30T06:26:41.028ZDEBGtest_disk_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:41.009553990Z } timeseries_name = collection_target:cpus_provisioned
77312025-04-30T06:26:41.029ZDEBGtest_disk_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77322025-04-30T06:26:41.029ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77332025-04-30T06:26:41.034ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 47f087ab-46fb-4eaa-8e7c-67dee952cbb6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
77342025-04-30T06:26:41.035ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 47f087ab-46fb-4eaa-8e7c-67dee952cbb6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
77352025-04-30T06:26:41.035ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 47f087ab-46fb-4eaa-8e7c-67dee952cbb6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
77362025-04-30T06:26:41.038ZTRACtest_disk_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77372025-04-30T06:26:41.038ZDEBGtest_disk_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:41.028968221Z } timeseries_name = collection_target:ram_provisioned
77382025-04-30T06:26:41.039ZDEBGtest_disk_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77392025-04-30T06:26:41.039ZTRACtest_disk_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77402025-04-30T06:26:41.041ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
77412025-04-30T06:26:41.042ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
77422025-04-30T06:26:41.042ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
77432025-04-30T06:26:41.042ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
77442025-04-30T06:26:41.043ZTRACtest_disk_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77452025-04-30T06:26:41.043ZDEBGtest_disk_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:41.039107212Z } timeseries_name = collection_target:virtual_disk_space_provisioned
77462025-04-30T06:26:41.055ZDEBGtest_disk_metrics (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
77472025-04-30T06:26:41.055ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
77482025-04-30T06:26:41.058ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 31182 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 47f087ab-46fb-4eaa-8e7c-67dee952cbb6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
77492025-04-30T06:26:41.061ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "47f087ab-46fb-4eaa-8e7c-67dee952cbb6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
77502025-04-30T06:26:41.061ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
77512025-04-30T06:26:41.061ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
77522025-04-30T06:26:41.062ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
77532025-04-30T06:26:41.068ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_string FORMAT Native
77542025-04-30T06:26:41.074ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 10 table_name = fields_string
77552025-04-30T06:26:41.074ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
77562025-04-30T06:26:41.098ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
77572025-04-30T06:26:41.100ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 5 table_name = fields_u16
77582025-04-30T06:26:41.100ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
77592025-04-30T06:26:41.112ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 17 table_name = fields_uuid
77602025-04-30T06:26:41.112ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
77612025-04-30T06:26:41.113ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
77622025-04-30T06:26:41.113ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
77632025-04-30T06:26:41.113ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 765.910586ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
77642025-04-30T06:26:41.113ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
77652025-04-30T06:26:41.113ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77662025-04-30T06:26:41.113ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77672025-04-30T06:26:41.113ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77682025-04-30T06:26:41.113ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77692025-04-30T06:26:41.114ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77702025-04-30T06:26:41.123ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77712025-04-30T06:26:41.129ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77722025-04-30T06:26:41.129ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77732025-04-30T06:26:41.129ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77742025-04-30T06:26:41.130ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77752025-04-30T06:26:41.131ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
77762025-04-30T06:26:41.132ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 5 table_name = measurements_histogramu64
77772025-04-30T06:26:41.132ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
77782025-04-30T06:26:41.137ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
77792025-04-30T06:26:41.137ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
77802025-04-30T06:26:41.138ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 12 table_name = measurements_i64
77812025-04-30T06:26:41.145ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77822025-04-30T06:26:41.145ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 84 29 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .)..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53004 req_id = e4be1fc4-94cb-436b-8910-d1ccfa930502
77832025-04-30T06:26:41.145ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33833,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53004 req_id = e4be1fc4-94cb-436b-8910-d1ccfa930502
77842025-04-30T06:26:41.145ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
77852025-04-30T06:26:41.145ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77862025-04-30T06:26:41.145ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53004 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e4be1fc4-94cb-436b-8910-d1ccfa930502
77872025-04-30T06:26:41.145ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3265, tv_nsec: 943056292 } })
77882025-04-30T06:26:41.145ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
77892025-04-30T06:26:41.145ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ad b1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55610 req_id = f9a0a1f8-7e3c-4034-b86e-b14c4a9d19a7
77902025-04-30T06:26:41.145ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44465,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55610 req_id = f9a0a1f8-7e3c-4034-b86e-b14c4a9d19a7
77912025-04-30T06:26:41.145ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77922025-04-30T06:26:41.145ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55610 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f9a0a1f8-7e3c-4034-b86e-b14c4a9d19a7
77932025-04-30T06:26:41.211ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
77942025-04-30T06:26:41.212ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77952025-04-30T06:26:41.212ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77962025-04-30T06:26:41.212ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77972025-04-30T06:26:41.212ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
77982025-04-30T06:26:41.214ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77992025-04-30T06:26:41.214ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78002025-04-30T06:26:41.214ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78012025-04-30T06:26:41.214ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
78022025-04-30T06:26:41.217ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
78032025-04-30T06:26:41.218ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c1 c7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56936 req_id = 2cb050cd-a8e8-4c0b-ad54-26fb8ae3a3af
78042025-04-30T06:26:41.219ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49607,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56936 req_id = 2cb050cd-a8e8-4c0b-ad54-26fb8ae3a3af
78052025-04-30T06:26:41.219ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
78062025-04-30T06:26:41.219ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49607, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56936 req_id = 2cb050cd-a8e8-4c0b-ad54-26fb8ae3a3af
78072025-04-30T06:26:41.219ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78082025-04-30T06:26:41.231ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78092025-04-30T06:26:41.231ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78102025-04-30T06:26:41.231ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78112025-04-30T06:26:41.231ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78122025-04-30T06:26:41.232ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
78132025-04-30T06:26:41.232ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
78142025-04-30T06:26:41.232ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
78152025-04-30T06:26:41.235ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
78162025-04-30T06:26:41.235ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
78172025-04-30T06:26:41.245ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78182025-04-30T06:26:41.246ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78192025-04-30T06:26:41.246ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78202025-04-30T06:26:41.247ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
78212025-04-30T06:26:41.247ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78222025-04-30T06:26:41.248ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "945a9f7e-2747-45c7-947a-2988175f1754", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78232025-04-30T06:26:41.248ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
78242025-04-30T06:26:41.251ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26f3f8aa-db09-4e00-9c5c-6d752ddf7919", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78252025-04-30T06:26:41.251ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
78262025-04-30T06:26:41.255ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d6854d4-803b-484d-bedd-b5df609b7553", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78272025-04-30T06:26:41.255ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
78282025-04-30T06:26:41.257ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bac7d2f0-e743-4761-8fe2-166f1243b4db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78292025-04-30T06:26:41.257ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
78302025-04-30T06:26:41.262ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4489df9d-6fa5-41b7-9e87-e3467e1462a3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78312025-04-30T06:26:41.262ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
78322025-04-30T06:26:41.264ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1e26740-34ba-4cce-a266-c9996c32dd14", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78332025-04-30T06:26:41.264ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
78342025-04-30T06:26:41.267ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed2b6990-1280-433d-b8ff-e561a58c451d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78352025-04-30T06:26:41.267ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
78362025-04-30T06:26:41.270ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50da4162-e042-4400-bfa5-a03de5c237b6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78372025-04-30T06:26:41.270ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
78382025-04-30T06:26:41.271ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c62115e2-fc77-48a6-8805-45f730e08102", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78392025-04-30T06:26:41.271ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
78402025-04-30T06:26:41.273ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "267934e2-bd74-4c4e-a74a-564fca951aa6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78412025-04-30T06:26:41.273ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
78422025-04-30T06:26:41.274ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "400ebc1c-f2e9-45ba-b38e-303b1f2f26c1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78432025-04-30T06:26:41.275ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
78442025-04-30T06:26:41.276ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4edf36db-558e-4b0e-8cb7-985de4218bd6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78452025-04-30T06:26:41.276ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
78462025-04-30T06:26:41.281ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
78472025-04-30T06:26:41.282ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb69faf5-a3ca-4ab4-aaaf-79a83c5f6a36", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78482025-04-30T06:26:41.283ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
78492025-04-30T06:26:41.287ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bb2e319-571f-4bae-91ef-91ae027612ae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78502025-04-30T06:26:41.287ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
78512025-04-30T06:26:41.292ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe792fde-b3af-4fd8-a54c-6de59e96b287", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78522025-04-30T06:26:41.292ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
78532025-04-30T06:26:41.300ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b32b8231-6eb6-47f5-90af-9c541b80bac3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78542025-04-30T06:26:41.300ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
78552025-04-30T06:26:41.302ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2352b42-7894-4c01-b20b-cd17a393eca0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78562025-04-30T06:26:41.303ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
78572025-04-30T06:26:41.304ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd14f226-383e-4a33-a047-7fc295f144b2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78582025-04-30T06:26:41.305ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
78592025-04-30T06:26:41.307ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e531dc23-bcc9-4d4f-88bd-9f2631d1b982", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78602025-04-30T06:26:41.307ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
78612025-04-30T06:26:41.308ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60aabe15-b7af-45af-b459-395b88b0e393", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78622025-04-30T06:26:41.308ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
78632025-04-30T06:26:41.310ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f90e6ea-22ac-441e-bbd6-ee0715b00c80", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78642025-04-30T06:26:41.310ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
78652025-04-30T06:26:41.312ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a03bdc3f-c489-4bd9-8e7e-630f1a9e187b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78662025-04-30T06:26:41.312ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
78672025-04-30T06:26:41.314ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0937d8c8-bdfd-4c77-983e-1c10dda6d05a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78682025-04-30T06:26:41.314ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
78692025-04-30T06:26:41.316ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
78702025-04-30T06:26:41.317ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b52a62f-32d1-444b-a692-a3e3b19ab4f5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78712025-04-30T06:26:41.317ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
78722025-04-30T06:26:41.320ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e60b586-8d9a-4213-9cd3-8f7ad8ad01da", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78732025-04-30T06:26:41.328ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
78742025-04-30T06:26:41.330ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c677ec6c-8076-4823-a7ee-512b3fe42aee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78752025-04-30T06:26:41.331ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
78762025-04-30T06:26:41.332ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78772025-04-30T06:26:41.332ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78782025-04-30T06:26:41.332ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78792025-04-30T06:26:41.332ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78802025-04-30T06:26:41.333ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78812025-04-30T06:26:41.335ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "852c09b6-2cd0-4adc-bae8-a4a3dd58e6d4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78822025-04-30T06:26:41.335ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
78832025-04-30T06:26:41.336ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78842025-04-30T06:26:41.345ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78852025-04-30T06:26:41.345ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78862025-04-30T06:26:41.345ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78872025-04-30T06:26:41.347ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78882025-04-30T06:26:41.351ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "094bc641-ebc9-40b6-865e-63b66ce58f11", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78892025-04-30T06:26:41.351ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
78902025-04-30T06:26:41.353ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3236398c-a273-4006-805d-e62a664b55be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78912025-04-30T06:26:41.354ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
78922025-04-30T06:26:41.354ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78932025-04-30T06:26:41.354ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78942025-04-30T06:26:41.355ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78952025-04-30T06:26:41.358ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e94dab7-191a-4942-8674-e15d0c65f194", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78962025-04-30T06:26:41.358ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
78972025-04-30T06:26:41.359ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7873c2c7-e22b-4a84-87f3-647d8134bc57", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
78982025-04-30T06:26:41.360ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
78992025-04-30T06:26:41.362ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de2fbc41-ad33-4b93-88b7-861190b77966", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79002025-04-30T06:26:41.362ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
79012025-04-30T06:26:41.363ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4f7b801-468b-4fbd-be96-1b1c0e539744", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79022025-04-30T06:26:41.364ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
79032025-04-30T06:26:41.366ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6842416f-b25e-4338-90bf-b478d8624c35", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79042025-04-30T06:26:41.366ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
79052025-04-30T06:26:41.369ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d26929f-9149-494b-bc5c-606455c1d05d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79062025-04-30T06:26:41.369ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
79072025-04-30T06:26:41.372ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d74af876-0534-4164-92bc-01d678e4e1de", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79082025-04-30T06:26:41.373ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
79092025-04-30T06:26:41.387ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df55ae1-de15-42a3-804a-9f1b19680aae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79102025-04-30T06:26:41.387ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
79112025-04-30T06:26:41.390ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6f48114-7d9f-4c3a-8271-c6aa0fc68fb0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79122025-04-30T06:26:41.391ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
79132025-04-30T06:26:41.394ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
79142025-04-30T06:26:41.399ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ddaa443-2f4d-43de-9f38-56b4f6e7956b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79152025-04-30T06:26:41.399ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
79162025-04-30T06:26:41.429ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c2d0d0-d8d8-4ef8-80e2-5d8103748c6d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79172025-04-30T06:26:41.429ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
79182025-04-30T06:26:41.433ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bb8a5fc-b430-43e4-83e2-6566ea30993d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79192025-04-30T06:26:41.433ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
79202025-04-30T06:26:41.434ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba08719-f0b3-4fab-8ca1-65f249922232", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79212025-04-30T06:26:41.435ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
79222025-04-30T06:26:41.438ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26549fa5-49cb-4047-95a0-567974364ed9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79232025-04-30T06:26:41.438ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
79242025-04-30T06:26:41.440ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ea886f7-bc13-4922-b2b7-4ca8beca1a26", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79252025-04-30T06:26:41.441ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
79262025-04-30T06:26:41.443ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "417049ff-6d30-4ee5-8a2e-8a75ce56b50e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79272025-04-30T06:26:41.444ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79282025-04-30T06:26:41.444ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79292025-04-30T06:26:41.444ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79302025-04-30T06:26:41.444ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79312025-04-30T06:26:41.446ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79322025-04-30T06:26:41.447ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79332025-04-30T06:26:41.447ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79342025-04-30T06:26:41.447ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79352025-04-30T06:26:41.447ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
79362025-04-30T06:26:41.449ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79372025-04-30T06:26:41.452ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79382025-04-30T06:26:41.466ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79392025-04-30T06:26:41.466ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79402025-04-30T06:26:41.466ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79412025-04-30T06:26:41.466ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79422025-04-30T06:26:41.466ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79432025-04-30T06:26:41.467ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79442025-04-30T06:26:41.471ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79452025-04-30T06:26:41.491ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79462025-04-30T06:26:41.491ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79472025-04-30T06:26:41.491ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
79482025-04-30T06:26:41.492ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79492025-04-30T06:26:41.495ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79502025-04-30T06:26:41.504ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79512025-04-30T06:26:41.504ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79522025-04-30T06:26:41.504ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79532025-04-30T06:26:41.505ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79542025-04-30T06:26:41.508ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79552025-04-30T06:26:41.508ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
79562025-04-30T06:26:41.509ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79572025-04-30T06:26:41.582ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
79582025-04-30T06:26:41.583ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
79592025-04-30T06:26:41.583ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:51087 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
79602025-04-30T06:26:41.584ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:51087 req_id = 21b24685-3e2f-4f9c-a990-c7503bacd9ea sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79612025-04-30T06:26:41.584ZINFOtest_disk_metrics (dropshot): request completed latency_us = 549 local_addr = [::1]:50989 method = GET remote_addr = [::1]:51087 req_id = 21b24685-3e2f-4f9c-a990-c7503bacd9ea response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79622025-04-30T06:26:41.585ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b24685-3e2f-4f9c-a990-c7503bacd9ea", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
79632025-04-30T06:26:41.587ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79642025-04-30T06:26:41.587ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79652025-04-30T06:26:41.587ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
79662025-04-30T06:26:41.591ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79672025-04-30T06:26:41.591ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79682025-04-30T06:26:41.591ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79692025-04-30T06:26:41.592ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79702025-04-30T06:26:41.593ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79712025-04-30T06:26:41.615ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79722025-04-30T06:26:41.633ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79732025-04-30T06:26:41.634ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79742025-04-30T06:26:41.634ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79752025-04-30T06:26:41.635ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79762025-04-30T06:26:41.691ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
79772025-04-30T06:26:41.695ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
79782025-04-30T06:26:41.696ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
79792025-04-30T06:26:41.696ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 582.717227ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
79802025-04-30T06:26:41.696ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
79812025-04-30T06:26:41.696ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79822025-04-30T06:26:41.696ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79832025-04-30T06:26:41.696ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79842025-04-30T06:26:41.696ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79852025-04-30T06:26:41.697ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79862025-04-30T06:26:41.701ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79872025-04-30T06:26:41.708ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79882025-04-30T06:26:41.708ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79892025-04-30T06:26:41.708ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79902025-04-30T06:26:41.709ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79912025-04-30T06:26:41.714ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
79922025-04-30T06:26:41.714ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
79932025-04-30T06:26:41.715ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7f 33 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .3..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61055 req_id = 5d141b78-a28d-451a-a214-52bc03ffd5ab
79942025-04-30T06:26:41.716ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32563,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61055 req_id = 5d141b78-a28d-451a-a214-52bc03ffd5ab
79952025-04-30T06:26:41.716ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
79962025-04-30T06:26:41.716ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79972025-04-30T06:26:41.716ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61055 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5d141b78-a28d-451a-a214-52bc03ffd5ab
79982025-04-30T06:26:41.717ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3266, tv_nsec: 516002260 } })
79992025-04-30T06:26:41.717ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
80002025-04-30T06:26:41.717ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 35 9d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 5...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56557 req_id = 89e531c7-232f-4ab5-af32-19278ecdfc78
80012025-04-30T06:26:41.717ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13725,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56557 req_id = 89e531c7-232f-4ab5-af32-19278ecdfc78
80022025-04-30T06:26:41.717ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80032025-04-30T06:26:41.717ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56557 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 89e531c7-232f-4ab5-af32-19278ecdfc78
80042025-04-30T06:26:41.817ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
80052025-04-30T06:26:41.818ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80062025-04-30T06:26:41.818ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80072025-04-30T06:26:41.818ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80082025-04-30T06:26:41.818ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80092025-04-30T06:26:41.818ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80102025-04-30T06:26:41.820ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ba 92 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57348 req_id = 9a9a88ed-213c-441a-a593-ef96eb391cba
80112025-04-30T06:26:41.820ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47762,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57348 req_id = 9a9a88ed-213c-441a-a593-ef96eb391cba
80122025-04-30T06:26:41.820ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
80132025-04-30T06:26:41.821ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47762, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57348 req_id = 9a9a88ed-213c-441a-a593-ef96eb391cba
80142025-04-30T06:26:41.821ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
80152025-04-30T06:26:41.821ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
80162025-04-30T06:26:41.821ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
80172025-04-30T06:26:41.822ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80182025-04-30T06:26:41.828ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80192025-04-30T06:26:41.828ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80202025-04-30T06:26:41.829ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80212025-04-30T06:26:41.829ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80222025-04-30T06:26:41.830ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "92a90465-ce8b-46eb-b70e-cc4d2f3969d0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80232025-04-30T06:26:41.831ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
80242025-04-30T06:26:41.832ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80252025-04-30T06:26:41.833ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80262025-04-30T06:26:41.833ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80272025-04-30T06:26:41.833ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac025a8a-d371-4638-a024-ab0912d07ed7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80282025-04-30T06:26:41.834ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
80292025-04-30T06:26:41.836ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74ccc217-1f49-4a22-b4fa-e09d71dbc7ca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80302025-04-30T06:26:41.836ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
80312025-04-30T06:26:41.839ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5f45d0b-5d4a-4c99-9d8b-b87aab020e82", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80322025-04-30T06:26:41.839ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
80332025-04-30T06:26:41.841ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "665b1894-ed38-4108-b1ad-a030ea11b1cd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80342025-04-30T06:26:41.842ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
80352025-04-30T06:26:41.844ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3b81745-eb9c-4c6a-9c85-df945b56f944", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80362025-04-30T06:26:41.844ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
80372025-04-30T06:26:41.862ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57ac4ed0-8c8f-4bae-91f0-33077835fd6c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80382025-04-30T06:26:41.862ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
80392025-04-30T06:26:41.865ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4284f011-5b8d-489e-985b-a4c959b3dff9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80402025-04-30T06:26:41.866ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
80412025-04-30T06:26:41.867ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f4b32e-f98a-40db-873f-15248fe15e21", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80422025-04-30T06:26:41.867ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
80432025-04-30T06:26:41.868ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98870230-7bc3-4db2-a43d-6b356b66eb0b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80442025-04-30T06:26:41.868ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
80452025-04-30T06:26:41.869ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e267a3f9-06c8-43c7-8dc6-289b6665e7f5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80462025-04-30T06:26:41.870ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
80472025-04-30T06:26:41.871ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbabeffb-d7ca-4dde-84da-9d94c91fbdc8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80482025-04-30T06:26:41.871ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
80492025-04-30T06:26:41.873ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75d9dfcb-e854-4481-b268-217aef669e44", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80502025-04-30T06:26:41.873ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
80512025-04-30T06:26:41.877ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
80522025-04-30T06:26:41.879ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0800b464-03ef-4a1e-bb4b-343d06274947", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80532025-04-30T06:26:41.879ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
80542025-04-30T06:26:41.881ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b36d44da-341e-4864-a109-d73bc741e94c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80552025-04-30T06:26:41.882ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
80562025-04-30T06:26:41.884ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "504725bb-7fb5-4d82-999a-12d51c64a01c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80572025-04-30T06:26:41.884ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
80582025-04-30T06:26:41.887ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36efb655-bfc7-4e75-871a-dfaa892b2fb5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80592025-04-30T06:26:41.888ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
80602025-04-30T06:26:41.893ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e18ee4c-c102-4bed-a896-19d9b520fcb4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80612025-04-30T06:26:41.893ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
80622025-04-30T06:26:41.897ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce526ee7-c34d-4cbf-8aa9-27ebfea2c60a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80632025-04-30T06:26:41.897ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
80642025-04-30T06:26:41.898ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a108b29c-8384-40ac-8ff5-5f0fae5b1b83", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80652025-04-30T06:26:41.899ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
80662025-04-30T06:26:41.900ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e16386a-3300-4a9b-9f2f-8be3e0d8d42d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80672025-04-30T06:26:41.900ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
80682025-04-30T06:26:41.907ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ba898d9-fc80-49c0-ac79-f2e696925885", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80692025-04-30T06:26:41.907ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
80702025-04-30T06:26:41.908ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d953101-6793-43ff-b793-0050618ffea5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80712025-04-30T06:26:41.908ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
80722025-04-30T06:26:41.910ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a342089c-41d1-41d0-9e84-27ccbe8baff9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80732025-04-30T06:26:41.910ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
80742025-04-30T06:26:41.913ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9154036b-d5d1-44f4-8bcd-5ece01dc285f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80752025-04-30T06:26:41.913ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
80762025-04-30T06:26:41.915ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba89659c-9cfe-461d-a66c-8c3426446828", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80772025-04-30T06:26:41.915ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
80782025-04-30T06:26:41.918ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc0bdf9b-125a-482b-abc1-c601e5ad4124", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80792025-04-30T06:26:41.918ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
80802025-04-30T06:26:41.922ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ad9808b-93b7-4c73-94cb-099723d9a10d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80812025-04-30T06:26:41.922ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
80822025-04-30T06:26:41.924ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9acdae9-4068-46c7-b3d7-24442ac956b2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80832025-04-30T06:26:41.924ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
80842025-04-30T06:26:41.926ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "179ad0b4-93a1-4d0d-b12c-9dfbe2ab841e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80852025-04-30T06:26:41.926ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
80862025-04-30T06:26:41.927ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65c2cf82-41a9-4115-8b23-9ca1fb449f2f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
80872025-04-30T06:26:41.927ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
80882025-04-30T06:26:41.928ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80892025-04-30T06:26:41.928ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80902025-04-30T06:26:41.928ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80912025-04-30T06:26:41.928ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80922025-04-30T06:26:41.929ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80932025-04-30T06:26:41.932ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80942025-04-30T06:26:41.943ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80952025-04-30T06:26:41.943ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80962025-04-30T06:26:41.943ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80972025-04-30T06:26:41.943ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80982025-04-30T06:26:41.948ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80992025-04-30T06:26:41.948ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
81002025-04-30T06:26:41.949ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81012025-04-30T06:26:41.949ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22d869dd-b0f9-44a3-9544-18bdfd75dfba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81022025-04-30T06:26:41.950ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
81032025-04-30T06:26:41.967ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30af3dc8-2bc8-4596-b773-37fb219c3986", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81042025-04-30T06:26:41.967ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
81052025-04-30T06:26:41.985ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81062025-04-30T06:26:41.985ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "005cd16d-fd2f-4777-ad99-b0aa07a266bc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81072025-04-30T06:26:41.986ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
81082025-04-30T06:26:41.986ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81092025-04-30T06:26:41.998ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81102025-04-30T06:26:41.999ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81112025-04-30T06:26:41.999ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81122025-04-30T06:26:41.999ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81132025-04-30T06:26:42.015ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d1f21b2-585d-49d9-b1c3-db5b210c36f3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81142025-04-30T06:26:42.016ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
81152025-04-30T06:26:42.020ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5bc0ea8-9bf7-458d-9f10-06e6982ca130", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81162025-04-30T06:26:42.021ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
81172025-04-30T06:26:42.023ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a45ce7-dc5e-4302-94e7-7df7368fbf77", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81182025-04-30T06:26:42.024ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
81192025-04-30T06:26:42.027ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
81202025-04-30T06:26:42.028ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "609447ba-9676-46d7-bce3-bb96d1b0f593", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81212025-04-30T06:26:42.028ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
81222025-04-30T06:26:42.031ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77e8ef3e-a972-4bf6-a8bd-4b17bf9f5ae2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81232025-04-30T06:26:42.031ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
81242025-04-30T06:26:42.033ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76c49708-aaeb-4096-85d0-ca9cd34e091a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81252025-04-30T06:26:42.033ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
81262025-04-30T06:26:42.035ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d99fdbb2-e123-4788-8a10-334b8244e54e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81272025-04-30T06:26:42.035ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
81282025-04-30T06:26:42.036ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54767c34-73ec-4808-b1fb-76282c67a6d2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81292025-04-30T06:26:42.036ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
81302025-04-30T06:26:42.037ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a34d31-7d10-4b60-9bcc-7027200d18f5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81312025-04-30T06:26:42.037ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
81322025-04-30T06:26:42.039ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "104b9ea0-b35d-4d39-aa3f-02d6ccdf2f46", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81332025-04-30T06:26:42.039ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
81342025-04-30T06:26:42.040ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "121276b8-cc98-48a3-b55c-18d6ed14e3d6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81352025-04-30T06:26:42.040ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81362025-04-30T06:26:42.040ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81372025-04-30T06:26:42.040ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81382025-04-30T06:26:42.040ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81392025-04-30T06:26:42.041ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81402025-04-30T06:26:42.044ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81412025-04-30T06:26:42.048ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81422025-04-30T06:26:42.049ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81432025-04-30T06:26:42.049ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81442025-04-30T06:26:42.049ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81452025-04-30T06:26:42.049ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81462025-04-30T06:26:42.049ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81472025-04-30T06:26:42.052ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81482025-04-30T06:26:42.057ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81492025-04-30T06:26:42.057ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81502025-04-30T06:26:42.057ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81512025-04-30T06:26:42.058ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81522025-04-30T06:26:42.131ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
81532025-04-30T06:26:42.131ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
81542025-04-30T06:26:42.131ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81552025-04-30T06:26:42.131ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81562025-04-30T06:26:42.131ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81572025-04-30T06:26:42.131ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
81582025-04-30T06:26:42.133ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81592025-04-30T06:26:42.133ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:44754 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
81602025-04-30T06:26:42.136ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:44754 req_id = 4fcbcce2-577c-48f8-8708-4f39b19a7ef7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81612025-04-30T06:26:42.137ZINFOtest_disk_metrics (dropshot): request completed latency_us = 491 local_addr = [::1]:50989 method = GET remote_addr = [::1]:44754 req_id = 4fcbcce2-577c-48f8-8708-4f39b19a7ef7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81622025-04-30T06:26:42.138ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fcbcce2-577c-48f8-8708-4f39b19a7ef7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
81632025-04-30T06:26:42.142ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81642025-04-30T06:26:42.142ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81652025-04-30T06:26:42.142ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
81662025-04-30T06:26:42.144ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81672025-04-30T06:26:42.144ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81682025-04-30T06:26:42.144ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81692025-04-30T06:26:42.144ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81702025-04-30T06:26:42.145ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81712025-04-30T06:26:42.146ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81722025-04-30T06:26:42.152ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81732025-04-30T06:26:42.152ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81742025-04-30T06:26:42.152ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
81752025-04-30T06:26:42.152ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81762025-04-30T06:26:42.157ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81772025-04-30T06:26:42.164ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81782025-04-30T06:26:42.164ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81792025-04-30T06:26:42.164ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81802025-04-30T06:26:42.165ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81812025-04-30T06:26:42.168ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81822025-04-30T06:26:42.168ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
81832025-04-30T06:26:42.168ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81842025-04-30T06:26:42.177ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81852025-04-30T06:26:42.177ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81862025-04-30T06:26:42.178ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
81872025-04-30T06:26:42.178ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
81882025-04-30T06:26:42.184ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
81892025-04-30T06:26:42.184ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
81902025-04-30T06:26:42.188ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
81912025-04-30T06:26:42.188ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81922025-04-30T06:26:42.210ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
81932025-04-30T06:26:42.210ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
81942025-04-30T06:26:42.211ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 514.582852ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
81952025-04-30T06:26:42.211ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
81962025-04-30T06:26:42.211ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81972025-04-30T06:26:42.211ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81982025-04-30T06:26:42.211ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81992025-04-30T06:26:42.212ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82002025-04-30T06:26:42.212ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82012025-04-30T06:26:42.213ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
82022025-04-30T06:26:42.227ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82032025-04-30T06:26:42.234ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82042025-04-30T06:26:42.234ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82052025-04-30T06:26:42.234ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82062025-04-30T06:26:42.235ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82072025-04-30T06:26:42.243ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
82082025-04-30T06:26:42.243ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
82092025-04-30T06:26:42.246ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a8 28 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .(..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59076 req_id = f717bdda-0f00-4229-aa67-5c95114797b2
82102025-04-30T06:26:42.246ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43048,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59076 req_id = f717bdda-0f00-4229-aa67-5c95114797b2
82112025-04-30T06:26:42.246ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
82122025-04-30T06:26:42.247ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82132025-04-30T06:26:42.247ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59076 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f717bdda-0f00-4229-aa67-5c95114797b2
82142025-04-30T06:26:42.247ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3267, tv_nsec: 46511384 } })
82152025-04-30T06:26:42.247ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
82162025-04-30T06:26:42.247ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 26 28 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e &(..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64289 req_id = 40b3698c-c0a3-485b-a169-dcb142c43355
82172025-04-30T06:26:42.248ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9768,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64289 req_id = 40b3698c-c0a3-485b-a169-dcb142c43355
82182025-04-30T06:26:42.248ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82192025-04-30T06:26:42.248ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64289 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 40b3698c-c0a3-485b-a169-dcb142c43355
82202025-04-30T06:26:42.326ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
82212025-04-30T06:26:42.327ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82222025-04-30T06:26:42.327ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82232025-04-30T06:26:42.327ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82242025-04-30T06:26:42.327ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82252025-04-30T06:26:42.328ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82262025-04-30T06:26:42.328ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
82272025-04-30T06:26:42.329ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b0 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .o..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51357 req_id = a67aba54-ef6a-4cc3-bf3f-37dcb8da753b
82282025-04-30T06:26:42.329ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45167,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51357 req_id = a67aba54-ef6a-4cc3-bf3f-37dcb8da753b
82292025-04-30T06:26:42.329ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
82302025-04-30T06:26:42.329ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45167, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51357 req_id = a67aba54-ef6a-4cc3-bf3f-37dcb8da753b
82312025-04-30T06:26:42.329ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
82322025-04-30T06:26:42.329ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
82332025-04-30T06:26:42.329ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
82342025-04-30T06:26:42.330ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82352025-04-30T06:26:42.338ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82362025-04-30T06:26:42.338ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82372025-04-30T06:26:42.338ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82382025-04-30T06:26:42.338ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82392025-04-30T06:26:42.351ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8274a59b-9bca-4008-ba7d-b5ccf013f12b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82402025-04-30T06:26:42.351ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
82412025-04-30T06:26:42.351ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82422025-04-30T06:26:42.351ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82432025-04-30T06:26:42.352ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82442025-04-30T06:26:42.353ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56ac6103-88a9-4825-8be5-252c0be3290d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82452025-04-30T06:26:42.353ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
82462025-04-30T06:26:42.358ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d85ad2b-3688-4488-bc5c-c2b9f8a87e9c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82472025-04-30T06:26:42.358ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
82482025-04-30T06:26:42.371ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73e71b92-9ed9-44ff-bc88-03a4555627a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82492025-04-30T06:26:42.371ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
82502025-04-30T06:26:42.379ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db7c6708-08b2-43d0-909a-0b2d7535a100", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82512025-04-30T06:26:42.379ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
82522025-04-30T06:26:42.388ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ced4226-f355-4678-9fdd-3c8903fc7770", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82532025-04-30T06:26:42.388ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
82542025-04-30T06:26:42.394ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
82552025-04-30T06:26:42.404ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c89aa97-5ece-48b6-b055-5289204aaa5f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82562025-04-30T06:26:42.404ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
82572025-04-30T06:26:42.409ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efd98974-7a0f-4b78-9cfd-680664f63767", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82582025-04-30T06:26:42.409ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
82592025-04-30T06:26:42.418ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "096849b0-13ed-404f-b1ee-23e2e9aa1a4f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82602025-04-30T06:26:42.418ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
82612025-04-30T06:26:42.423ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40f2131-a116-46e5-97e4-2798980307a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82622025-04-30T06:26:42.423ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
82632025-04-30T06:26:42.425ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "458a59ef-2f06-4650-824e-d282c4d6a80a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82642025-04-30T06:26:42.426ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
82652025-04-30T06:26:42.426ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8997811-8498-4a54-acf8-8fdd5e3eeddb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82662025-04-30T06:26:42.427ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
82672025-04-30T06:26:42.429ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a81f1b07-d54c-4448-b624-3565ae0b1649", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82682025-04-30T06:26:42.429ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
82692025-04-30T06:26:42.431ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fad9214b-7ce5-4d37-90d6-b85daa6aa5de", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82702025-04-30T06:26:42.431ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
82712025-04-30T06:26:42.434ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1593af5d-9ef5-41dd-8f24-cb980a5f81b3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82722025-04-30T06:26:42.434ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
82732025-04-30T06:26:42.444ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82742025-04-30T06:26:42.444ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82752025-04-30T06:26:42.444ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82762025-04-30T06:26:42.444ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82772025-04-30T06:26:42.445ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82782025-04-30T06:26:42.446ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fde2df3b-d0f1-4d64-b6c4-70e219d8bfb1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82792025-04-30T06:26:42.446ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
82802025-04-30T06:26:42.448ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82812025-04-30T06:26:42.454ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82822025-04-30T06:26:42.454ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82832025-04-30T06:26:42.454ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82842025-04-30T06:26:42.454ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82852025-04-30T06:26:42.458ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82862025-04-30T06:26:42.458ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82872025-04-30T06:26:42.458ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82882025-04-30T06:26:42.464ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc019f8f-f268-49df-a211-c9b09f793b04", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82892025-04-30T06:26:42.464ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
82902025-04-30T06:26:42.469ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "133a3605-9f86-445e-bfaf-5c9a0918b336", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82912025-04-30T06:26:42.469ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
82922025-04-30T06:26:42.477ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76c2378e-989e-4e44-b8d8-1b8033f53d74", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82932025-04-30T06:26:42.477ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
82942025-04-30T06:26:42.479ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54304469-c1c5-44c8-9801-9a811c985cd6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82952025-04-30T06:26:42.479ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
82962025-04-30T06:26:42.480ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab7bcc3-3971-47c3-86ab-20541a148d74", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82972025-04-30T06:26:42.481ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
82982025-04-30T06:26:42.482ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4332712-e65a-494b-a417-3de625e4bec7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
82992025-04-30T06:26:42.483ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
83002025-04-30T06:26:42.486ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "535a20df-5b8a-486e-8839-6940e31a6689", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83012025-04-30T06:26:42.486ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
83022025-04-30T06:26:42.489ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7611fb3d-0c02-4ae8-bcb8-f5d18f17a7a7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83032025-04-30T06:26:42.490ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
83042025-04-30T06:26:42.497ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ea64659-5250-4fb6-94f2-f0c1c30e29ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83052025-04-30T06:26:42.498ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
83062025-04-30T06:26:42.498ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
83072025-04-30T06:26:42.515ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93624e21-0cf8-47d7-abc3-52b469c4f73a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83082025-04-30T06:26:42.515ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
83092025-04-30T06:26:42.522ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d96146c-8ea0-4f6d-bb63-259ae9cac6d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83102025-04-30T06:26:42.522ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
83112025-04-30T06:26:42.535ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e33e0a7-4c7d-4111-8327-34ac8dc51882", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83122025-04-30T06:26:42.535ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
83132025-04-30T06:26:42.537ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b1b7712-e889-454a-9db0-34f75dc9090a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83142025-04-30T06:26:42.538ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
83152025-04-30T06:26:42.541ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7077e7d6-916e-48b4-9d30-4610c188e7cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83162025-04-30T06:26:42.541ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
83172025-04-30T06:26:42.542ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a8e30df-578b-47ad-8c1f-aae854929539", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83182025-04-30T06:26:42.542ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
83192025-04-30T06:26:42.543ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af064f20-857b-4726-b0c4-5af002e2e9c9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83202025-04-30T06:26:42.544ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
83212025-04-30T06:26:42.544ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63b2b879-ec94-4b41-bca7-279e91186712", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83222025-04-30T06:26:42.544ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
83232025-04-30T06:26:42.546ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0b642b9-f7e4-4d5d-a4f5-b9c39e6a3ba2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83242025-04-30T06:26:42.546ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
83252025-04-30T06:26:42.548ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aa940ca-1c3a-4efa-b768-8d7167ba4282", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83262025-04-30T06:26:42.548ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
83272025-04-30T06:26:42.549ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83282025-04-30T06:26:42.549ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83292025-04-30T06:26:42.549ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83302025-04-30T06:26:42.549ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83312025-04-30T06:26:42.551ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83322025-04-30T06:26:42.552ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70ab2493-936b-4486-852b-98e39291ee05", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83332025-04-30T06:26:42.553ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
83342025-04-30T06:26:42.555ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83352025-04-30T06:26:42.565ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83362025-04-30T06:26:42.565ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83372025-04-30T06:26:42.565ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83382025-04-30T06:26:42.566ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83392025-04-30T06:26:42.568ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "494d5243-2363-4172-83dd-61b5383e152b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83402025-04-30T06:26:42.568ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
83412025-04-30T06:26:42.570ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83422025-04-30T06:26:42.570ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83432025-04-30T06:26:42.571ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83442025-04-30T06:26:42.571ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b567457-5664-42aa-90c7-4eff5d8740b6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83452025-04-30T06:26:42.571ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
83462025-04-30T06:26:42.576ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d13aa81c-7632-4f06-b02f-7dfd37f1bd41", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83472025-04-30T06:26:42.576ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
83482025-04-30T06:26:42.578ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef480d2b-9405-4d18-bbfc-22f52196e47f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83492025-04-30T06:26:42.578ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
83502025-04-30T06:26:42.580ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff598331-32d5-401f-8a9d-ef26d0f6fa6f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83512025-04-30T06:26:42.580ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
83522025-04-30T06:26:42.581ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65ba2f2b-fbaf-4754-8bd2-40b06a86e9dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83532025-04-30T06:26:42.581ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
83542025-04-30T06:26:42.582ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90920938-ff07-4ea9-b364-a5a84a38b239", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83552025-04-30T06:26:42.582ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
83562025-04-30T06:26:42.585ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2171177c-ce3e-4ba8-bc93-aaf20629faca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83572025-04-30T06:26:42.585ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
83582025-04-30T06:26:42.587ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b06862e7-5a64-40d8-9806-5b095a983176", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83592025-04-30T06:26:42.587ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83602025-04-30T06:26:42.587ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83612025-04-30T06:26:42.587ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83622025-04-30T06:26:42.587ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83632025-04-30T06:26:42.587ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83642025-04-30T06:26:42.590ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83652025-04-30T06:26:42.594ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83662025-04-30T06:26:42.594ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83672025-04-30T06:26:42.594ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83682025-04-30T06:26:42.594ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83692025-04-30T06:26:42.594ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83702025-04-30T06:26:42.594ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83712025-04-30T06:26:42.611ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83722025-04-30T06:26:42.614ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83732025-04-30T06:26:42.615ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83742025-04-30T06:26:42.615ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83752025-04-30T06:26:42.615ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83762025-04-30T06:26:42.712ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
83772025-04-30T06:26:42.712ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
83782025-04-30T06:26:42.712ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
83792025-04-30T06:26:42.713ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:54795 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83802025-04-30T06:26:42.713ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:54795 req_id = ab4f3d49-db00-4239-a05e-cce44d0a8949 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83812025-04-30T06:26:42.714ZINFOtest_disk_metrics (dropshot): request completed latency_us = 573 local_addr = [::1]:50989 method = GET remote_addr = [::1]:54795 req_id = ab4f3d49-db00-4239-a05e-cce44d0a8949 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83822025-04-30T06:26:42.714ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab4f3d49-db00-4239-a05e-cce44d0a8949", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
83832025-04-30T06:26:42.717ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83842025-04-30T06:26:42.717ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83852025-04-30T06:26:42.717ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
83862025-04-30T06:26:42.719ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83872025-04-30T06:26:42.719ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83882025-04-30T06:26:42.719ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83892025-04-30T06:26:42.719ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83902025-04-30T06:26:42.720ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83912025-04-30T06:26:42.723ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83922025-04-30T06:26:42.727ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83932025-04-30T06:26:42.728ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83942025-04-30T06:26:42.728ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83952025-04-30T06:26:42.729ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83962025-04-30T06:26:42.764ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83972025-04-30T06:26:42.764ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83982025-04-30T06:26:42.764ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83992025-04-30T06:26:42.764ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84002025-04-30T06:26:42.765ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84012025-04-30T06:26:42.765ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
84022025-04-30T06:26:42.765ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
84032025-04-30T06:26:42.765ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 554.885383ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84042025-04-30T06:26:42.765ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
84052025-04-30T06:26:42.766ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84062025-04-30T06:26:42.766ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84072025-04-30T06:26:42.766ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84082025-04-30T06:26:42.766ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84092025-04-30T06:26:42.766ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84102025-04-30T06:26:42.769ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84112025-04-30T06:26:42.777ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84122025-04-30T06:26:42.777ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84132025-04-30T06:26:42.777ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84142025-04-30T06:26:42.778ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84152025-04-30T06:26:42.779ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84162025-04-30T06:26:42.783ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84172025-04-30T06:26:42.783ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84182025-04-30T06:26:42.783ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84192025-04-30T06:26:42.784ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84202025-04-30T06:26:42.788ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84212025-04-30T06:26:42.788ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84222025-04-30T06:26:42.789ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84232025-04-30T06:26:42.789ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
84242025-04-30T06:26:42.789ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84252025-04-30T06:26:42.791ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 25 18 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 %............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60164 req_id = e3277f01-4b02-4c19-9f97-1e11d6a6d283
84262025-04-30T06:26:42.791ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9496,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60164 req_id = e3277f01-4b02-4c19-9f97-1e11d6a6d283
84272025-04-30T06:26:42.791ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
84282025-04-30T06:26:42.791ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84292025-04-30T06:26:42.791ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60164 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e3277f01-4b02-4c19-9f97-1e11d6a6d283
84302025-04-30T06:26:42.792ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3267, tv_nsec: 591749876 } })
84312025-04-30T06:26:42.793ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
84322025-04-30T06:26:42.793ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 00 93 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58624 req_id = 68ab331e-dc4e-43cc-83a5-d77fb10cf08d
84332025-04-30T06:26:42.794ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 147,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58624 req_id = 68ab331e-dc4e-43cc-83a5-d77fb10cf08d
84342025-04-30T06:26:42.795ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84352025-04-30T06:26:42.795ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58624 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 68ab331e-dc4e-43cc-83a5-d77fb10cf08d
84362025-04-30T06:26:42.882ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84372025-04-30T06:26:42.883ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
84382025-04-30T06:26:42.883ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84392025-04-30T06:26:42.885ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = e164f645-578d-477c-a9a7-f96b29375966 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84402025-04-30T06:26:42.885ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 34 f3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 4............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55717 req_id = 5b457670-20e7-43c6-a35b-a3a2488601b4
84412025-04-30T06:26:42.885ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13555,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55717 req_id = 5b457670-20e7-43c6-a35b-a3a2488601b4
84422025-04-30T06:26:42.885ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
84432025-04-30T06:26:42.885ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13555, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55717 req_id = 5b457670-20e7-43c6-a35b-a3a2488601b4
84442025-04-30T06:26:42.886ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = e164f645-578d-477c-a9a7-f96b29375966 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84452025-04-30T06:26:42.886ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = e164f645-578d-477c-a9a7-f96b29375966 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84462025-04-30T06:26:42.886ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = e164f645-578d-477c-a9a7-f96b29375966 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84472025-04-30T06:26:42.887ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
84482025-04-30T06:26:42.889ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
84492025-04-30T06:26:42.889ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
84502025-04-30T06:26:42.892ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 7253 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = e164f645-578d-477c-a9a7-f96b29375966 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84512025-04-30T06:26:42.893ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "e164f645-578d-477c-a9a7-f96b29375966", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84522025-04-30T06:26:42.893ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
84532025-04-30T06:26:42.895ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 19b11dcf-81b5-4c3c-a505-f9c4009bf71d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
84542025-04-30T06:26:42.895ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 19b11dcf-81b5-4c3c-a505-f9c4009bf71d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
84552025-04-30T06:26:42.895ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 19b11dcf-81b5-4c3c-a505-f9c4009bf71d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
84562025-04-30T06:26:42.903ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 19b11dcf-81b5-4c3c-a505-f9c4009bf71d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
84572025-04-30T06:26:42.904ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a05eaf15-6cf2-4dcc-b118-4e3448719bcb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84582025-04-30T06:26:42.905ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
84592025-04-30T06:26:42.909ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 14102 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 19b11dcf-81b5-4c3c-a505-f9c4009bf71d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
84602025-04-30T06:26:42.909ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "19b11dcf-81b5-4c3c-a505-f9c4009bf71d", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84612025-04-30T06:26:42.910ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
84622025-04-30T06:26:42.910ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84632025-04-30T06:26:42.910ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
84642025-04-30T06:26:42.912ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2c112f7-b0f5-41de-af63-89d3def43dc2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84652025-04-30T06:26:42.912ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
84662025-04-30T06:26:42.914ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "effa4e9a-96d7-4f35-ae6d-c9aabf19dad7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84672025-04-30T06:26:42.915ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
84682025-04-30T06:26:42.924ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
84692025-04-30T06:26:42.924ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2ec2531-7f3f-464e-9784-42c1de999def", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84702025-04-30T06:26:42.925ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
84712025-04-30T06:26:42.939ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01f5359c-0c34-4dda-80f6-86185196c877", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84722025-04-30T06:26:42.940ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
84732025-04-30T06:26:43.009ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84742025-04-30T06:26:43.009ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84752025-04-30T06:26:43.009ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84762025-04-30T06:26:43.009ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84772025-04-30T06:26:43.010ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84782025-04-30T06:26:43.011ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84792025-04-30T06:26:43.011ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84802025-04-30T06:26:43.039ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84812025-04-30T06:26:43.040ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84822025-04-30T06:26:43.040ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84832025-04-30T06:26:43.040ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84842025-04-30T06:26:43.041ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84852025-04-30T06:26:43.048ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84862025-04-30T06:26:43.048ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84872025-04-30T06:26:43.048ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84882025-04-30T06:26:43.052ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84892025-04-30T06:26:43.062ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84902025-04-30T06:26:43.063ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84912025-04-30T06:26:43.063ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84922025-04-30T06:26:43.064ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb47018d-6514-4664-97f9-2fc6b4181a49", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84932025-04-30T06:26:43.064ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
84942025-04-30T06:26:43.069ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f5155fc-d652-453f-aa0f-f52b01214520", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84952025-04-30T06:26:43.069ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
84962025-04-30T06:26:43.073ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b6b177b-0af4-4bad-8c82-f1cdad4d110f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84972025-04-30T06:26:43.074ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
84982025-04-30T06:26:43.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44c9db0c-dc92-43e7-9214-0223ea6f4fd7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
84992025-04-30T06:26:43.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
85002025-04-30T06:26:43.077ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af225244-39c5-483b-b28d-62af87173027", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85012025-04-30T06:26:43.077ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
85022025-04-30T06:26:43.079ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "290951ea-64d0-428a-adf7-08a1c5cb51c5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85032025-04-30T06:26:43.079ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
85042025-04-30T06:26:43.080ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4bb93e3-723e-4e01-b125-1a34b4c804bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85052025-04-30T06:26:43.081ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
85062025-04-30T06:26:43.082ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab9a7f7c-71a1-44ae-8a3a-170113457f76", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85072025-04-30T06:26:43.082ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
85082025-04-30T06:26:43.086ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8db6145-6f01-40e9-89d3-489c1673559f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85092025-04-30T06:26:43.086ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
85102025-04-30T06:26:43.090ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af4fa89f-5aa2-4385-b2c1-3a4d770b6844", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85112025-04-30T06:26:43.090ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
85122025-04-30T06:26:43.096ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
85132025-04-30T06:26:43.099ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae354d4-6972-4470-9219-e10c39165912", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85142025-04-30T06:26:43.099ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
85152025-04-30T06:26:43.101ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fda5fed-a939-476b-8267-e80d32e1f5f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85162025-04-30T06:26:43.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
85172025-04-30T06:26:43.104ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fb0eb58-e8e0-4353-8d04-4c6ea0fb99a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85182025-04-30T06:26:43.104ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
85192025-04-30T06:26:43.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04f40dde-8840-4cb3-becd-02225b1b6de1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85202025-04-30T06:26:43.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
85212025-04-30T06:26:43.109ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38ccafdf-b440-4dfd-984e-c7dcf784202f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85222025-04-30T06:26:43.109ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
85232025-04-30T06:26:43.111ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cf9d3b7-35ec-4f1f-bcc5-55668e77900a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85242025-04-30T06:26:43.111ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
85252025-04-30T06:26:43.113ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7beaf67-edda-4ce4-990f-af35241fbd4a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85262025-04-30T06:26:43.120ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
85272025-04-30T06:26:43.121ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4729a6c3-e194-4973-b8a7-3bca0228762e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85282025-04-30T06:26:43.122ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
85292025-04-30T06:26:43.123ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cd92f71-5610-4663-aaaa-bfe161b0cee6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85302025-04-30T06:26:43.124ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
85312025-04-30T06:26:43.126ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34dc4dd2-f8ec-4fd2-9d76-70c4fbcc0455", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85322025-04-30T06:26:43.126ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
85332025-04-30T06:26:43.129ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df0b823b-1337-4468-b4f9-4930a23df8be", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85342025-04-30T06:26:43.129ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
85352025-04-30T06:26:43.132ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "656927f9-8a55-433c-8800-30861b420c26", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85362025-04-30T06:26:43.132ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
85372025-04-30T06:26:43.134ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b72a577-b24f-4e3b-9cc0-69f44cf9693a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85382025-04-30T06:26:43.135ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
85392025-04-30T06:26:43.137ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da418a6c-a0ca-4477-9b2d-c746bea6025b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85402025-04-30T06:26:43.137ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
85412025-04-30T06:26:43.140ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "664d6fdd-5067-43e2-a09c-2c3010b32b14", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85422025-04-30T06:26:43.140ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
85432025-04-30T06:26:43.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9fa7a29-afe2-471f-85a8-14098d1c1d42", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85442025-04-30T06:26:43.141ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
85452025-04-30T06:26:43.142ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec73fc6-e811-4070-b1c4-ea5028487c19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85462025-04-30T06:26:43.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
85472025-04-30T06:26:43.144ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9229772f-ee46-4d44-a4c0-a6e7b7a31916", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85482025-04-30T06:26:43.144ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
85492025-04-30T06:26:43.145ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d9f0c9-daf1-4cb2-a1b0-8f9b0a2b42ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85502025-04-30T06:26:43.145ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
85512025-04-30T06:26:43.147ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85522025-04-30T06:26:43.147ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85532025-04-30T06:26:43.147ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85542025-04-30T06:26:43.147ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
85552025-04-30T06:26:43.148ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85562025-04-30T06:26:43.154ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e887ce-c719-4b1d-bcd3-f8c22f6aca4a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85572025-04-30T06:26:43.154ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
85582025-04-30T06:26:43.156ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85592025-04-30T06:26:43.164ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85602025-04-30T06:26:43.164ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85612025-04-30T06:26:43.165ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
85622025-04-30T06:26:43.165ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85632025-04-30T06:26:43.168ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85642025-04-30T06:26:43.168ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
85652025-04-30T06:26:43.168ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85662025-04-30T06:26:43.170ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aea90cdf-4fff-461d-989b-feb1aade04b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85672025-04-30T06:26:43.170ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
85682025-04-30T06:26:43.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85692025-04-30T06:26:43.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85702025-04-30T06:26:43.176ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
85712025-04-30T06:26:43.177ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
85722025-04-30T06:26:43.181ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef4aa591-2944-4ff8-8233-d0bf1f9dabc4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85732025-04-30T06:26:43.181ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
85742025-04-30T06:26:43.183ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
85752025-04-30T06:26:43.183ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
85762025-04-30T06:26:43.186ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a892db9-1075-4b73-af70-9480303ed43f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85772025-04-30T06:26:43.186ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
85782025-04-30T06:26:43.187ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
85792025-04-30T06:26:43.187ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85802025-04-30T06:26:43.191ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b699634-f97c-481b-ad05-1f4681616ef5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85812025-04-30T06:26:43.191ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
85822025-04-30T06:26:43.194ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4573e33f-f06f-4f2b-96b1-32784a27ee15", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85832025-04-30T06:26:43.194ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
85842025-04-30T06:26:43.198ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f226f41d-3f03-4562-b001-a728c988fdc3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85852025-04-30T06:26:43.198ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
85862025-04-30T06:26:43.202ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9838916-5c7b-41ed-b8d7-ce78b26f5b22", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85872025-04-30T06:26:43.203ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
85882025-04-30T06:26:43.205ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec0f5572-5776-4f54-8eeb-0586bca93b4e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85892025-04-30T06:26:43.205ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
85902025-04-30T06:26:43.210ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
85912025-04-30T06:26:43.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3439f244-6f70-49df-8263-16e5b1811d0d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85922025-04-30T06:26:43.212ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
85932025-04-30T06:26:43.213ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2553d4f0-0129-4638-8617-da91245cceb1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
85942025-04-30T06:26:43.213ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85952025-04-30T06:26:43.213ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85962025-04-30T06:26:43.214ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85972025-04-30T06:26:43.214ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85982025-04-30T06:26:43.214ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85992025-04-30T06:26:43.217ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86002025-04-30T06:26:43.222ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86012025-04-30T06:26:43.222ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86022025-04-30T06:26:43.222ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86032025-04-30T06:26:43.222ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86042025-04-30T06:26:43.222ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86052025-04-30T06:26:43.223ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86062025-04-30T06:26:43.225ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86072025-04-30T06:26:43.231ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86082025-04-30T06:26:43.231ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86092025-04-30T06:26:43.231ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86102025-04-30T06:26:43.231ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86112025-04-30T06:26:43.282ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
86122025-04-30T06:26:43.282ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
86132025-04-30T06:26:43.282ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86142025-04-30T06:26:43.282ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86152025-04-30T06:26:43.283ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86162025-04-30T06:26:43.283ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86172025-04-30T06:26:43.284ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86182025-04-30T06:26:43.284ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:37939 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
86192025-04-30T06:26:43.284ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
86202025-04-30T06:26:43.285ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:37939 req_id = 32d88abd-37f5-4067-ba92-23063fdc9068 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86212025-04-30T06:26:43.286ZINFOtest_disk_metrics (dropshot): request completed latency_us = 595 local_addr = [::1]:50989 method = GET remote_addr = [::1]:37939 req_id = 32d88abd-37f5-4067-ba92-23063fdc9068 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86222025-04-30T06:26:43.287ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "32d88abd-37f5-4067-ba92-23063fdc9068", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
86232025-04-30T06:26:43.289ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86242025-04-30T06:26:43.290ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86252025-04-30T06:26:43.290ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
86262025-04-30T06:26:43.293ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86272025-04-30T06:26:43.293ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86282025-04-30T06:26:43.293ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86292025-04-30T06:26:43.293ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86302025-04-30T06:26:43.294ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86312025-04-30T06:26:43.295ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86322025-04-30T06:26:43.300ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86332025-04-30T06:26:43.300ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86342025-04-30T06:26:43.300ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86352025-04-30T06:26:43.300ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86362025-04-30T06:26:43.305ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86372025-04-30T06:26:43.309ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86382025-04-30T06:26:43.309ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86392025-04-30T06:26:43.310ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86402025-04-30T06:26:43.312ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86412025-04-30T06:26:43.314ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86422025-04-30T06:26:43.314ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86432025-04-30T06:26:43.315ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86442025-04-30T06:26:43.350ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86452025-04-30T06:26:43.351ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86462025-04-30T06:26:43.351ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 585.276685ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86472025-04-30T06:26:43.352ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
86482025-04-30T06:26:43.352ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86492025-04-30T06:26:43.352ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86502025-04-30T06:26:43.352ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86512025-04-30T06:26:43.352ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86522025-04-30T06:26:43.352ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86532025-04-30T06:26:43.366ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86542025-04-30T06:26:43.372ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86552025-04-30T06:26:43.372ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86562025-04-30T06:26:43.372ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86572025-04-30T06:26:43.373ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86582025-04-30T06:26:43.374ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
86592025-04-30T06:26:43.387ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
86602025-04-30T06:26:43.387ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
86612025-04-30T06:26:43.388ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9d 13 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55965 req_id = 779a7629-45f4-45ad-a57e-96dad4993b05
86622025-04-30T06:26:43.388ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40211,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55965 req_id = 779a7629-45f4-45ad-a57e-96dad4993b05
86632025-04-30T06:26:43.388ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
86642025-04-30T06:26:43.389ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86652025-04-30T06:26:43.389ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55965 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 779a7629-45f4-45ad-a57e-96dad4993b05
86662025-04-30T06:26:43.389ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3268, tv_nsec: 188877183 } })
86672025-04-30T06:26:43.390ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
86682025-04-30T06:26:43.391ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 98 37 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .7..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60996 req_id = a512d6fe-3f3f-4446-80d8-7632150db76f
86692025-04-30T06:26:43.391ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38967,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60996 req_id = a512d6fe-3f3f-4446-80d8-7632150db76f
86702025-04-30T06:26:43.392ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86712025-04-30T06:26:43.392ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60996 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a512d6fe-3f3f-4446-80d8-7632150db76f
86722025-04-30T06:26:43.457ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
86732025-04-30T06:26:43.457ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86742025-04-30T06:26:43.458ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86752025-04-30T06:26:43.458ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86762025-04-30T06:26:43.458ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86772025-04-30T06:26:43.468ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86782025-04-30T06:26:43.469ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bf 6b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .k..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55065 req_id = 6cc9c909-713c-44fc-a190-54c7cf8aa693
86792025-04-30T06:26:43.470ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49003,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55065 req_id = 6cc9c909-713c-44fc-a190-54c7cf8aa693
86802025-04-30T06:26:43.470ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
86812025-04-30T06:26:43.470ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49003, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55065 req_id = 6cc9c909-713c-44fc-a190-54c7cf8aa693
86822025-04-30T06:26:43.472ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
86832025-04-30T06:26:43.472ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
86842025-04-30T06:26:43.472ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
86852025-04-30T06:26:43.473ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86862025-04-30T06:26:43.481ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86872025-04-30T06:26:43.481ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86882025-04-30T06:26:43.482ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86892025-04-30T06:26:43.483ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86902025-04-30T06:26:43.486ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86912025-04-30T06:26:43.486ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86922025-04-30T06:26:43.489ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86932025-04-30T06:26:43.489ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa47f64e-6f09-4583-a432-09fb86ae20be", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
86942025-04-30T06:26:43.489ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
86952025-04-30T06:26:43.494ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad93bdf6-fb2e-42f4-8b06-eeacfba0248c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
86962025-04-30T06:26:43.494ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
86972025-04-30T06:26:43.500ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d9ee31-237c-4d09-9f37-1da30e72ac4d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
86982025-04-30T06:26:43.500ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
86992025-04-30T06:26:43.507ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f35080d3-010f-4fb7-ac8a-e7d7df1d7564", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87002025-04-30T06:26:43.508ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
87012025-04-30T06:26:43.511ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e79516a3-993b-499d-a970-2fa79b449c0a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87022025-04-30T06:26:43.511ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
87032025-04-30T06:26:43.528ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
87042025-04-30T06:26:43.529ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d57c9fce-de93-4fd0-9150-aa592ab29011", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87052025-04-30T06:26:43.529ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
87062025-04-30T06:26:43.532ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11d6bf16-4fed-49d7-80bb-8cb52cab25e1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87072025-04-30T06:26:43.532ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
87082025-04-30T06:26:43.536ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73a21b69-4ac9-4cb5-83e6-b81354de5848", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87092025-04-30T06:26:43.536ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
87102025-04-30T06:26:43.537ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc6ba47c-9726-46de-ae03-8c53a7096513", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87112025-04-30T06:26:43.537ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
87122025-04-30T06:26:43.538ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c7d441d-e9dc-4afd-8e2d-25353d402420", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87132025-04-30T06:26:43.538ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
87142025-04-30T06:26:43.540ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "645c56af-2914-4d8d-a979-56b9d8803973", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87152025-04-30T06:26:43.540ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
87162025-04-30T06:26:43.541ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e40e29f-8907-4e46-9375-ce9b2d19629f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87172025-04-30T06:26:43.541ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
87182025-04-30T06:26:43.543ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "181486d5-08e9-47d0-a06b-5051704d33ec", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87192025-04-30T06:26:43.544ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
87202025-04-30T06:26:43.546ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7283e03c-48c4-4411-a37c-af743f53eeea", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87212025-04-30T06:26:43.546ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
87222025-04-30T06:26:43.548ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d0aa7b8-decb-4fd1-9573-b053453ca5f0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87232025-04-30T06:26:43.549ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
87242025-04-30T06:26:43.571ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b59dd138-3302-4168-837b-022623290120", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87252025-04-30T06:26:43.571ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
87262025-04-30T06:26:43.574ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "180da784-1b84-4c3d-9aad-c73a29498903", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87272025-04-30T06:26:43.574ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
87282025-04-30T06:26:43.579ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87292025-04-30T06:26:43.579ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87302025-04-30T06:26:43.579ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87312025-04-30T06:26:43.579ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87322025-04-30T06:26:43.581ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87332025-04-30T06:26:43.584ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "032d7ff4-c81b-4d85-b97d-d6023efe14a0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87342025-04-30T06:26:43.584ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
87352025-04-30T06:26:43.585ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87362025-04-30T06:26:43.592ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87372025-04-30T06:26:43.592ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87382025-04-30T06:26:43.592ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87392025-04-30T06:26:43.592ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87402025-04-30T06:26:43.592ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87412025-04-30T06:26:43.592ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87422025-04-30T06:26:43.593ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87432025-04-30T06:26:43.596ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6010587-a59a-4981-b8d1-5feffe28d6c8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87442025-04-30T06:26:43.596ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
87452025-04-30T06:26:43.601ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37368859-8059-4c02-875a-3349c92a63ca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87462025-04-30T06:26:43.601ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
87472025-04-30T06:26:43.602ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4791236d-7187-4765-84e6-f4ae7aae0d81", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87482025-04-30T06:26:43.603ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
87492025-04-30T06:26:43.605ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eaf9933-1752-4501-a8ba-a4a7c2752d95", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87502025-04-30T06:26:43.605ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
87512025-04-30T06:26:43.607ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44142c73-7e99-45cc-85bc-5f305b8fbd80", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87522025-04-30T06:26:43.607ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
87532025-04-30T06:26:43.610ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ec90e39-e448-40ed-924e-c37a353d82c1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87542025-04-30T06:26:43.610ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
87552025-04-30T06:26:43.616ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1592432c-9832-40ba-ac42-c42f59249529", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87562025-04-30T06:26:43.616ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
87572025-04-30T06:26:43.624ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfbbc2f6-c244-4018-b4b7-4855de8f795e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87582025-04-30T06:26:43.624ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
87592025-04-30T06:26:43.628ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
87602025-04-30T06:26:43.632ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cf204f5-8ca4-4aa7-9db5-06c6d5026f0a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87612025-04-30T06:26:43.632ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
87622025-04-30T06:26:43.636ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68fc630f-5995-403b-b2e5-003bc5d1e325", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87632025-04-30T06:26:43.636ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
87642025-04-30T06:26:43.642ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45fe44ae-d563-45c2-8f36-70fd6406cd32", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87652025-04-30T06:26:43.642ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
87662025-04-30T06:26:43.648ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffdd6bd7-0ee5-4f7f-89b2-ff173f1b32e5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87672025-04-30T06:26:43.648ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
87682025-04-30T06:26:43.650ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3f0611c-ef23-4bf0-a8ed-61a61a612714", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87692025-04-30T06:26:43.650ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
87702025-04-30T06:26:43.652ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b165bf8-b10a-4784-9f03-51de60af58a3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87712025-04-30T06:26:43.653ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
87722025-04-30T06:26:43.654ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17ee23cc-e5eb-414a-b293-ab767d3b9a89", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87732025-04-30T06:26:43.655ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
87742025-04-30T06:26:43.656ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a7d027-e226-423d-88dd-aaf9a5a51094", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87752025-04-30T06:26:43.656ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
87762025-04-30T06:26:43.659ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a97af0c1-40d4-4a03-a7fa-4bffd4c6fafc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87772025-04-30T06:26:43.659ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
87782025-04-30T06:26:43.663ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b1a42b3-f728-4915-a420-0a3d088b0bc3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87792025-04-30T06:26:43.663ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
87802025-04-30T06:26:43.667ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a60d441b-0061-41e0-bee7-53a665f5209f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87812025-04-30T06:26:43.667ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
87822025-04-30T06:26:43.672ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bccf134-445a-4a74-9526-76066c56ed5a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87832025-04-30T06:26:43.672ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
87842025-04-30T06:26:43.677ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b1b1437-efe7-4834-abdd-26c7d3039ff3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87852025-04-30T06:26:43.678ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
87862025-04-30T06:26:43.679ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87872025-04-30T06:26:43.679ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87882025-04-30T06:26:43.679ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87892025-04-30T06:26:43.679ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87902025-04-30T06:26:43.681ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87912025-04-30T06:26:43.683ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c632aec-b367-4fa3-a062-9c6c7895dbf6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
87922025-04-30T06:26:43.684ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
87932025-04-30T06:26:43.685ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87942025-04-30T06:26:43.697ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87952025-04-30T06:26:43.697ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87962025-04-30T06:26:43.697ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87972025-04-30T06:26:43.698ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87982025-04-30T06:26:43.708ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87992025-04-30T06:26:43.708ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
88002025-04-30T06:26:43.710ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88012025-04-30T06:26:43.710ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "179b3591-1065-4e64-bcc8-18f9a2613dc4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
88022025-04-30T06:26:43.710ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
88032025-04-30T06:26:43.712ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb899b0a-a3d0-43d4-8ae3-a6f9103ae61a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
88042025-04-30T06:26:43.712ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
88052025-04-30T06:26:43.713ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "230ef7b4-5e94-43bf-ae77-a3c934a919f5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
88062025-04-30T06:26:43.714ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
88072025-04-30T06:26:43.716ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71229fcc-0729-406e-9496-7ba7e6f26a3a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
88082025-04-30T06:26:43.717ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
88092025-04-30T06:26:43.719ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f234e75-6839-4ae6-b4bd-a971f8c6932c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
88102025-04-30T06:26:43.719ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88112025-04-30T06:26:43.719ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88122025-04-30T06:26:43.719ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88132025-04-30T06:26:43.719ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88142025-04-30T06:26:43.720ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88152025-04-30T06:26:43.730ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88162025-04-30T06:26:43.735ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88172025-04-30T06:26:43.737ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88182025-04-30T06:26:43.737ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88192025-04-30T06:26:43.737ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88202025-04-30T06:26:43.737ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88212025-04-30T06:26:43.740ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88222025-04-30T06:26:43.743ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88232025-04-30T06:26:43.754ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88242025-04-30T06:26:43.754ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88252025-04-30T06:26:43.754ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88262025-04-30T06:26:43.754ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88272025-04-30T06:26:43.825ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
88282025-04-30T06:26:43.825ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
88292025-04-30T06:26:43.827ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:44891 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
88302025-04-30T06:26:43.830ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:44891 req_id = 5775fdd4-1c18-48e6-aee5-371ad41e434f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88312025-04-30T06:26:43.831ZINFOtest_disk_metrics (dropshot): request completed latency_us = 646 local_addr = [::1]:50989 method = GET remote_addr = [::1]:44891 req_id = 5775fdd4-1c18-48e6-aee5-371ad41e434f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88322025-04-30T06:26:43.831ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5775fdd4-1c18-48e6-aee5-371ad41e434f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
88332025-04-30T06:26:43.835ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88342025-04-30T06:26:43.835ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88352025-04-30T06:26:43.835ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
88362025-04-30T06:26:43.837ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88372025-04-30T06:26:43.838ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88382025-04-30T06:26:43.838ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88392025-04-30T06:26:43.838ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88402025-04-30T06:26:43.839ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88412025-04-30T06:26:43.842ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88422025-04-30T06:26:43.857ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88432025-04-30T06:26:43.858ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88442025-04-30T06:26:43.859ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88452025-04-30T06:26:43.859ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88462025-04-30T06:26:43.865ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
88472025-04-30T06:26:43.892ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
88482025-04-30T06:26:43.892ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
88492025-04-30T06:26:43.892ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 541.2023ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
88502025-04-30T06:26:43.892ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
88512025-04-30T06:26:43.892ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88522025-04-30T06:26:43.892ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88532025-04-30T06:26:43.892ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88542025-04-30T06:26:43.892ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88552025-04-30T06:26:43.893ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88562025-04-30T06:26:43.896ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88572025-04-30T06:26:43.901ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88582025-04-30T06:26:43.901ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88592025-04-30T06:26:43.901ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88602025-04-30T06:26:43.902ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88612025-04-30T06:26:43.906ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
88622025-04-30T06:26:43.906ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
88632025-04-30T06:26:43.907ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 74 12 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 t............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54030 req_id = 416d5db8-40e8-4088-9380-571cd9f16aa0
88642025-04-30T06:26:43.907ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29714,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54030 req_id = 416d5db8-40e8-4088-9380-571cd9f16aa0
88652025-04-30T06:26:43.907ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
88662025-04-30T06:26:43.909ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88672025-04-30T06:26:43.909ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54030 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 416d5db8-40e8-4088-9380-571cd9f16aa0
88682025-04-30T06:26:43.909ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3268, tv_nsec: 708099629 } })
88692025-04-30T06:26:43.909ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
88702025-04-30T06:26:43.910ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 88 f6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53190 req_id = 32e90e9a-b1d5-4ed5-818f-eb3d92be5978
88712025-04-30T06:26:43.910ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35062,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53190 req_id = 32e90e9a-b1d5-4ed5-818f-eb3d92be5978
88722025-04-30T06:26:43.910ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88732025-04-30T06:26:43.910ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53190 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 32e90e9a-b1d5-4ed5-818f-eb3d92be5978
88742025-04-30T06:26:43.958ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
88752025-04-30T06:26:43.958ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88762025-04-30T06:26:43.958ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88772025-04-30T06:26:43.959ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88782025-04-30T06:26:43.959ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
88792025-04-30T06:26:43.959ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88802025-04-30T06:26:43.962ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: fc 49 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .I..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63555 req_id = 1f47da06-a356-472d-9699-6e6fd99b02f5
88812025-04-30T06:26:43.962ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64585,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63555 req_id = 1f47da06-a356-472d-9699-6e6fd99b02f5
88822025-04-30T06:26:43.962ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
88832025-04-30T06:26:43.962ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64585, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63555 req_id = 1f47da06-a356-472d-9699-6e6fd99b02f5
88842025-04-30T06:26:43.963ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
88852025-04-30T06:26:43.963ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
88862025-04-30T06:26:43.963ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
88872025-04-30T06:26:43.966ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88882025-04-30T06:26:43.973ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88892025-04-30T06:26:43.973ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88902025-04-30T06:26:43.973ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
88912025-04-30T06:26:43.973ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88922025-04-30T06:26:43.976ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88932025-04-30T06:26:43.976ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
88942025-04-30T06:26:43.978ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88952025-04-30T06:26:43.987ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88962025-04-30T06:26:43.987ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88972025-04-30T06:26:43.992ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88982025-04-30T06:26:43.993ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88992025-04-30T06:26:43.993ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89002025-04-30T06:26:43.993ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89012025-04-30T06:26:43.996ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c56f88ba-ce03-49c5-8201-a01654ed374e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89022025-04-30T06:26:43.997ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
89032025-04-30T06:26:44.011ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c228e07-9f64-43e2-bd45-2f0589ff8412", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89042025-04-30T06:26:44.011ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
89052025-04-30T06:26:44.021ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "323b2ee1-6cc3-4a15-82ba-ba925ee62ec6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89062025-04-30T06:26:44.021ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
89072025-04-30T06:26:44.046ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f931f81f-e10f-43e5-953e-a4e4b47009b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89082025-04-30T06:26:44.046ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
89092025-04-30T06:26:44.058ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad037918-d5fa-49c8-8891-c25ce35cc701", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89102025-04-30T06:26:44.059ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
89112025-04-30T06:26:44.062ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46540d0e-7917-44a7-aa81-059084b32287", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89122025-04-30T06:26:44.062ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
89132025-04-30T06:26:44.066ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a5f6ad4-4eb1-401e-9f71-bf43c86c08b6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89142025-04-30T06:26:44.066ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
89152025-04-30T06:26:44.067ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
89162025-04-30T06:26:44.071ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b21c42cc-3f27-4366-9743-4b848a66088d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89172025-04-30T06:26:44.072ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
89182025-04-30T06:26:44.081ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6ee0dd4-eab4-48fb-9223-8508343a6f5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89192025-04-30T06:26:44.082ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
89202025-04-30T06:26:44.084ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1de61898-fb39-4f9d-9c0a-7b7a286eb429", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89212025-04-30T06:26:44.085ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
89222025-04-30T06:26:44.086ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "00bfc51b-e875-4470-b3af-0611f8ad28ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89232025-04-30T06:26:44.086ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
89242025-04-30T06:26:44.088ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f07286d9-8098-4bd3-8e38-19bed1e9f0ef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89252025-04-30T06:26:44.088ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
89262025-04-30T06:26:44.090ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e862eb53-a06e-45fc-99d2-3d7b46f534a2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89272025-04-30T06:26:44.090ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
89282025-04-30T06:26:44.093ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6617366a-6ec0-45ba-bc6e-e6947ebcf70e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89292025-04-30T06:26:44.093ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
89302025-04-30T06:26:44.099ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41db1225-a7e3-46a0-844e-c19a6b5f1348", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89312025-04-30T06:26:44.099ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
89322025-04-30T06:26:44.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4ffc892-1a6d-41e6-9aa1-4a1b35e67f0e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89332025-04-30T06:26:44.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
89342025-04-30T06:26:44.109ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "288e232f-1bf1-400b-bc34-71463875481c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89352025-04-30T06:26:44.109ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
89362025-04-30T06:26:44.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc2e241-adee-47c6-b069-6a207975e683", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89372025-04-30T06:26:44.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
89382025-04-30T06:26:44.118ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89392025-04-30T06:26:44.118ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89402025-04-30T06:26:44.118ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89412025-04-30T06:26:44.119ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
89422025-04-30T06:26:44.124ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89432025-04-30T06:26:44.127ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eb8b00e-3246-4bb1-8862-9146e76c9ea8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89442025-04-30T06:26:44.127ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
89452025-04-30T06:26:44.128ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89462025-04-30T06:26:44.134ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89472025-04-30T06:26:44.135ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89482025-04-30T06:26:44.135ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
89492025-04-30T06:26:44.135ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89502025-04-30T06:26:44.144ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3abeac9-08fe-4e12-87e8-afcf46cdeb90", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89512025-04-30T06:26:44.144ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
89522025-04-30T06:26:44.145ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eee82571-1998-4d35-9cb5-4d778e977128", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89532025-04-30T06:26:44.146ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
89542025-04-30T06:26:44.146ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89552025-04-30T06:26:44.146ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
89562025-04-30T06:26:44.146ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89572025-04-30T06:26:44.147ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5447e67d-34ee-44de-86ef-f9c88e5edc00", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89582025-04-30T06:26:44.147ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
89592025-04-30T06:26:44.149ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1693c9f6-0f56-4295-ba8f-e2c8617af61d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89602025-04-30T06:26:44.149ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
89612025-04-30T06:26:44.154ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69900466-2c82-4bcf-82fd-d33211ccb01c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89622025-04-30T06:26:44.154ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
89632025-04-30T06:26:44.159ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "257bf757-95d1-4adf-831c-00af326003d7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89642025-04-30T06:26:44.159ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
89652025-04-30T06:26:44.162ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6db2120-a9b2-47e1-b384-bae86ecbe5de", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89662025-04-30T06:26:44.162ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
89672025-04-30T06:26:44.169ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5c24e69-3297-4c5e-aa7d-fdfad59e35cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89682025-04-30T06:26:44.169ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
89692025-04-30T06:26:44.171ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b62b4d13-425f-4e49-afbf-e3adcf4cb314", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89702025-04-30T06:26:44.171ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
89712025-04-30T06:26:44.176ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89722025-04-30T06:26:44.176ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89732025-04-30T06:26:44.177ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
89742025-04-30T06:26:44.178ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
89752025-04-30T06:26:44.178ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
89762025-04-30T06:26:44.179ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68484100-6608-4ce0-a3bc-0c5436009e5c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89772025-04-30T06:26:44.179ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
89782025-04-30T06:26:44.182ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
89792025-04-30T06:26:44.182ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
89802025-04-30T06:26:44.183ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2082d2d-04e5-478c-b124-56934cf56ceb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89812025-04-30T06:26:44.183ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
89822025-04-30T06:26:44.185ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1107af47-4807-4699-b6f3-36b19f451164", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89832025-04-30T06:26:44.186ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
89842025-04-30T06:26:44.187ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b5664b8-c6a1-4f4c-8de1-9824187affca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89852025-04-30T06:26:44.187ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
89862025-04-30T06:26:44.189ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0636d70c-ee0f-4afa-a1d0-9a09333e8d29", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89872025-04-30T06:26:44.190ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
89882025-04-30T06:26:44.190ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
89892025-04-30T06:26:44.190ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89902025-04-30T06:26:44.190ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c37bc6d7-8d48-47b3-a6d7-7bfc7d467a7e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89912025-04-30T06:26:44.190ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
89922025-04-30T06:26:44.193ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "281bdc18-8378-46c9-9978-55ff073891df", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89932025-04-30T06:26:44.193ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
89942025-04-30T06:26:44.196ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dae07f3-a740-4ae1-a05a-de3e3cff9e01", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89952025-04-30T06:26:44.196ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
89962025-04-30T06:26:44.199ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6a5cda7-a67b-4387-90be-a5d1bde97ae3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89972025-04-30T06:26:44.199ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
89982025-04-30T06:26:44.201ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c91e319-5897-4038-9979-6d05ab8e9b7c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
89992025-04-30T06:26:44.201ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
90002025-04-30T06:26:44.203ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec2646b-97e2-48e4-b599-a7fcd52c6e3f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90012025-04-30T06:26:44.204ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
90022025-04-30T06:26:44.207ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61cf3c9a-1db6-4b42-a605-966533a63990", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90032025-04-30T06:26:44.207ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
90042025-04-30T06:26:44.211ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e3ffa50-8008-4746-83e6-9fa00c621ab6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90052025-04-30T06:26:44.211ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
90062025-04-30T06:26:44.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c23cd25-a527-4eb6-b387-407ad752d7e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90072025-04-30T06:26:44.213ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
90082025-04-30T06:26:44.214ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a8eaa2f-d817-4a4a-8925-65d0016de0fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90092025-04-30T06:26:44.215ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
90102025-04-30T06:26:44.216ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "381606a8-7483-469a-be8e-e866e8770392", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90112025-04-30T06:26:44.216ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
90122025-04-30T06:26:44.218ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82164360-0eb2-4dd1-a5a0-bd213180e92a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90132025-04-30T06:26:44.218ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90142025-04-30T06:26:44.218ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90152025-04-30T06:26:44.218ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90162025-04-30T06:26:44.218ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90172025-04-30T06:26:44.219ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90182025-04-30T06:26:44.225ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90192025-04-30T06:26:44.237ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90202025-04-30T06:26:44.237ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90212025-04-30T06:26:44.237ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90222025-04-30T06:26:44.237ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90232025-04-30T06:26:44.237ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90242025-04-30T06:26:44.238ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90252025-04-30T06:26:44.239ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90262025-04-30T06:26:44.239ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90272025-04-30T06:26:44.239ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90282025-04-30T06:26:44.239ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90292025-04-30T06:26:44.244ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90302025-04-30T06:26:44.244ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90312025-04-30T06:26:44.256ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90322025-04-30T06:26:44.256ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90332025-04-30T06:26:44.256ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90342025-04-30T06:26:44.259ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90352025-04-30T06:26:44.259ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90362025-04-30T06:26:44.278ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90372025-04-30T06:26:44.278ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90382025-04-30T06:26:44.278ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90392025-04-30T06:26:44.284ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90402025-04-30T06:26:44.369ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
90412025-04-30T06:26:44.369ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
90422025-04-30T06:26:44.370ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:36130 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90432025-04-30T06:26:44.371ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
90442025-04-30T06:26:44.372ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90452025-04-30T06:26:44.372ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90462025-04-30T06:26:44.377ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90472025-04-30T06:26:44.377ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:36130 req_id = 688e1c01-0267-4123-8c1a-50a2283220ca sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90482025-04-30T06:26:44.378ZINFOtest_disk_metrics (dropshot): request completed latency_us = 764 local_addr = [::1]:50989 method = GET remote_addr = [::1]:36130 req_id = 688e1c01-0267-4123-8c1a-50a2283220ca response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90492025-04-30T06:26:44.379ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "688e1c01-0267-4123-8c1a-50a2283220ca", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
90502025-04-30T06:26:44.381ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90512025-04-30T06:26:44.381ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90522025-04-30T06:26:44.381ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
90532025-04-30T06:26:44.383ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90542025-04-30T06:26:44.384ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90552025-04-30T06:26:44.384ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90562025-04-30T06:26:44.384ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90572025-04-30T06:26:44.385ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90582025-04-30T06:26:44.388ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90592025-04-30T06:26:44.394ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90602025-04-30T06:26:44.395ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90612025-04-30T06:26:44.395ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90622025-04-30T06:26:44.396ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90632025-04-30T06:26:44.449ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
90642025-04-30T06:26:44.450ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
90652025-04-30T06:26:44.450ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 557.348882ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
90662025-04-30T06:26:44.450ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
90672025-04-30T06:26:44.450ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90682025-04-30T06:26:44.450ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90692025-04-30T06:26:44.450ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90702025-04-30T06:26:44.450ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90712025-04-30T06:26:44.451ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90722025-04-30T06:26:44.455ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
90732025-04-30T06:26:44.456ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90742025-04-30T06:26:44.466ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90752025-04-30T06:26:44.466ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90762025-04-30T06:26:44.466ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90772025-04-30T06:26:44.466ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90782025-04-30T06:26:44.470ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
90792025-04-30T06:26:44.470ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
90802025-04-30T06:26:44.470ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f3 1d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57967 req_id = 749b5496-c660-4bbe-8efe-2cc057c2e9ae
90812025-04-30T06:26:44.470ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62237,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57967 req_id = 749b5496-c660-4bbe-8efe-2cc057c2e9ae
90822025-04-30T06:26:44.471ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
90832025-04-30T06:26:44.471ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90842025-04-30T06:26:44.471ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57967 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 749b5496-c660-4bbe-8efe-2cc057c2e9ae
90852025-04-30T06:26:44.471ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3269, tv_nsec: 270552911 } })
90862025-04-30T06:26:44.471ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
90872025-04-30T06:26:44.472ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8c 6f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .o..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59693 req_id = bace093f-19c4-4fd6-98b1-677fc91d7ab7
90882025-04-30T06:26:44.472ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35951,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59693 req_id = bace093f-19c4-4fd6-98b1-677fc91d7ab7
90892025-04-30T06:26:44.472ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90902025-04-30T06:26:44.472ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59693 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bace093f-19c4-4fd6-98b1-677fc91d7ab7
90912025-04-30T06:26:44.560ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
90922025-04-30T06:26:44.560ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90932025-04-30T06:26:44.561ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90942025-04-30T06:26:44.561ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90952025-04-30T06:26:44.561ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90962025-04-30T06:26:44.562ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90972025-04-30T06:26:44.562ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d9 50 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .P..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63697 req_id = 3b528ae3-684a-49dd-9b73-12d7ba9eee35
90982025-04-30T06:26:44.562ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55632,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63697 req_id = 3b528ae3-684a-49dd-9b73-12d7ba9eee35
90992025-04-30T06:26:44.562ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
91002025-04-30T06:26:44.562ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55632, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63697 req_id = 3b528ae3-684a-49dd-9b73-12d7ba9eee35
91012025-04-30T06:26:44.563ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
91022025-04-30T06:26:44.563ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
91032025-04-30T06:26:44.563ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
91042025-04-30T06:26:44.564ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91052025-04-30T06:26:44.571ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91062025-04-30T06:26:44.571ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91072025-04-30T06:26:44.571ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91082025-04-30T06:26:44.572ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91092025-04-30T06:26:44.572ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b38c7eb2-4fe5-4a11-b3b6-6a6badc7bf58", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91102025-04-30T06:26:44.572ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
91112025-04-30T06:26:44.574ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb74a805-29c5-4578-82ce-eda98afe7639", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91122025-04-30T06:26:44.575ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
91132025-04-30T06:26:44.575ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91142025-04-30T06:26:44.575ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91152025-04-30T06:26:44.576ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91162025-04-30T06:26:44.578ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "840c31cb-023c-48ae-9583-87a088fd03a5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91172025-04-30T06:26:44.578ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
91182025-04-30T06:26:44.590ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d87b3fa1-2414-421b-bbac-45e7e324b035", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91192025-04-30T06:26:44.593ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
91202025-04-30T06:26:44.597ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e622850-9351-4d4e-ad41-3ec48096e0d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91212025-04-30T06:26:44.597ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
91222025-04-30T06:26:44.602ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10a66940-2c56-40a2-8ce9-125d8a7e704a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91232025-04-30T06:26:44.602ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
91242025-04-30T06:26:44.608ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28d51cc5-25b3-4175-9d27-3222cec441e6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91252025-04-30T06:26:44.610ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
91262025-04-30T06:26:44.615ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e6b044-5252-4d93-b3f5-835ddb135bb2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91272025-04-30T06:26:44.616ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
91282025-04-30T06:26:44.631ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68b94ec4-02b2-46fb-861e-790f2e09fceb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91292025-04-30T06:26:44.631ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
91302025-04-30T06:26:44.632ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
91312025-04-30T06:26:44.635ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cac9c64f-eca8-4462-97e4-50ad550cebc0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91322025-04-30T06:26:44.636ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
91332025-04-30T06:26:44.638ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "971fe95b-189f-480f-acde-f5405271773b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91342025-04-30T06:26:44.638ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
91352025-04-30T06:26:44.640ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a56a3811-01ec-4145-91e6-16abb97ce4db", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91362025-04-30T06:26:44.640ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
91372025-04-30T06:26:44.642ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ddac8da-6f04-4ac3-8e92-612ffb5d6e60", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91382025-04-30T06:26:44.643ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
91392025-04-30T06:26:44.645ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8115cd5e-5175-4c20-8d3b-6b521d6c7d1c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91402025-04-30T06:26:44.645ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
91412025-04-30T06:26:44.648ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "72a34b43-ecfb-4ef1-b4fa-4c41a851a323", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91422025-04-30T06:26:44.648ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
91432025-04-30T06:26:44.655ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f333f04b-8454-47a9-9c33-b7ca6b4e2c29", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91442025-04-30T06:26:44.655ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
91452025-04-30T06:26:44.667ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e630304a-c77a-4ccd-85ba-f3eb84fc71b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91462025-04-30T06:26:44.667ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
91472025-04-30T06:26:44.670ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f0b0ac-2787-41e0-8234-016a872fe400", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91482025-04-30T06:26:44.670ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
91492025-04-30T06:26:44.672ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f774bea-f7de-4df5-a88b-3b3d02989604", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91502025-04-30T06:26:44.672ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
91512025-04-30T06:26:44.673ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c18cb6f0-1c26-408b-bbbb-a3f9687c4125", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91522025-04-30T06:26:44.674ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
91532025-04-30T06:26:44.675ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aa1af7a-9562-4c50-a84c-7bd22ce47ae1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91542025-04-30T06:26:44.675ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
91552025-04-30T06:26:44.678ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5041b4cc-7e90-41d9-9b01-1dda58b5a298", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91562025-04-30T06:26:44.679ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
91572025-04-30T06:26:44.680ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bad5b3f4-f53f-435a-943a-1ec42b668b05", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91582025-04-30T06:26:44.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
91592025-04-30T06:26:44.683ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91602025-04-30T06:26:44.683ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91612025-04-30T06:26:44.683ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91622025-04-30T06:26:44.683ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91632025-04-30T06:26:44.686ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91642025-04-30T06:26:44.687ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b53d37b-9e35-4e9c-8439-527f15b07f9a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91652025-04-30T06:26:44.687ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
91662025-04-30T06:26:44.690ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91672025-04-30T06:26:44.704ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91682025-04-30T06:26:44.704ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91692025-04-30T06:26:44.704ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91702025-04-30T06:26:44.705ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91712025-04-30T06:26:44.708ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aec0132-7721-40e8-8447-5a5f4e3d1c96", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91722025-04-30T06:26:44.708ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
91732025-04-30T06:26:44.709ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91742025-04-30T06:26:44.709ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91752025-04-30T06:26:44.711ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91762025-04-30T06:26:44.718ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a85a26c5-a268-4973-8488-a73990daefc3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91772025-04-30T06:26:44.718ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
91782025-04-30T06:26:44.731ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b51e3c9-34cf-470d-ad82-14001e69465c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91792025-04-30T06:26:44.731ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
91802025-04-30T06:26:44.740ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
91812025-04-30T06:26:44.741ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "793412c1-19c8-41c6-8412-200390f4a7ac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91822025-04-30T06:26:44.741ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
91832025-04-30T06:26:44.743ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d87660d-271a-4f08-b13c-c1b9b8f7f2fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91842025-04-30T06:26:44.744ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
91852025-04-30T06:26:44.746ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8e0552e-9480-4ce3-8d3c-f4829ef9fb93", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91862025-04-30T06:26:44.746ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
91872025-04-30T06:26:44.748ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af424dae-35ba-4a29-8243-fe54afaa1ee2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91882025-04-30T06:26:44.748ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
91892025-04-30T06:26:44.749ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "305a465a-ee60-4ccf-ae65-e552f9e6b168", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91902025-04-30T06:26:44.750ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
91912025-04-30T06:26:44.750ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33d714c9-6caf-4165-b9bf-5b35bf37c458", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91922025-04-30T06:26:44.751ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
91932025-04-30T06:26:44.753ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73a5817c-0116-4226-89e2-686e947d0796", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91942025-04-30T06:26:44.754ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
91952025-04-30T06:26:44.755ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c38b538b-8df7-474d-827c-01b1120d6ff0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91962025-04-30T06:26:44.756ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
91972025-04-30T06:26:44.760ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95aeb642-f81a-40d2-8c4a-b05ae3ce6bd6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
91982025-04-30T06:26:44.760ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
91992025-04-30T06:26:44.764ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca2c114-6377-4a20-9f3b-5cde1cb64529", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92002025-04-30T06:26:44.764ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
92012025-04-30T06:26:44.767ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32715b96-6b44-4ef6-9895-51ea0ec98d41", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92022025-04-30T06:26:44.767ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
92032025-04-30T06:26:44.769ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5151c7-5bfa-43ad-9137-d2434af7bf3e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92042025-04-30T06:26:44.769ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
92052025-04-30T06:26:44.771ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d1ed9ad-a217-4b91-a5ad-d75932e47024", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92062025-04-30T06:26:44.771ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
92072025-04-30T06:26:44.773ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "240360fb-625e-4153-80f9-1829a873d51b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92082025-04-30T06:26:44.773ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
92092025-04-30T06:26:44.775ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e1dbfb7-2ea9-4967-b380-82fd8bb466dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92102025-04-30T06:26:44.776ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
92112025-04-30T06:26:44.777ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b945072b-b5de-4e48-9ed8-0ad3f47b831b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92122025-04-30T06:26:44.777ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
92132025-04-30T06:26:44.779ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60e30f6d-ffcb-42c7-a1e4-68cd97276c4a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92142025-04-30T06:26:44.779ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
92152025-04-30T06:26:44.780ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd32b06f-d065-4f3f-a3d2-1cafeb996575", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92162025-04-30T06:26:44.780ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92172025-04-30T06:26:44.781ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92182025-04-30T06:26:44.781ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92192025-04-30T06:26:44.781ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92202025-04-30T06:26:44.782ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92212025-04-30T06:26:44.784ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92222025-04-30T06:26:44.791ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92232025-04-30T06:26:44.791ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92242025-04-30T06:26:44.791ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92252025-04-30T06:26:44.791ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92262025-04-30T06:26:44.791ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92272025-04-30T06:26:44.792ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92282025-04-30T06:26:44.793ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92292025-04-30T06:26:44.793ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92302025-04-30T06:26:44.793ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92312025-04-30T06:26:44.794ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
92322025-04-30T06:26:44.794ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92332025-04-30T06:26:44.796ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92342025-04-30T06:26:44.802ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92352025-04-30T06:26:44.802ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92362025-04-30T06:26:44.802ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92372025-04-30T06:26:44.803ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92382025-04-30T06:26:44.806ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92392025-04-30T06:26:44.816ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92402025-04-30T06:26:44.816ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92412025-04-30T06:26:44.816ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
92422025-04-30T06:26:44.817ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92432025-04-30T06:26:44.897ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
92442025-04-30T06:26:44.897ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
92452025-04-30T06:26:44.897ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:39259 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
92462025-04-30T06:26:44.898ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
92472025-04-30T06:26:44.899ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
92482025-04-30T06:26:44.900ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92492025-04-30T06:26:44.900ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
92502025-04-30T06:26:44.901ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92512025-04-30T06:26:44.901ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 25e2b4e8-d758-4956-9a66-e2c17a158224 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
92522025-04-30T06:26:44.901ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:39259 req_id = efe9a304-9f58-4918-a03a-c6ad97015e8a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92532025-04-30T06:26:44.902ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 25e2b4e8-d758-4956-9a66-e2c17a158224 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
92542025-04-30T06:26:44.902ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 25e2b4e8-d758-4956-9a66-e2c17a158224 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
92552025-04-30T06:26:44.902ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 25e2b4e8-d758-4956-9a66-e2c17a158224 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
92562025-04-30T06:26:44.903ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1507 local_addr = [::1]:50989 method = GET remote_addr = [::1]:39259 req_id = efe9a304-9f58-4918-a03a-c6ad97015e8a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92572025-04-30T06:26:44.903ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "efe9a304-9f58-4918-a03a-c6ad97015e8a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92582025-04-30T06:26:44.906ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92592025-04-30T06:26:44.906ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92602025-04-30T06:26:44.906ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
92612025-04-30T06:26:44.909ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92622025-04-30T06:26:44.909ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92632025-04-30T06:26:44.909ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92642025-04-30T06:26:44.911ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92652025-04-30T06:26:44.912ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92662025-04-30T06:26:44.915ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 13799 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 25e2b4e8-d758-4956-9a66-e2c17a158224 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
92672025-04-30T06:26:44.918ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92682025-04-30T06:26:44.923ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92692025-04-30T06:26:44.923ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92702025-04-30T06:26:44.924ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92712025-04-30T06:26:44.924ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92722025-04-30T06:26:44.927ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "25e2b4e8-d758-4956-9a66-e2c17a158224", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92732025-04-30T06:26:44.927ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
92742025-04-30T06:26:44.928ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = c4ec60f8-4493-4b2c-b6b8-9e26048a98ce uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
92752025-04-30T06:26:44.928ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = c4ec60f8-4493-4b2c-b6b8-9e26048a98ce resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
92762025-04-30T06:26:44.928ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = c4ec60f8-4493-4b2c-b6b8-9e26048a98ce roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
92772025-04-30T06:26:44.929ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = c4ec60f8-4493-4b2c-b6b8-9e26048a98ce resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
92782025-04-30T06:26:44.943ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 15323 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = c4ec60f8-4493-4b2c-b6b8-9e26048a98ce response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
92792025-04-30T06:26:44.945ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4ec60f8-4493-4b2c-b6b8-9e26048a98ce", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
92802025-04-30T06:26:44.946ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
92812025-04-30T06:26:44.946ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92822025-04-30T06:26:44.946ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
92832025-04-30T06:26:44.985ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92842025-04-30T06:26:44.985ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92852025-04-30T06:26:44.998ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92862025-04-30T06:26:44.999ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92872025-04-30T06:26:44.999ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92882025-04-30T06:26:44.999ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92892025-04-30T06:26:45.009ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
92902025-04-30T06:26:45.010ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
92912025-04-30T06:26:45.010ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 559.080481ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
92922025-04-30T06:26:45.010ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
92932025-04-30T06:26:45.010ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92942025-04-30T06:26:45.010ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92952025-04-30T06:26:45.010ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92962025-04-30T06:26:45.010ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92972025-04-30T06:26:45.010ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92982025-04-30T06:26:45.023ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92992025-04-30T06:26:45.029ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93002025-04-30T06:26:45.029ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93012025-04-30T06:26:45.030ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93022025-04-30T06:26:45.030ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93032025-04-30T06:26:45.035ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
93042025-04-30T06:26:45.038ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
93052025-04-30T06:26:45.038ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
93062025-04-30T06:26:45.039ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: df 9c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57108 req_id = 1d2c6c5d-f38e-4d0d-8212-34ba46c0da6d
93072025-04-30T06:26:45.039ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57244,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57108 req_id = 1d2c6c5d-f38e-4d0d-8212-34ba46c0da6d
93082025-04-30T06:26:45.040ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
93092025-04-30T06:26:45.040ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93102025-04-30T06:26:45.040ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57108 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1d2c6c5d-f38e-4d0d-8212-34ba46c0da6d
93112025-04-30T06:26:45.041ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3269, tv_nsec: 839891440 } })
93122025-04-30T06:26:45.041ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
93132025-04-30T06:26:45.042ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7a e2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e z...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58924 req_id = e7348aaf-304f-4987-9122-ca55bc00b9d6
93142025-04-30T06:26:45.042ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31458,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58924 req_id = e7348aaf-304f-4987-9122-ca55bc00b9d6
93152025-04-30T06:26:45.042ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93162025-04-30T06:26:45.042ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58924 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e7348aaf-304f-4987-9122-ca55bc00b9d6
93172025-04-30T06:26:45.111ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
93182025-04-30T06:26:45.111ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93192025-04-30T06:26:45.111ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93202025-04-30T06:26:45.111ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93212025-04-30T06:26:45.111ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
93222025-04-30T06:26:45.112ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93232025-04-30T06:26:45.112ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 88 ed 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55740 req_id = 92e2301e-5d83-4fb3-b180-e231319c949e
93242025-04-30T06:26:45.112ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35053,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55740 req_id = 92e2301e-5d83-4fb3-b180-e231319c949e
93252025-04-30T06:26:45.112ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
93262025-04-30T06:26:45.112ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35053, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55740 req_id = 92e2301e-5d83-4fb3-b180-e231319c949e
93272025-04-30T06:26:45.113ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
93282025-04-30T06:26:45.113ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
93292025-04-30T06:26:45.113ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
93302025-04-30T06:26:45.115ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93312025-04-30T06:26:45.121ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93322025-04-30T06:26:45.121ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93332025-04-30T06:26:45.121ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
93342025-04-30T06:26:45.121ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93352025-04-30T06:26:45.124ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "de36ae62-8a3d-4368-a2d3-8281b9397265", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93362025-04-30T06:26:45.124ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
93372025-04-30T06:26:45.137ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93382025-04-30T06:26:45.137ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
93392025-04-30T06:26:45.138ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93402025-04-30T06:26:45.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01419e44-f334-4030-adb2-bb4b2f268e14", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93412025-04-30T06:26:45.141ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
93422025-04-30T06:26:45.146ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6841306-910c-4002-9d78-d3be9bb8eb54", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93432025-04-30T06:26:45.146ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
93442025-04-30T06:26:45.151ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6783e163-b56a-4535-9516-a7fa719ebabc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93452025-04-30T06:26:45.151ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
93462025-04-30T06:26:45.155ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d63d2cd4-5bbb-457b-95e9-cf2400fdd86e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93472025-04-30T06:26:45.156ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
93482025-04-30T06:26:45.159ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fbead84-471b-406d-bf5c-e059aa72318d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93492025-04-30T06:26:45.159ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
93502025-04-30T06:26:45.163ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e224515-459f-4f4c-bb3b-eec000a109bb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93512025-04-30T06:26:45.163ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
93522025-04-30T06:26:45.165ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd66733e-1e65-4e78-ac08-4180e294c247", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93532025-04-30T06:26:45.165ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
93542025-04-30T06:26:45.168ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd636268-1d89-4d63-9642-419795640e2e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93552025-04-30T06:26:45.168ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
93562025-04-30T06:26:45.169ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08cb797e-e9c6-4bfc-8eff-c4cbcc23a12e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93572025-04-30T06:26:45.169ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
93582025-04-30T06:26:45.174ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a121ac22-7559-4db9-8a76-76dc5efb90fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93592025-04-30T06:26:45.174ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
93602025-04-30T06:26:45.177ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
93612025-04-30T06:26:45.177ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
93622025-04-30T06:26:45.177ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
93632025-04-30T06:26:45.179ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
93642025-04-30T06:26:45.181ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b98a8ab-add2-4b39-8add-b65e76e571d9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93652025-04-30T06:26:45.181ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
93662025-04-30T06:26:45.183ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "228f2ec5-4403-4b10-b9f7-52c134b195ef", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93672025-04-30T06:26:45.183ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
93682025-04-30T06:26:45.184ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
93692025-04-30T06:26:45.184ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
93702025-04-30T06:26:45.184ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
93712025-04-30T06:26:45.187ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "340d6319-ea5f-457d-9128-2cebbfc4080a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93722025-04-30T06:26:45.187ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
93732025-04-30T06:26:45.188ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
93742025-04-30T06:26:45.188ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
93752025-04-30T06:26:45.191ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f080a5fc-dc33-469e-b261-8646827fa77b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93762025-04-30T06:26:45.191ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
93772025-04-30T06:26:45.194ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f63cc3f-19cc-4fd2-b2ab-c54dbf0e91a1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93782025-04-30T06:26:45.194ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
93792025-04-30T06:26:45.196ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a0131b1-7dc9-41ad-9c71-bdfcfdb3eec4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93802025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
93812025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "126037cc-6ecb-4e66-9a9a-899c66919301", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93822025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
93832025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43a372f7-68cf-4af4-b039-1a3b5b359969", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93842025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
93852025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b44cb10f-7886-4593-ae5c-1b1e97c77004", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93862025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
93872025-04-30T06:26:45.210ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b9538cf-900e-4c53-9cc3-4ceb60b903bc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93882025-04-30T06:26:45.211ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
93892025-04-30T06:26:45.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1912a30-b8a2-4ab2-a762-bb7d2aa53ba9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93902025-04-30T06:26:45.212ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
93912025-04-30T06:26:45.213ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24ea8ba8-27c8-43c6-a7f7-bb05eeb22971", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93922025-04-30T06:26:45.213ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
93932025-04-30T06:26:45.214ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "459b44e6-036d-4287-97d9-fcff732d6995", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93942025-04-30T06:26:45.215ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
93952025-04-30T06:26:45.218ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5f56344-ad2d-48db-a440-c7b69883f64d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93962025-04-30T06:26:45.218ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
93972025-04-30T06:26:45.220ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c1b7d4-4cc5-470c-bc3b-15278b5e3dd0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
93982025-04-30T06:26:45.220ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
93992025-04-30T06:26:45.224ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e81b1b4-1525-47a4-bd7c-8745997c8ede", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94002025-04-30T06:26:45.229ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
94012025-04-30T06:26:45.230ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "542b16d2-9736-4d31-ae7c-29bae5af0544", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94022025-04-30T06:26:45.231ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
94032025-04-30T06:26:45.233ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fec4eee-289c-421a-9aab-a77e50e3d36c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94042025-04-30T06:26:45.233ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
94052025-04-30T06:26:45.235ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94062025-04-30T06:26:45.235ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94072025-04-30T06:26:45.235ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94082025-04-30T06:26:45.235ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94092025-04-30T06:26:45.236ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94102025-04-30T06:26:45.237ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e923569b-0f85-4c87-af28-ac43af73f65d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94112025-04-30T06:26:45.238ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
94122025-04-30T06:26:45.239ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f725877-c6e9-4fef-9b6a-5c85d38cb210", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94132025-04-30T06:26:45.240ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
94142025-04-30T06:26:45.241ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94152025-04-30T06:26:45.248ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94162025-04-30T06:26:45.248ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94172025-04-30T06:26:45.248ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94182025-04-30T06:26:45.249ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94192025-04-30T06:26:45.251ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcf59e6f-1a74-4c4b-9509-e155a31df9bd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94202025-04-30T06:26:45.252ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
94212025-04-30T06:26:45.253ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5de0114-41ba-4956-b421-f09de21ca31e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94222025-04-30T06:26:45.254ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
94232025-04-30T06:26:45.255ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94242025-04-30T06:26:45.256ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94252025-04-30T06:26:45.257ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94262025-04-30T06:26:45.258ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24bca0c3-2beb-4a05-9f3b-b9d1bb88a786", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94272025-04-30T06:26:45.258ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
94282025-04-30T06:26:45.260ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f10b46b-81a0-4d1f-a098-16ad4934144d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94292025-04-30T06:26:45.260ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
94302025-04-30T06:26:45.263ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4d1c6e4-7036-4089-a4e4-4345156db92b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94312025-04-30T06:26:45.264ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
94322025-04-30T06:26:45.266ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63c8cc72-abf0-436a-b610-e88002bea8a4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94332025-04-30T06:26:45.266ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
94342025-04-30T06:26:45.270ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f00097-5420-45e5-bf27-916ab7538950", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94352025-04-30T06:26:45.270ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
94362025-04-30T06:26:45.272ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b12f509d-9679-4060-9b4e-b82601a1c630", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94372025-04-30T06:26:45.272ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
94382025-04-30T06:26:45.275ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da6bc7f9-54f6-45ed-ac3e-9a90d16863be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94392025-04-30T06:26:45.275ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
94402025-04-30T06:26:45.277ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31675849-2bec-4e5e-85b0-8d39614dce50", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94412025-04-30T06:26:45.277ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
94422025-04-30T06:26:45.278ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
94432025-04-30T06:26:45.278ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fedb1aa1-6e35-4377-a1f5-6f51f233b725", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94442025-04-30T06:26:45.278ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
94452025-04-30T06:26:45.279ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "290df608-2fb6-4e6c-807f-1905f0769140", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94462025-04-30T06:26:45.279ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
94472025-04-30T06:26:45.281ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "227abe35-0a24-4749-95f7-4505bc44a4da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94482025-04-30T06:26:45.281ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
94492025-04-30T06:26:45.282ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f553ed3e-875e-4b75-a8ab-3daab307fbd1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94502025-04-30T06:26:45.283ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94512025-04-30T06:26:45.283ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94522025-04-30T06:26:45.283ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94532025-04-30T06:26:45.283ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94542025-04-30T06:26:45.284ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94552025-04-30T06:26:45.288ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94562025-04-30T06:26:45.294ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94572025-04-30T06:26:45.294ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94582025-04-30T06:26:45.295ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94592025-04-30T06:26:45.295ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94602025-04-30T06:26:45.295ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94612025-04-30T06:26:45.296ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94622025-04-30T06:26:45.314ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94632025-04-30T06:26:45.319ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94642025-04-30T06:26:45.319ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94652025-04-30T06:26:45.319ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94662025-04-30T06:26:45.320ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94672025-04-30T06:26:45.425ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
94682025-04-30T06:26:45.425ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
94692025-04-30T06:26:45.427ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:39721 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94702025-04-30T06:26:45.429ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:39721 req_id = 0088c313-8920-42c6-99a8-9e0d0605beed sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94712025-04-30T06:26:45.429ZINFOtest_disk_metrics (dropshot): request completed latency_us = 843 local_addr = [::1]:50989 method = GET remote_addr = [::1]:39721 req_id = 0088c313-8920-42c6-99a8-9e0d0605beed response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94722025-04-30T06:26:45.431ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0088c313-8920-42c6-99a8-9e0d0605beed", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
94732025-04-30T06:26:45.433ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94742025-04-30T06:26:45.433ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94752025-04-30T06:26:45.433ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
94762025-04-30T06:26:45.437ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94772025-04-30T06:26:45.437ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94782025-04-30T06:26:45.437ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94792025-04-30T06:26:45.438ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94802025-04-30T06:26:45.439ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94812025-04-30T06:26:45.442ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
94822025-04-30T06:26:45.444ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94832025-04-30T06:26:45.451ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94842025-04-30T06:26:45.451ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94852025-04-30T06:26:45.452ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94862025-04-30T06:26:45.452ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94872025-04-30T06:26:45.485ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
94882025-04-30T06:26:45.486ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
94892025-04-30T06:26:45.486ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 476.660124ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
94902025-04-30T06:26:45.486ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
94912025-04-30T06:26:45.486ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94922025-04-30T06:26:45.486ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94932025-04-30T06:26:45.486ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94942025-04-30T06:26:45.486ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94952025-04-30T06:26:45.487ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94962025-04-30T06:26:45.490ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94972025-04-30T06:26:45.495ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94982025-04-30T06:26:45.495ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94992025-04-30T06:26:45.495ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95002025-04-30T06:26:45.495ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95012025-04-30T06:26:45.496ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95022025-04-30T06:26:45.496ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95032025-04-30T06:26:45.496ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95042025-04-30T06:26:45.496ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
95052025-04-30T06:26:45.497ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95062025-04-30T06:26:45.500ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
95072025-04-30T06:26:45.500ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
95082025-04-30T06:26:45.503ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4e e5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 N............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64537 req_id = 824593b3-cbcf-4c2f-9338-fa4711c6a5b0
95092025-04-30T06:26:45.505ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20197,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64537 req_id = 824593b3-cbcf-4c2f-9338-fa4711c6a5b0
95102025-04-30T06:26:45.505ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
95112025-04-30T06:26:45.505ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95122025-04-30T06:26:45.506ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64537 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 824593b3-cbcf-4c2f-9338-fa4711c6a5b0
95132025-04-30T06:26:45.506ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95142025-04-30T06:26:45.520ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95152025-04-30T06:26:45.520ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95162025-04-30T06:26:45.521ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
95172025-04-30T06:26:45.521ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95182025-04-30T06:26:45.522ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3270, tv_nsec: 321369374 } })
95192025-04-30T06:26:45.522ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
95202025-04-30T06:26:45.524ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3c b0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e <...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62268 req_id = 634e7d6e-9078-4188-8f1d-382ae4145810
95212025-04-30T06:26:45.524ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15536,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62268 req_id = 634e7d6e-9078-4188-8f1d-382ae4145810
95222025-04-30T06:26:45.524ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95232025-04-30T06:26:45.524ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62268 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 634e7d6e-9078-4188-8f1d-382ae4145810
95242025-04-30T06:26:45.610ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
95252025-04-30T06:26:45.620ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95262025-04-30T06:26:45.622ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
95272025-04-30T06:26:45.622ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95282025-04-30T06:26:45.627ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9d 7c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .|..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62054 req_id = 425cfa96-dea3-40f2-9c3d-98889339f834
95292025-04-30T06:26:45.627ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40316,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62054 req_id = 425cfa96-dea3-40f2-9c3d-98889339f834
95302025-04-30T06:26:45.629ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
95312025-04-30T06:26:45.629ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40316, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62054 req_id = 425cfa96-dea3-40f2-9c3d-98889339f834
95322025-04-30T06:26:45.629ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
95332025-04-30T06:26:45.629ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
95342025-04-30T06:26:45.630ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
95352025-04-30T06:26:45.634ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8f81ead-a0ca-4c04-baec-32eb1dab4402", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95362025-04-30T06:26:45.635ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
95372025-04-30T06:26:45.639ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8755a76-cb19-4a40-bf0b-3ac70b3896d1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95382025-04-30T06:26:45.639ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
95392025-04-30T06:26:45.642ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa6897dc-236f-4b28-b5a8-5c7a62b6b329", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95402025-04-30T06:26:45.645ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
95412025-04-30T06:26:45.651ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c16b144-70ff-47ab-91a6-105ca1b937b5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95422025-04-30T06:26:45.652ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
95432025-04-30T06:26:45.661ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "124a23da-2120-4ca8-8eda-07640cc349dc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95442025-04-30T06:26:45.661ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
95452025-04-30T06:26:45.667ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2971e992-3a40-4620-aff5-ad7709c22723", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95462025-04-30T06:26:45.667ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
95472025-04-30T06:26:45.674ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
95482025-04-30T06:26:45.676ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d38d54ca-3a51-4ce9-bb4a-1057ce6e1ce9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95492025-04-30T06:26:45.676ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
95502025-04-30T06:26:45.679ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "768600fa-be70-49b4-8fae-bebb6dbde694", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95512025-04-30T06:26:45.680ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
95522025-04-30T06:26:45.680ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "476cd631-d688-426b-b57b-8add48c9921d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95532025-04-30T06:26:45.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
95542025-04-30T06:26:45.682ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05764587-fcf4-4fd1-97d3-c5f5cbab9f31", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95552025-04-30T06:26:45.682ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
95562025-04-30T06:26:45.683ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf384e9f-128e-4d7e-83f0-0b0384ecfd98", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95572025-04-30T06:26:45.683ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
95582025-04-30T06:26:45.684ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "300d61e5-0a8c-41ca-b3d7-d037eeedec0e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95592025-04-30T06:26:45.684ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
95602025-04-30T06:26:45.686ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4243b8d-5d27-4b55-9531-d6e23cb5d563", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95612025-04-30T06:26:45.686ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
95622025-04-30T06:26:45.689ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d82feb-4f35-4c0b-aeaa-8cc245700a88", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95632025-04-30T06:26:45.689ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
95642025-04-30T06:26:45.691ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fdfe1cb-2a9e-4059-b5a3-d69d6b8b1ff1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95652025-04-30T06:26:45.691ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
95662025-04-30T06:26:45.693ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0741762f-1e6a-4ed3-b8c8-e1ed9c699fbc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95672025-04-30T06:26:45.693ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
95682025-04-30T06:26:45.695ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9075d384-47a0-4cd0-a555-3f9a5cbd0a4c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95692025-04-30T06:26:45.695ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
95702025-04-30T06:26:45.698ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39d840ee-945a-4b02-aafd-8127dd39dfcf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95712025-04-30T06:26:45.698ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
95722025-04-30T06:26:45.702ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8fe2c19-5043-4e56-9fe4-f1d55cf830f1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95732025-04-30T06:26:45.702ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
95742025-04-30T06:26:45.704ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86255818-8bc0-4bdb-acf7-87cc2dc99cc0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95752025-04-30T06:26:45.704ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
95762025-04-30T06:26:45.706ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af4e78ea-d5a6-4773-b179-3982a58b19b0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95772025-04-30T06:26:45.706ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
95782025-04-30T06:26:45.708ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3992dba4-460b-479d-9c2f-2755f4f26ecf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95792025-04-30T06:26:45.708ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
95802025-04-30T06:26:45.711ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4eb7c76-1523-40f0-880e-6f93bc8e9c1d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95812025-04-30T06:26:45.711ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
95822025-04-30T06:26:45.712ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f6e14f-0373-47c8-8e59-e164976c4f21", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95832025-04-30T06:26:45.712ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
95842025-04-30T06:26:45.714ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93df8dee-161a-49b9-9c23-b629733f6848", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95852025-04-30T06:26:45.715ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
95862025-04-30T06:26:45.716ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9504586e-9a3f-48ce-a89d-18172b656349", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95872025-04-30T06:26:45.717ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
95882025-04-30T06:26:45.718ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f446ffd6-672d-43e5-9ffd-d9af83ff0072", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95892025-04-30T06:26:45.719ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
95902025-04-30T06:26:45.720ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6673351-f522-4022-8add-9c3acff9262d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95912025-04-30T06:26:45.721ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
95922025-04-30T06:26:45.723ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86358c5a-c9ae-4f69-8718-b8cfd303f5ba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
95932025-04-30T06:26:45.723ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
95942025-04-30T06:26:45.727ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95952025-04-30T06:26:45.727ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95962025-04-30T06:26:45.727ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95972025-04-30T06:26:45.727ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
95982025-04-30T06:26:45.729ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95992025-04-30T06:26:45.734ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96002025-04-30T06:26:45.743ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96012025-04-30T06:26:45.743ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96022025-04-30T06:26:45.743ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96032025-04-30T06:26:45.744ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96042025-04-30T06:26:45.745ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c814ebeb-324d-48f6-a016-b809c82e2539", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96052025-04-30T06:26:45.745ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
96062025-04-30T06:26:45.748ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27c3fef2-2552-4477-8003-2e6147594044", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96072025-04-30T06:26:45.749ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
96082025-04-30T06:26:45.749ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96092025-04-30T06:26:45.749ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96102025-04-30T06:26:45.750ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96112025-04-30T06:26:45.751ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6c99558-6a3f-4b37-88af-9c6c864374e4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96122025-04-30T06:26:45.760ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
96132025-04-30T06:26:45.761ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dfd9977-790a-4b79-a3dc-3a0be922dfe4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96142025-04-30T06:26:45.761ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
96152025-04-30T06:26:45.765ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92f21ee6-81d1-4cf2-a493-6857ea2e1ea8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96162025-04-30T06:26:45.765ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
96172025-04-30T06:26:45.770ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9faedfec-2d65-4020-9916-3acce014d54c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96182025-04-30T06:26:45.771ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
96192025-04-30T06:26:45.777ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74aeda0f-d116-4ae7-8556-b8b733d8bf2d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96202025-04-30T06:26:45.777ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
96212025-04-30T06:26:45.782ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64008ebf-5393-4b26-b011-0924798bcddc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96222025-04-30T06:26:45.783ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
96232025-04-30T06:26:45.797ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a774adf-4c9a-413a-aefb-af511f98e0b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96242025-04-30T06:26:45.797ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
96252025-04-30T06:26:45.798ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
96262025-04-30T06:26:45.801ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dd49dce-d824-41b4-bf8c-042775f351e9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96272025-04-30T06:26:45.801ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
96282025-04-30T06:26:45.804ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d022095-502c-4d23-be1a-90fdcbc96140", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96292025-04-30T06:26:45.804ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
96302025-04-30T06:26:45.806ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1637f38d-c2ab-4f16-a47b-5bf6fdaa040e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96312025-04-30T06:26:45.807ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
96322025-04-30T06:26:45.808ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d89659f8-3bdf-44c9-ae3b-6e3f18ab93bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96332025-04-30T06:26:45.808ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
96342025-04-30T06:26:45.809ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34bccd9e-9285-4861-8bed-250fcf550bfd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96352025-04-30T06:26:45.810ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
96362025-04-30T06:26:45.811ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "022b5648-c1dc-4f8f-bb09-c7d7503822bd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96372025-04-30T06:26:45.811ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
96382025-04-30T06:26:45.812ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edc51841-aee8-4523-a765-d1c3780ae455", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96392025-04-30T06:26:45.813ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96402025-04-30T06:26:45.813ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96412025-04-30T06:26:45.813ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96422025-04-30T06:26:45.813ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96432025-04-30T06:26:45.814ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96442025-04-30T06:26:45.817ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96452025-04-30T06:26:45.822ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96462025-04-30T06:26:45.822ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96472025-04-30T06:26:45.822ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96482025-04-30T06:26:45.822ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96492025-04-30T06:26:45.822ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96502025-04-30T06:26:45.823ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96512025-04-30T06:26:45.830ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96522025-04-30T06:26:45.834ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96532025-04-30T06:26:45.834ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96542025-04-30T06:26:45.835ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96552025-04-30T06:26:45.835ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96562025-04-30T06:26:45.934ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
96572025-04-30T06:26:45.934ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
96582025-04-30T06:26:45.934ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96592025-04-30T06:26:45.934ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96602025-04-30T06:26:45.934ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96612025-04-30T06:26:45.934ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96622025-04-30T06:26:45.936ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96632025-04-30T06:26:45.936ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:50835 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
96642025-04-30T06:26:45.938ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:50835 req_id = 430979bc-1497-44a5-a860-be39e983723b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96652025-04-30T06:26:45.939ZINFOtest_disk_metrics (dropshot): request completed latency_us = 521 local_addr = [::1]:50989 method = GET remote_addr = [::1]:50835 req_id = 430979bc-1497-44a5-a860-be39e983723b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96662025-04-30T06:26:45.943ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "430979bc-1497-44a5-a860-be39e983723b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
96672025-04-30T06:26:45.949ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96682025-04-30T06:26:45.949ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96692025-04-30T06:26:45.949ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
96702025-04-30T06:26:45.951ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96712025-04-30T06:26:45.951ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96722025-04-30T06:26:45.951ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96732025-04-30T06:26:45.951ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96742025-04-30T06:26:45.952ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96752025-04-30T06:26:45.955ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96762025-04-30T06:26:45.961ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96772025-04-30T06:26:45.961ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96782025-04-30T06:26:45.961ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96792025-04-30T06:26:45.994ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96802025-04-30T06:26:46.013ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96812025-04-30T06:26:46.028ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
96822025-04-30T06:26:46.028ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96832025-04-30T06:26:46.028ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96842025-04-30T06:26:46.032ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96852025-04-30T06:26:46.034ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96862025-04-30T06:26:46.039ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96872025-04-30T06:26:46.053ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96882025-04-30T06:26:46.054ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96892025-04-30T06:26:46.054ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96902025-04-30T06:26:46.064ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96912025-04-30T06:26:46.067ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96922025-04-30T06:26:46.068ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96932025-04-30T06:26:46.068ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96942025-04-30T06:26:46.169ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
96952025-04-30T06:26:46.170ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
96962025-04-30T06:26:46.170ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 684.541203ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
96972025-04-30T06:26:46.170ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
96982025-04-30T06:26:46.171ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96992025-04-30T06:26:46.171ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97002025-04-30T06:26:46.171ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97012025-04-30T06:26:46.171ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97022025-04-30T06:26:46.174ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97032025-04-30T06:26:46.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97042025-04-30T06:26:46.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97052025-04-30T06:26:46.175ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
97062025-04-30T06:26:46.176ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
97072025-04-30T06:26:46.179ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97082025-04-30T06:26:46.183ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97092025-04-30T06:26:46.183ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97102025-04-30T06:26:46.183ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97112025-04-30T06:26:46.184ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97122025-04-30T06:26:46.192ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
97132025-04-30T06:26:46.192ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
97142025-04-30T06:26:46.198ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
97152025-04-30T06:26:46.198ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
97162025-04-30T06:26:46.199ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
97172025-04-30T06:26:46.202ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
97182025-04-30T06:26:46.202ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97192025-04-30T06:26:46.202ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 05 b6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49342 req_id = 5590ee8f-2943-49e9-8278-eb5636ed7b4c
97202025-04-30T06:26:46.202ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1462,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49342 req_id = 5590ee8f-2943-49e9-8278-eb5636ed7b4c
97212025-04-30T06:26:46.203ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
97222025-04-30T06:26:46.203ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97232025-04-30T06:26:46.203ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49342 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5590ee8f-2943-49e9-8278-eb5636ed7b4c
97242025-04-30T06:26:46.203ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3271, tv_nsec: 2791693 } })
97252025-04-30T06:26:46.203ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
97262025-04-30T06:26:46.205ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8d 99 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57194 req_id = 6d34fa57-221d-4e2e-9dd5-0be55ff85e1d
97272025-04-30T06:26:46.205ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36249,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57194 req_id = 6d34fa57-221d-4e2e-9dd5-0be55ff85e1d
97282025-04-30T06:26:46.205ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97292025-04-30T06:26:46.206ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57194 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6d34fa57-221d-4e2e-9dd5-0be55ff85e1d
97302025-04-30T06:26:46.267ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
97312025-04-30T06:26:46.268ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97322025-04-30T06:26:46.268ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97332025-04-30T06:26:46.268ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97342025-04-30T06:26:46.268ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97352025-04-30T06:26:46.269ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97362025-04-30T06:26:46.270ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 90 8e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54018 req_id = 91b6df4e-5d5d-491e-8b3e-4e94a499a06a
97372025-04-30T06:26:46.271ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37006,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54018 req_id = 91b6df4e-5d5d-491e-8b3e-4e94a499a06a
97382025-04-30T06:26:46.272ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
97392025-04-30T06:26:46.272ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37006, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54018 req_id = 91b6df4e-5d5d-491e-8b3e-4e94a499a06a
97402025-04-30T06:26:46.273ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
97412025-04-30T06:26:46.273ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
97422025-04-30T06:26:46.273ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
97432025-04-30T06:26:46.274ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97442025-04-30T06:26:46.281ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97452025-04-30T06:26:46.281ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97462025-04-30T06:26:46.281ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97472025-04-30T06:26:46.282ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97482025-04-30T06:26:46.283ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
97492025-04-30T06:26:46.284ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9699e18-352e-48ee-89b2-ca2733cb7420", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97502025-04-30T06:26:46.284ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
97512025-04-30T06:26:46.285ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97522025-04-30T06:26:46.285ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97532025-04-30T06:26:46.286ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97542025-04-30T06:26:46.287ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16ce7040-629d-4776-93ab-b7b226761f16", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97552025-04-30T06:26:46.288ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
97562025-04-30T06:26:46.292ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b86afcc6-d369-4cd5-a67f-95e210d125a1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97572025-04-30T06:26:46.292ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
97582025-04-30T06:26:46.300ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2c812be-03f0-4066-85b3-55622e18a874", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97592025-04-30T06:26:46.300ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
97602025-04-30T06:26:46.302ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cf4b585-d122-404b-a7ab-c651fa484ec9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97612025-04-30T06:26:46.303ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
97622025-04-30T06:26:46.310ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6374b8d9-b201-4a1f-82a1-72f13d7d5b4a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97632025-04-30T06:26:46.310ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
97642025-04-30T06:26:46.311ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8696fd85-c93a-4b0f-831a-596a39e02106", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97652025-04-30T06:26:46.312ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
97662025-04-30T06:26:46.314ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ae1b43b-1052-4981-9afe-8d00e5ae65f2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97672025-04-30T06:26:46.314ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
97682025-04-30T06:26:46.315ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "848b7178-511b-47e4-85fd-46e50a0eb2c5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97692025-04-30T06:26:46.316ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
97702025-04-30T06:26:46.317ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3841893f-13a5-4f25-b48e-2a5790c60fa2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97712025-04-30T06:26:46.317ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
97722025-04-30T06:26:46.322ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8cb7479-60ac-4469-9216-638db6824fb7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97732025-04-30T06:26:46.323ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
97742025-04-30T06:26:46.326ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c85d3150-638c-42a3-80b6-05e74306866d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97752025-04-30T06:26:46.326ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
97762025-04-30T06:26:46.329ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c95a7be-8ad5-44de-ace0-3398b91ebb1a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97772025-04-30T06:26:46.329ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
97782025-04-30T06:26:46.333ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
97792025-04-30T06:26:46.335ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "353e17ab-1ae9-4078-8376-ca57b5dda4c0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97802025-04-30T06:26:46.335ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
97812025-04-30T06:26:46.337ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea923176-27c9-4f72-bf5b-fa48938afc68", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97822025-04-30T06:26:46.337ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
97832025-04-30T06:26:46.340ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1d0e8c3-e72d-4d9e-99e8-7c528bfead4e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97842025-04-30T06:26:46.340ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
97852025-04-30T06:26:46.344ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "670faafe-702d-40d6-83f2-72fcdb794e15", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97862025-04-30T06:26:46.344ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
97872025-04-30T06:26:46.352ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fb508c9-6e54-4bde-96f2-662fa4d3606f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97882025-04-30T06:26:46.352ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
97892025-04-30T06:26:46.355ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3e7e581-c405-410e-8982-133de24c6333", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97902025-04-30T06:26:46.356ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
97912025-04-30T06:26:46.359ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ed85fd0-6440-4516-9468-991c9f92e0ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97922025-04-30T06:26:46.359ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
97932025-04-30T06:26:46.362ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2312edba-b9af-45c8-a5c8-fbdfea21c2f0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97942025-04-30T06:26:46.362ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
97952025-04-30T06:26:46.365ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3496a47-395d-433c-a3f5-82ecd63b36ea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97962025-04-30T06:26:46.365ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
97972025-04-30T06:26:46.366ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7f8bc8b-020e-4596-a32d-98aee6090c82", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
97982025-04-30T06:26:46.366ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
97992025-04-30T06:26:46.368ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67c9074-6af6-4789-a5c3-1e78bc372bfb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98002025-04-30T06:26:46.368ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
98012025-04-30T06:26:46.371ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64fa92a6-f36c-4c69-91c1-263abb274245", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98022025-04-30T06:26:46.371ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
98032025-04-30T06:26:46.373ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "232180f6-376b-402d-8d0a-aa17e7e43f12", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98042025-04-30T06:26:46.374ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
98052025-04-30T06:26:46.375ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2aab3e4-eecd-41f3-98f6-f3e800ecc2a7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98062025-04-30T06:26:46.376ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
98072025-04-30T06:26:46.381ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26c411de-4077-482f-91d3-e1c742d470c8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98082025-04-30T06:26:46.381ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
98092025-04-30T06:26:46.384ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b099de9d-c9e2-4b33-8dc6-a57b647860d4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98102025-04-30T06:26:46.385ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
98112025-04-30T06:26:46.385ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98122025-04-30T06:26:46.385ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98132025-04-30T06:26:46.385ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98142025-04-30T06:26:46.385ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98152025-04-30T06:26:46.387ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98162025-04-30T06:26:46.398ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98172025-04-30T06:26:46.422ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98182025-04-30T06:26:46.422ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98192025-04-30T06:26:46.423ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98202025-04-30T06:26:46.424ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98212025-04-30T06:26:46.428ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98222025-04-30T06:26:46.428ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98232025-04-30T06:26:46.428ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98242025-04-30T06:26:46.431ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69ab97c0-d615-4de3-a38f-78ac5c79c52a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98252025-04-30T06:26:46.431ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
98262025-04-30T06:26:46.438ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf2779f7-4c4e-40f9-aa95-0add931ecc22", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98272025-04-30T06:26:46.438ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
98282025-04-30T06:26:46.440ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aace6322-e267-4252-b80f-1b22955d17a8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98292025-04-30T06:26:46.441ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
98302025-04-30T06:26:46.443ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca774630-ce0e-4dda-affa-9b6597a72448", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98312025-04-30T06:26:46.443ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
98322025-04-30T06:26:46.444ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19a7ce85-6beb-449b-b151-b62983bdca6e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98332025-04-30T06:26:46.445ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
98342025-04-30T06:26:46.447ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bdcdcad-1a2b-4f70-8480-bc8d8fe391aa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98352025-04-30T06:26:46.447ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
98362025-04-30T06:26:46.451ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01c81137-2bce-48cf-822d-9f4ceca9a989", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98372025-04-30T06:26:46.451ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
98382025-04-30T06:26:46.456ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cac587be-0c0e-409d-a0fe-94550ff166c5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98392025-04-30T06:26:46.456ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
98402025-04-30T06:26:46.462ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b395f62e-ed4d-41d1-847d-347eacc77db3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98412025-04-30T06:26:46.463ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
98422025-04-30T06:26:46.463ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
98432025-04-30T06:26:46.467ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d634b1c4-6425-45bf-a80a-767119773bc9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98442025-04-30T06:26:46.468ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
98452025-04-30T06:26:46.472ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4aff0d8-5337-491a-9c9a-78d48b67378e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98462025-04-30T06:26:46.472ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
98472025-04-30T06:26:46.476ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d24741ad-d03e-497a-9ea8-49c242888e10", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98482025-04-30T06:26:46.476ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
98492025-04-30T06:26:46.477ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb333f6d-7fdd-4e31-9d6c-a4deb106833b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98502025-04-30T06:26:46.477ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
98512025-04-30T06:26:46.479ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a4a07af-e2c4-4a76-892a-eccbdc953e99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98522025-04-30T06:26:46.479ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
98532025-04-30T06:26:46.480ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03312568-7447-479a-bbcd-2ea29153eedb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98542025-04-30T06:26:46.480ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
98552025-04-30T06:26:46.482ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ce0438f-1786-43d3-9733-f7678e642c32", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98562025-04-30T06:26:46.482ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98572025-04-30T06:26:46.482ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98582025-04-30T06:26:46.482ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98592025-04-30T06:26:46.482ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98602025-04-30T06:26:46.483ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98612025-04-30T06:26:46.485ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98622025-04-30T06:26:46.496ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98632025-04-30T06:26:46.497ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98642025-04-30T06:26:46.497ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98652025-04-30T06:26:46.497ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98662025-04-30T06:26:46.497ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98672025-04-30T06:26:46.497ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98682025-04-30T06:26:46.503ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98692025-04-30T06:26:46.507ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98702025-04-30T06:26:46.507ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98712025-04-30T06:26:46.507ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98722025-04-30T06:26:46.508ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98732025-04-30T06:26:46.609ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
98742025-04-30T06:26:46.610ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
98752025-04-30T06:26:46.610ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98762025-04-30T06:26:46.610ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98772025-04-30T06:26:46.611ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98782025-04-30T06:26:46.611ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98792025-04-30T06:26:46.611ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98802025-04-30T06:26:46.612ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:50425 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
98812025-04-30T06:26:46.612ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:50425 req_id = ef30e21f-fab5-4463-b3e2-cd5f828b78d7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98822025-04-30T06:26:46.614ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1225 local_addr = [::1]:50989 method = GET remote_addr = [::1]:50425 req_id = ef30e21f-fab5-4463-b3e2-cd5f828b78d7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98832025-04-30T06:26:46.616ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef30e21f-fab5-4463-b3e2-cd5f828b78d7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
98842025-04-30T06:26:46.619ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98852025-04-30T06:26:46.619ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98862025-04-30T06:26:46.619ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
98872025-04-30T06:26:46.621ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
98882025-04-30T06:26:46.621ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98892025-04-30T06:26:46.621ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98902025-04-30T06:26:46.621ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98912025-04-30T06:26:46.622ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98922025-04-30T06:26:46.622ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98932025-04-30T06:26:46.629ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98942025-04-30T06:26:46.629ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98952025-04-30T06:26:46.629ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98962025-04-30T06:26:46.630ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98972025-04-30T06:26:46.634ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98982025-04-30T06:26:46.643ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98992025-04-30T06:26:46.643ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99002025-04-30T06:26:46.644ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99012025-04-30T06:26:46.646ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99022025-04-30T06:26:46.648ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99032025-04-30T06:26:46.648ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99042025-04-30T06:26:46.649ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99052025-04-30T06:26:46.717ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
99062025-04-30T06:26:46.718ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
99072025-04-30T06:26:46.718ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
99082025-04-30T06:26:46.718ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 547.91788ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
99092025-04-30T06:26:46.718ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
99102025-04-30T06:26:46.718ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99112025-04-30T06:26:46.718ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99122025-04-30T06:26:46.719ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99132025-04-30T06:26:46.719ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99142025-04-30T06:26:46.720ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99152025-04-30T06:26:46.728ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99162025-04-30T06:26:46.736ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99172025-04-30T06:26:46.736ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99182025-04-30T06:26:46.736ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99192025-04-30T06:26:46.737ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99202025-04-30T06:26:46.742ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
99212025-04-30T06:26:46.742ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
99222025-04-30T06:26:46.743ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5e dc 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ^............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53715 req_id = c928dce3-ea7b-4e78-a662-2beefd0c30d6
99232025-04-30T06:26:46.743ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24284,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53715 req_id = c928dce3-ea7b-4e78-a662-2beefd0c30d6
99242025-04-30T06:26:46.743ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
99252025-04-30T06:26:46.744ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99262025-04-30T06:26:46.744ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53715 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c928dce3-ea7b-4e78-a662-2beefd0c30d6
99272025-04-30T06:26:46.744ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3271, tv_nsec: 543622153 } })
99282025-04-30T06:26:46.744ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
99292025-04-30T06:26:46.745ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4c 18 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e L...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64566 req_id = 50787782-6d94-4446-aae1-40a779975152
99302025-04-30T06:26:46.745ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19480,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64566 req_id = 50787782-6d94-4446-aae1-40a779975152
99312025-04-30T06:26:46.745ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99322025-04-30T06:26:46.745ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64566 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 50787782-6d94-4446-aae1-40a779975152
99332025-04-30T06:26:46.814ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
99342025-04-30T06:26:46.814ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99352025-04-30T06:26:46.814ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99362025-04-30T06:26:46.814ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99372025-04-30T06:26:46.814ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99382025-04-30T06:26:46.815ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99392025-04-30T06:26:46.816ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e9 12 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61584 req_id = a8fbf7d3-7ba2-4ef0-b6ce-4317f2074f55
99402025-04-30T06:26:46.816ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59666,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61584 req_id = a8fbf7d3-7ba2-4ef0-b6ce-4317f2074f55
99412025-04-30T06:26:46.816ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
99422025-04-30T06:26:46.817ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59666, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61584 req_id = a8fbf7d3-7ba2-4ef0-b6ce-4317f2074f55
99432025-04-30T06:26:46.817ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
99442025-04-30T06:26:46.817ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
99452025-04-30T06:26:46.817ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
99462025-04-30T06:26:46.818ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99472025-04-30T06:26:46.826ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99482025-04-30T06:26:46.826ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99492025-04-30T06:26:46.826ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99502025-04-30T06:26:46.827ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99512025-04-30T06:26:46.830ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0aa89c06-2756-4d78-aa0f-697c17aa0b31", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99522025-04-30T06:26:46.831ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
99532025-04-30T06:26:46.831ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99542025-04-30T06:26:46.831ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99552025-04-30T06:26:46.833ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99562025-04-30T06:26:46.841ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ef5aa3f-92eb-462a-b706-266347eeba1b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99572025-04-30T06:26:46.841ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
99582025-04-30T06:26:46.849ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "796d6bbe-d0ec-4ad4-816c-0a2ce2fa0b90", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99592025-04-30T06:26:46.850ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
99602025-04-30T06:26:46.863ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
99612025-04-30T06:26:46.864ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99622025-04-30T06:26:46.865ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6725630a-9405-4974-882c-d93da258f18e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99632025-04-30T06:26:46.865ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6725630a-9405-4974-882c-d93da258f18e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99642025-04-30T06:26:46.865ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6725630a-9405-4974-882c-d93da258f18e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99652025-04-30T06:26:46.866ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6725630a-9405-4974-882c-d93da258f18e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99662025-04-30T06:26:46.868ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69f3bae6-4bfb-4320-91a9-39dd438c60e5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99672025-04-30T06:26:46.868ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
99682025-04-30T06:26:46.870ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 5807 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6725630a-9405-4974-882c-d93da258f18e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99692025-04-30T06:26:46.871ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6725630a-9405-4974-882c-d93da258f18e", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99702025-04-30T06:26:46.872ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99712025-04-30T06:26:46.873ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 548796c2-e1d5-451e-b1cb-b7bd23157515 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99722025-04-30T06:26:46.873ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 548796c2-e1d5-451e-b1cb-b7bd23157515 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99732025-04-30T06:26:46.873ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 548796c2-e1d5-451e-b1cb-b7bd23157515 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99742025-04-30T06:26:46.874ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 548796c2-e1d5-451e-b1cb-b7bd23157515 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99752025-04-30T06:26:46.884ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 11598 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 548796c2-e1d5-451e-b1cb-b7bd23157515 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99762025-04-30T06:26:46.887ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
99772025-04-30T06:26:46.897ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "548796c2-e1d5-451e-b1cb-b7bd23157515", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99782025-04-30T06:26:46.897ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
99792025-04-30T06:26:46.897ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99802025-04-30T06:26:46.897ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
99812025-04-30T06:26:46.899ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "192d011e-230e-4a82-b1e0-7ec1b674ee6c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99822025-04-30T06:26:46.899ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
99832025-04-30T06:26:46.902ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d524c8ee-e6b5-4316-a246-697cf8db43cd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99842025-04-30T06:26:46.903ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
99852025-04-30T06:26:46.906ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e85b861b-6660-48b1-b58f-8397e002c17a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99862025-04-30T06:26:46.906ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
99872025-04-30T06:26:46.910ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62b1e9ee-0324-44b9-8b3e-567f6aaaf85a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99882025-04-30T06:26:46.911ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
99892025-04-30T06:26:46.913ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eef32744-3a27-44e7-86ec-abdf7c9cb466", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99902025-04-30T06:26:46.913ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
99912025-04-30T06:26:46.920ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edd18908-fcac-480d-a3a8-b92eb729c630", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99922025-04-30T06:26:46.920ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
99932025-04-30T06:26:46.921ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cde4a29-0918-40e6-abb9-7dd94abc8641", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99942025-04-30T06:26:46.921ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
99952025-04-30T06:26:46.926ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ec67de9-19a2-4c98-bfc8-121bcb04ece3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99962025-04-30T06:26:46.926ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
99972025-04-30T06:26:46.929ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c1d83ea-9847-4949-8cc6-095154682708", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
99982025-04-30T06:26:46.929ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
99992025-04-30T06:26:46.938ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100002025-04-30T06:26:46.939ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100012025-04-30T06:26:46.939ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100022025-04-30T06:26:46.939ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100032025-04-30T06:26:46.940ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100042025-04-30T06:26:46.948ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100052025-04-30T06:26:46.956ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100062025-04-30T06:26:46.956ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100072025-04-30T06:26:46.956ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100082025-04-30T06:26:46.956ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100092025-04-30T06:26:46.960ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100102025-04-30T06:26:46.960ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100112025-04-30T06:26:46.961ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100122025-04-30T06:26:46.976ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9adf7fe-2d67-4c36-a898-b4dd2d10c498", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100132025-04-30T06:26:46.976ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
100142025-04-30T06:26:46.982ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82866cd6-755a-4096-a0d7-79f07ace35ad", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100152025-04-30T06:26:46.983ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
100162025-04-30T06:26:46.984ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100172025-04-30T06:26:46.984ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100182025-04-30T06:26:46.986ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100192025-04-30T06:26:46.986ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100202025-04-30T06:26:46.986ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100212025-04-30T06:26:46.986ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100222025-04-30T06:26:46.989ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97664836-0e9e-475b-8b79-e023ce67442b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100232025-04-30T06:26:47.003ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
100242025-04-30T06:26:47.015ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "578724f7-4cd9-4c53-9999-ef95dd42dc57", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100252025-04-30T06:26:47.015ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
100262025-04-30T06:26:47.022ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a75e454-7842-47ce-917d-42bddb05def2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100272025-04-30T06:26:47.022ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
100282025-04-30T06:26:47.029ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f98a5f2-f122-4444-b96d-f13ebf3b20b9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100292025-04-30T06:26:47.029ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
100302025-04-30T06:26:47.030ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
100312025-04-30T06:26:47.032ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "457556f5-d9c9-4699-bd45-7417c7e34138", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100322025-04-30T06:26:47.032ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
100332025-04-30T06:26:47.034ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "821aba0f-713d-4bdd-9732-112ad6da99b1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100342025-04-30T06:26:47.034ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
100352025-04-30T06:26:47.036ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc953606-4279-4b71-b253-1b8af46fa4b9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100362025-04-30T06:26:47.036ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
100372025-04-30T06:26:47.052ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44030839-14e3-45c0-937e-7cd4060e37f3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100382025-04-30T06:26:47.052ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
100392025-04-30T06:26:47.056ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a080fd47-7473-4726-8eba-29e4d8a03814", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100402025-04-30T06:26:47.056ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
100412025-04-30T06:26:47.060ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c06a9cca-fc14-4f57-ad85-188801163bfb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100422025-04-30T06:26:47.060ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
100432025-04-30T06:26:47.062ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53d98adf-f01d-4218-9dc6-03403e2c8a0d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100442025-04-30T06:26:47.062ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
100452025-04-30T06:26:47.065ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e135deae-fdc2-40c4-953d-c7f1199678d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100462025-04-30T06:26:47.065ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
100472025-04-30T06:26:47.068ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d19a2c2-5096-4bac-a579-922fa9fcfba0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100482025-04-30T06:26:47.068ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
100492025-04-30T06:26:47.071ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1919ead1-ceb4-467a-91db-1c890adf5d52", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100502025-04-30T06:26:47.071ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
100512025-04-30T06:26:47.074ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a92ac438-5dd4-4db4-b948-6d71ff5af4d7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100522025-04-30T06:26:47.083ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
100532025-04-30T06:26:47.083ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100542025-04-30T06:26:47.083ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100552025-04-30T06:26:47.083ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100562025-04-30T06:26:47.083ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100572025-04-30T06:26:47.084ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100582025-04-30T06:26:47.085ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "935e2f93-1410-49c7-8213-9bc57f38dbc9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100592025-04-30T06:26:47.085ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
100602025-04-30T06:26:47.088ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100612025-04-30T06:26:47.095ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100622025-04-30T06:26:47.095ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100632025-04-30T06:26:47.095ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100642025-04-30T06:26:47.096ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100652025-04-30T06:26:47.097ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94ff2c71-8ef0-4af5-a5af-93b2fcb5441f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100662025-04-30T06:26:47.097ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
100672025-04-30T06:26:47.098ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100682025-04-30T06:26:47.099ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100692025-04-30T06:26:47.099ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100702025-04-30T06:26:47.099ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff3dee40-ef43-4a6f-923d-02268f12fdcf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100712025-04-30T06:26:47.099ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
100722025-04-30T06:26:47.103ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "462e4ebc-e226-4f5c-9a77-be27cfcd6d73", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100732025-04-30T06:26:47.104ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
100742025-04-30T06:26:47.105ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba1f083-0316-40b5-ba98-d2c18a7048c5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100752025-04-30T06:26:47.105ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
100762025-04-30T06:26:47.111ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a91f14b-705b-4471-bbbd-deb4d1eb4a1a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100772025-04-30T06:26:47.111ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
100782025-04-30T06:26:47.120ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0eba29b-3dd3-49d7-94ae-3c168b7faf56", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100792025-04-30T06:26:47.120ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
100802025-04-30T06:26:47.125ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5d34620-a3f2-4655-a95f-961f097f97b8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100812025-04-30T06:26:47.125ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
100822025-04-30T06:26:47.129ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c5580d1-dc9f-44ce-8595-1197b5746226", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100832025-04-30T06:26:47.129ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
100842025-04-30T06:26:47.134ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be15767c-9a68-4132-8f9c-d36f5d7f4dde", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100852025-04-30T06:26:47.134ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
100862025-04-30T06:26:47.135ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
100872025-04-30T06:26:47.137ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c71912-f05f-46c8-925c-2698e5b1bdaa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100882025-04-30T06:26:47.137ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
100892025-04-30T06:26:47.139ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f8a4deb-539b-441e-9759-c6a432567e49", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100902025-04-30T06:26:47.139ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
100912025-04-30T06:26:47.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e374cfc7-2a9f-4bb6-9c65-abf768a6f017", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100922025-04-30T06:26:47.141ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
100932025-04-30T06:26:47.142ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ff11150-34d2-45c1-8122-7a2e0666f3ba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100942025-04-30T06:26:47.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
100952025-04-30T06:26:47.144ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3fe7829-9433-4755-adf1-a57b026dec14", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
100962025-04-30T06:26:47.144ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
100972025-04-30T06:26:47.144ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100982025-04-30T06:26:47.144ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100992025-04-30T06:26:47.144ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101002025-04-30T06:26:47.145ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101012025-04-30T06:26:47.149ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101022025-04-30T06:26:47.153ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101032025-04-30T06:26:47.161ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101042025-04-30T06:26:47.161ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101052025-04-30T06:26:47.161ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101062025-04-30T06:26:47.161ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101072025-04-30T06:26:47.163ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101082025-04-30T06:26:47.166ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101092025-04-30T06:26:47.173ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101102025-04-30T06:26:47.173ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101112025-04-30T06:26:47.173ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101122025-04-30T06:26:47.174ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101132025-04-30T06:26:47.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101142025-04-30T06:26:47.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101152025-04-30T06:26:47.175ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
101162025-04-30T06:26:47.177ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
101172025-04-30T06:26:47.242ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
101182025-04-30T06:26:47.242ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
101192025-04-30T06:26:47.243ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101202025-04-30T06:26:47.243ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101212025-04-30T06:26:47.243ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101222025-04-30T06:26:47.243ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101232025-04-30T06:26:47.243ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101242025-04-30T06:26:47.243ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:62939 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
101252025-04-30T06:26:47.244ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
101262025-04-30T06:26:47.244ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
101272025-04-30T06:26:47.250ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:62939 req_id = cd02919b-20d8-4b97-9c25-34066d41b002 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101282025-04-30T06:26:47.251ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1203 local_addr = [::1]:50989 method = GET remote_addr = [::1]:62939 req_id = cd02919b-20d8-4b97-9c25-34066d41b002 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101292025-04-30T06:26:47.252ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
101302025-04-30T06:26:47.252ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101312025-04-30T06:26:47.253ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd02919b-20d8-4b97-9c25-34066d41b002", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
101322025-04-30T06:26:47.254ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101332025-04-30T06:26:47.254ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101342025-04-30T06:26:47.254ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
101352025-04-30T06:26:47.256ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101362025-04-30T06:26:47.256ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101372025-04-30T06:26:47.256ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101382025-04-30T06:26:47.256ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101392025-04-30T06:26:47.257ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101402025-04-30T06:26:47.258ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101412025-04-30T06:26:47.263ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101422025-04-30T06:26:47.263ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101432025-04-30T06:26:47.263ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101442025-04-30T06:26:47.263ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101452025-04-30T06:26:47.269ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101462025-04-30T06:26:47.269ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101472025-04-30T06:26:47.270ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101482025-04-30T06:26:47.270ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101492025-04-30T06:26:47.275ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101502025-04-30T06:26:47.275ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101512025-04-30T06:26:47.275ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101522025-04-30T06:26:47.276ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101532025-04-30T06:26:47.276ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
101542025-04-30T06:26:47.316ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101552025-04-30T06:26:47.317ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101562025-04-30T06:26:47.317ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 599.033685ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101572025-04-30T06:26:47.318ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
101582025-04-30T06:26:47.318ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101592025-04-30T06:26:47.318ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101602025-04-30T06:26:47.318ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101612025-04-30T06:26:47.318ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101622025-04-30T06:26:47.319ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101632025-04-30T06:26:47.319ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
101642025-04-30T06:26:47.321ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101652025-04-30T06:26:47.325ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101662025-04-30T06:26:47.325ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101672025-04-30T06:26:47.325ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101682025-04-30T06:26:47.327ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101692025-04-30T06:26:47.331ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
101702025-04-30T06:26:47.331ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101712025-04-30T06:26:47.332ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 26 83 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 &............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53232 req_id = bb6a2323-9c80-47c9-b0d4-604a627a7fe2
101722025-04-30T06:26:47.332ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9859,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53232 req_id = bb6a2323-9c80-47c9-b0d4-604a627a7fe2
101732025-04-30T06:26:47.333ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
101742025-04-30T06:26:47.333ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101752025-04-30T06:26:47.333ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53232 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bb6a2323-9c80-47c9-b0d4-604a627a7fe2
101762025-04-30T06:26:47.334ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3272, tv_nsec: 133292532 } })
101772025-04-30T06:26:47.334ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
101782025-04-30T06:26:47.334ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: eb ae 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54474 req_id = c9018ffa-426f-4313-89f9-db293c9a4d86
101792025-04-30T06:26:47.334ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60334,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54474 req_id = c9018ffa-426f-4313-89f9-db293c9a4d86
101802025-04-30T06:26:47.335ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101812025-04-30T06:26:47.335ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54474 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c9018ffa-426f-4313-89f9-db293c9a4d86
101822025-04-30T06:26:47.383ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101832025-04-30T06:26:47.385ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101842025-04-30T06:26:47.385ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101852025-04-30T06:26:47.385ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101862025-04-30T06:26:47.385ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101872025-04-30T06:26:47.386ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101882025-04-30T06:26:47.387ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e4 35 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .5..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55932 req_id = 56f0369d-28d3-4d1e-b87f-103f89b0f97e
101892025-04-30T06:26:47.387ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58421,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55932 req_id = 56f0369d-28d3-4d1e-b87f-103f89b0f97e
101902025-04-30T06:26:47.387ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
101912025-04-30T06:26:47.387ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58421, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55932 req_id = 56f0369d-28d3-4d1e-b87f-103f89b0f97e
101922025-04-30T06:26:47.387ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
101932025-04-30T06:26:47.388ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
101942025-04-30T06:26:47.388ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
101952025-04-30T06:26:47.389ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101962025-04-30T06:26:47.404ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101972025-04-30T06:26:47.404ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101982025-04-30T06:26:47.404ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101992025-04-30T06:26:47.406ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102002025-04-30T06:26:47.410ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "77170b66-3eb9-40e5-9deb-4987ede8e5df", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102012025-04-30T06:26:47.410ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
102022025-04-30T06:26:47.411ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57851213-ba06-45b5-8eef-93dd0619db2e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102032025-04-30T06:26:47.412ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
102042025-04-30T06:26:47.412ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102052025-04-30T06:26:47.413ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102062025-04-30T06:26:47.413ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102072025-04-30T06:26:47.415ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed97413-859c-401b-a7b2-f4c44e0bb03f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102082025-04-30T06:26:47.415ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
102092025-04-30T06:26:47.417ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef8feb0e-707d-42e5-8f05-2fda70923287", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102102025-04-30T06:26:47.418ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
102112025-04-30T06:26:47.420ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "728341c0-a06b-4ea7-8a25-4b90ea16b54c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102122025-04-30T06:26:47.420ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
102132025-04-30T06:26:47.425ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5d3cc2f-4a6d-4495-8d21-c40981df3cda", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102142025-04-30T06:26:47.425ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
102152025-04-30T06:26:47.430ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bd23b9c-c425-4a3f-a9ca-ae4d5d0bc975", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102162025-04-30T06:26:47.435ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
102172025-04-30T06:26:47.444ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "627a9985-18ff-4f3a-86cc-d7244bc459cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102182025-04-30T06:26:47.444ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
102192025-04-30T06:26:47.445ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6008cc50-6b4b-4893-ad00-3721c8f15a49", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102202025-04-30T06:26:47.446ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
102212025-04-30T06:26:47.451ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1243d73-14a9-40e8-8dd6-cdde82cb93cc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102222025-04-30T06:26:47.452ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
102232025-04-30T06:26:47.453ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
102242025-04-30T06:26:47.457ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "913fddcd-24d2-474f-a4d6-8b6acd75a809", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102252025-04-30T06:26:47.458ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
102262025-04-30T06:26:47.459ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96ae7401-fa4f-483b-85a9-413d192e176c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102272025-04-30T06:26:47.459ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
102282025-04-30T06:26:47.461ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d21b1df-83ef-461e-95ac-5cdcce8eb33f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102292025-04-30T06:26:47.462ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
102302025-04-30T06:26:47.465ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dbb8273-b02e-4d65-8898-8dd5c93d7c8a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102312025-04-30T06:26:47.466ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
102322025-04-30T06:26:47.471ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b43d14a-830c-42fa-bf8e-5f249ed7f004", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102332025-04-30T06:26:47.471ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
102342025-04-30T06:26:47.476ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a10ab666-dd0d-44f1-bd51-15eea5bf9ca9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102352025-04-30T06:26:47.476ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
102362025-04-30T06:26:47.477ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f7d00a7-abd6-49ef-ae2c-ed43cb01ceb5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102372025-04-30T06:26:47.477ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
102382025-04-30T06:26:47.480ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b1db075-0cd3-430e-9532-8ee2f64158b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102392025-04-30T06:26:47.480ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
102402025-04-30T06:26:47.482ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ee22c81-c685-49a2-bfa7-609a7ad5e653", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102412025-04-30T06:26:47.482ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
102422025-04-30T06:26:47.483ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b498f72c-d73f-4c59-adcb-702842a53bb2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102432025-04-30T06:26:47.484ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
102442025-04-30T06:26:47.485ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38ce1e8f-d854-4ed0-91c6-22d2c0ef23eb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102452025-04-30T06:26:47.485ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
102462025-04-30T06:26:47.486ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91c19c89-b7ca-44b9-ad09-c3595beca8f2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102472025-04-30T06:26:47.487ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
102482025-04-30T06:26:47.487ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9fd7710-bb3d-4ba4-9e12-9dbbdcb1dcb7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102492025-04-30T06:26:47.488ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
102502025-04-30T06:26:47.489ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3ce749-d4fb-4167-b0a8-ffc8bab3c290", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102512025-04-30T06:26:47.489ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
102522025-04-30T06:26:47.497ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a603e7a-cfc7-40ef-b869-35dfd0cd3f5f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102532025-04-30T06:26:47.497ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
102542025-04-30T06:26:47.499ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93ecb9ac-a139-42dd-870d-4562d64524ff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102552025-04-30T06:26:47.499ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
102562025-04-30T06:26:47.503ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b03f9c5f-7bd3-45a4-a65a-d4f53c233a88", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102572025-04-30T06:26:47.503ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
102582025-04-30T06:26:47.504ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102592025-04-30T06:26:47.504ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102602025-04-30T06:26:47.504ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102612025-04-30T06:26:47.504ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102622025-04-30T06:26:47.506ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102632025-04-30T06:26:47.508ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102642025-04-30T06:26:47.518ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102652025-04-30T06:26:47.519ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102662025-04-30T06:26:47.519ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102672025-04-30T06:26:47.519ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102682025-04-30T06:26:47.521ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "074eb1a4-17bd-449d-91a7-7da5b05e1791", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102692025-04-30T06:26:47.521ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
102702025-04-30T06:26:47.522ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102712025-04-30T06:26:47.522ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102722025-04-30T06:26:47.522ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102732025-04-30T06:26:47.525ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef5fc31f-8053-4155-afe1-77c9267bd8a3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102742025-04-30T06:26:47.525ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
102752025-04-30T06:26:47.530ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a222095-2847-4e27-a061-94a51c9cd466", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102762025-04-30T06:26:47.530ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
102772025-04-30T06:26:47.535ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94614de3-822d-4403-a329-b3aa58d9e725", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102782025-04-30T06:26:47.536ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
102792025-04-30T06:26:47.537ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5665e254-f432-4520-a587-76498640dc1e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102802025-04-30T06:26:47.537ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
102812025-04-30T06:26:47.538ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c320d2a9-c219-4cb6-9de3-102ada8645df", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102822025-04-30T06:26:47.539ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
102832025-04-30T06:26:47.542ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8780cbca-59aa-4c6a-b5fb-81812ab875a4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102842025-04-30T06:26:47.543ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
102852025-04-30T06:26:47.545ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93366a2b-b852-4348-a57b-67908019ba6f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102862025-04-30T06:26:47.545ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
102872025-04-30T06:26:47.549ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b4809ba-d1f2-44da-bd5c-2a94adaa2e99", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102882025-04-30T06:26:47.549ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
102892025-04-30T06:26:47.555ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38f0d671-7a84-4f93-9d12-fb60a0fa5c83", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102902025-04-30T06:26:47.555ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
102912025-04-30T06:26:47.562ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
102922025-04-30T06:26:47.564ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72f7c17b-c220-4afc-a5b6-3932ac303337", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102932025-04-30T06:26:47.564ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
102942025-04-30T06:26:47.566ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8b4011d-155a-4a48-ac96-3599b0812e6d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102952025-04-30T06:26:47.566ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
102962025-04-30T06:26:47.569ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05deebaf-8fa8-4eee-936a-122b840e32f6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102972025-04-30T06:26:47.569ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
102982025-04-30T06:26:47.571ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e74eb456-40b6-4041-a7ec-7098c7c04349", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
102992025-04-30T06:26:47.571ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
103002025-04-30T06:26:47.577ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3126463c-f689-4901-bfab-ceb98f40a70a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
103012025-04-30T06:26:47.578ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
103022025-04-30T06:26:47.579ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a21ef0a4-933d-413d-b372-87175daf8cf8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
103032025-04-30T06:26:47.581ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
103042025-04-30T06:26:47.582ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97bc3c94-18f9-43e4-a139-c58084f12b9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
103052025-04-30T06:26:47.584ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
103062025-04-30T06:26:47.585ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43c698aa-7df1-4802-adee-e6763da98af3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
103072025-04-30T06:26:47.585ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103082025-04-30T06:26:47.586ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103092025-04-30T06:26:47.586ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103102025-04-30T06:26:47.586ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103112025-04-30T06:26:47.587ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103122025-04-30T06:26:47.599ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103132025-04-30T06:26:47.617ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103142025-04-30T06:26:47.618ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103152025-04-30T06:26:47.618ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103162025-04-30T06:26:47.618ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103172025-04-30T06:26:47.618ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103182025-04-30T06:26:47.619ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103192025-04-30T06:26:47.619ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103202025-04-30T06:26:47.620ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103212025-04-30T06:26:47.620ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103222025-04-30T06:26:47.620ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
103232025-04-30T06:26:47.621ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103242025-04-30T06:26:47.623ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103252025-04-30T06:26:47.630ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103262025-04-30T06:26:47.630ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103272025-04-30T06:26:47.630ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103282025-04-30T06:26:47.631ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103292025-04-30T06:26:47.632ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103302025-04-30T06:26:47.641ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103312025-04-30T06:26:47.641ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103322025-04-30T06:26:47.641ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
103332025-04-30T06:26:47.642ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103342025-04-30T06:26:47.719ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
103352025-04-30T06:26:47.719ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
103362025-04-30T06:26:47.719ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:36943 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
103372025-04-30T06:26:47.720ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:36943 req_id = af5ddb8c-feed-4d07-8314-226319315770 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103382025-04-30T06:26:47.720ZINFOtest_disk_metrics (dropshot): request completed latency_us = 460 local_addr = [::1]:50989 method = GET remote_addr = [::1]:36943 req_id = af5ddb8c-feed-4d07-8314-226319315770 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103392025-04-30T06:26:47.720ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103402025-04-30T06:26:47.720ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
103412025-04-30T06:26:47.723ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103422025-04-30T06:26:47.723ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "af5ddb8c-feed-4d07-8314-226319315770", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
103432025-04-30T06:26:47.725ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103442025-04-30T06:26:47.725ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103452025-04-30T06:26:47.725ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
103462025-04-30T06:26:47.735ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103472025-04-30T06:26:47.735ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103482025-04-30T06:26:47.735ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103492025-04-30T06:26:47.735ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103502025-04-30T06:26:47.736ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103512025-04-30T06:26:47.741ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103522025-04-30T06:26:47.754ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103532025-04-30T06:26:47.754ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103542025-04-30T06:26:47.754ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103552025-04-30T06:26:47.755ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103562025-04-30T06:26:47.796ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
103572025-04-30T06:26:47.800ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
103582025-04-30T06:26:47.800ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
103592025-04-30T06:26:47.801ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 482.738183ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
103602025-04-30T06:26:47.801ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
103612025-04-30T06:26:47.801ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103622025-04-30T06:26:47.801ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103632025-04-30T06:26:47.801ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103642025-04-30T06:26:47.801ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103652025-04-30T06:26:47.803ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103662025-04-30T06:26:47.808ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103672025-04-30T06:26:47.818ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103682025-04-30T06:26:47.818ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103692025-04-30T06:26:47.818ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103702025-04-30T06:26:47.820ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103712025-04-30T06:26:47.825ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
103722025-04-30T06:26:47.825ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
103732025-04-30T06:26:47.826ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 09 86 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52382 req_id = 7d1c5f97-37a5-4681-ba3e-f1add15d24c7
103742025-04-30T06:26:47.826ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2438,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52382 req_id = 7d1c5f97-37a5-4681-ba3e-f1add15d24c7
103752025-04-30T06:26:47.826ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
103762025-04-30T06:26:47.827ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103772025-04-30T06:26:47.827ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52382 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7d1c5f97-37a5-4681-ba3e-f1add15d24c7
103782025-04-30T06:26:47.827ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3272, tv_nsec: 626410707 } })
103792025-04-30T06:26:47.827ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
103802025-04-30T06:26:47.828ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 61 4d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e aM..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51868 req_id = 176caba7-10c7-4a95-b3a0-2d38f4f331e1
103812025-04-30T06:26:47.828ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24909,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51868 req_id = 176caba7-10c7-4a95-b3a0-2d38f4f331e1
103822025-04-30T06:26:47.828ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103832025-04-30T06:26:47.828ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51868 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 176caba7-10c7-4a95-b3a0-2d38f4f331e1
103842025-04-30T06:26:47.892ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
103852025-04-30T06:26:47.893ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103862025-04-30T06:26:47.893ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103872025-04-30T06:26:47.893ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103882025-04-30T06:26:47.893ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
103892025-04-30T06:26:47.894ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103902025-04-30T06:26:47.894ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 30 95 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 0............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60885 req_id = 87b5b75d-3a97-4b36-b9a5-6c0bf5f06dc8
103912025-04-30T06:26:47.894ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12437,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60885 req_id = 87b5b75d-3a97-4b36-b9a5-6c0bf5f06dc8
103922025-04-30T06:26:47.895ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
103932025-04-30T06:26:47.895ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12437, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60885 req_id = 87b5b75d-3a97-4b36-b9a5-6c0bf5f06dc8
103942025-04-30T06:26:47.895ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
103952025-04-30T06:26:47.895ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
103962025-04-30T06:26:47.895ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
103972025-04-30T06:26:47.897ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c983cd70-be9c-4cf0-9cb6-72df8db45b90", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
103982025-04-30T06:26:47.897ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
103992025-04-30T06:26:47.898ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104002025-04-30T06:26:47.906ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104012025-04-30T06:26:47.906ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104022025-04-30T06:26:47.907ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104032025-04-30T06:26:47.907ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104042025-04-30T06:26:47.913ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104052025-04-30T06:26:47.913ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104062025-04-30T06:26:47.913ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104072025-04-30T06:26:47.914ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "394a3d70-e962-429d-9670-b7046053aa20", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104082025-04-30T06:26:47.914ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
104092025-04-30T06:26:47.919ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9624d32-2073-457a-8628-1e03e3da8f4e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104102025-04-30T06:26:47.919ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
104112025-04-30T06:26:47.924ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09df5f67-ba96-485e-a512-6b15a4317dc0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104122025-04-30T06:26:47.924ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
104132025-04-30T06:26:47.927ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36e0312b-99fa-4fce-8cb5-98ec0bd6df35", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104142025-04-30T06:26:47.927ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
104152025-04-30T06:26:47.976ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
104162025-04-30T06:26:47.984ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104172025-04-30T06:26:47.984ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104182025-04-30T06:26:47.986ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16f7a5b1-8505-4afd-a648-454f06f0e57f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104192025-04-30T06:26:47.986ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
104202025-04-30T06:26:47.987ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104212025-04-30T06:26:47.989ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104222025-04-30T06:26:47.989ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104232025-04-30T06:26:47.989ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104242025-04-30T06:26:47.991ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5136e22-4603-4a07-bda5-38a6790ad0f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104252025-04-30T06:26:47.992ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
104262025-04-30T06:26:47.995ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6282aa5-522a-4fb5-b547-278dbfc9dd5e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104272025-04-30T06:26:47.995ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
104282025-04-30T06:26:47.998ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50b7311b-43a0-4918-b2f3-d5624346cb78", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104292025-04-30T06:26:47.998ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
104302025-04-30T06:26:48.003ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6c3fd33-6bf0-4599-b9ee-3600aa58e74f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104312025-04-30T06:26:48.003ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
104322025-04-30T06:26:48.004ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d81d9ed-1885-4c35-bdd2-fa5a1ac4b088", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104332025-04-30T06:26:48.004ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
104342025-04-30T06:26:48.006ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c49d204-7d92-4282-9bad-d9dee38f6898", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104352025-04-30T06:26:48.006ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
104362025-04-30T06:26:48.008ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c07fe92b-f396-4b1e-bdd4-be740501f563", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104372025-04-30T06:26:48.009ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
104382025-04-30T06:26:48.011ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ea2701a-9e4c-4f57-a924-3d1a2c444d14", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104392025-04-30T06:26:48.011ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
104402025-04-30T06:26:48.014ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aa34533-65c7-474e-8224-f684846e159f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104412025-04-30T06:26:48.014ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
104422025-04-30T06:26:48.016ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "257771b1-afc8-4e5b-875e-6019a2eb42ec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104432025-04-30T06:26:48.016ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
104442025-04-30T06:26:48.020ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16b1c6ca-e126-4b6d-b021-47eef1d11b19", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104452025-04-30T06:26:48.020ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
104462025-04-30T06:26:48.023ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a750e643-20ae-482b-80e0-d052889c8457", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104472025-04-30T06:26:48.023ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
104482025-04-30T06:26:48.025ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3a9b73-3822-4563-9b20-5aa60bf789ad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104492025-04-30T06:26:48.025ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
104502025-04-30T06:26:48.027ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f85f2071-1cb9-43c7-99d1-8a5ddfaba6ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104512025-04-30T06:26:48.027ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
104522025-04-30T06:26:48.027ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104532025-04-30T06:26:48.027ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104542025-04-30T06:26:48.027ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104552025-04-30T06:26:48.028ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104562025-04-30T06:26:48.029ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104572025-04-30T06:26:48.032ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89dd8456-fdcf-46ff-9053-549c803162d6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104582025-04-30T06:26:48.032ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
104592025-04-30T06:26:48.033ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d61538b6-5225-465c-9129-31ac07ae913c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104602025-04-30T06:26:48.033ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
104612025-04-30T06:26:48.033ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104622025-04-30T06:26:48.043ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104632025-04-30T06:26:48.043ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104642025-04-30T06:26:48.043ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104652025-04-30T06:26:48.044ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104662025-04-30T06:26:48.045ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbace63c-f68b-499e-8d91-f88e3ce26322", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104672025-04-30T06:26:48.046ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
104682025-04-30T06:26:48.049ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104692025-04-30T06:26:48.049ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104702025-04-30T06:26:48.051ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104712025-04-30T06:26:48.052ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3712548-2d70-4dc8-9c2b-659e45d74a21", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104722025-04-30T06:26:48.052ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
104732025-04-30T06:26:48.058ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b09c46f-e4a8-474e-88a8-2d48a5e84b48", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104742025-04-30T06:26:48.058ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
104752025-04-30T06:26:48.074ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae2c99fd-5df9-46d1-a30d-07cc1fe30e47", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104762025-04-30T06:26:48.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
104772025-04-30T06:26:48.079ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11698d22-7408-4113-aa51-f0c1fdc563f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104782025-04-30T06:26:48.079ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
104792025-04-30T06:26:48.083ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f234e69d-e7a0-4ec2-9b61-ddc7a43dec56", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104802025-04-30T06:26:48.083ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
104812025-04-30T06:26:48.087ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab76f285-1b54-4293-bd44-b72151409a9e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104822025-04-30T06:26:48.087ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
104832025-04-30T06:26:48.091ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ac29cc4-51ac-4e70-9907-bf87ea3ae86c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104842025-04-30T06:26:48.091ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
104852025-04-30T06:26:48.092ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
104862025-04-30T06:26:48.094ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76b34a7a-36fc-4158-9644-5e2755abbdff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104872025-04-30T06:26:48.095ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
104882025-04-30T06:26:48.098ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fa5b3c2-7117-4fd9-b863-6148fb09b93d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104892025-04-30T06:26:48.098ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
104902025-04-30T06:26:48.099ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9d1acbb-8083-4af0-8399-99ec9e945061", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104912025-04-30T06:26:48.100ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
104922025-04-30T06:26:48.101ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e75469b-aac4-47df-9972-895814279bf0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104932025-04-30T06:26:48.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
104942025-04-30T06:26:48.103ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a9ab59-3213-4da8-a7b4-e07465ab488c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104952025-04-30T06:26:48.103ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
104962025-04-30T06:26:48.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22974940-c301-4d14-90bc-fb1cf4144fe7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104972025-04-30T06:26:48.108ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
104982025-04-30T06:26:48.113ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2843e3d9-cc1d-46e7-86c6-68c5cde3c5a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
104992025-04-30T06:26:48.113ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
105002025-04-30T06:26:48.120ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9af7b41-0e61-4d7d-ab6d-df65a6465958", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105012025-04-30T06:26:48.120ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
105022025-04-30T06:26:48.125ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83c96156-fc0f-42c4-9d04-dc0d6572bbb0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105032025-04-30T06:26:48.125ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
105042025-04-30T06:26:48.129ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c602a6-6ce0-4f7f-8739-3365917dc71f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105052025-04-30T06:26:48.129ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
105062025-04-30T06:26:48.134ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d3a2bb4-e61a-4a35-a9ba-2386d6f7f0e1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105072025-04-30T06:26:48.134ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
105082025-04-30T06:26:48.135ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1ead145-7ea4-4d98-8a79-120e4a993735", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105092025-04-30T06:26:48.136ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
105102025-04-30T06:26:48.137ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a94cad61-286c-416b-ad97-03bbf615779a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105112025-04-30T06:26:48.137ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
105122025-04-30T06:26:48.139ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4deb9b55-30ef-491e-9120-bce216203117", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105132025-04-30T06:26:48.139ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
105142025-04-30T06:26:48.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a109129e-ebf5-48cb-a775-686bc0ba012e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105152025-04-30T06:26:48.141ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105162025-04-30T06:26:48.141ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105172025-04-30T06:26:48.141ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105182025-04-30T06:26:48.141ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105192025-04-30T06:26:48.142ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105202025-04-30T06:26:48.144ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105212025-04-30T06:26:48.144ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105222025-04-30T06:26:48.144ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105232025-04-30T06:26:48.144ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105242025-04-30T06:26:48.144ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105252025-04-30T06:26:48.159ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105262025-04-30T06:26:48.180ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105272025-04-30T06:26:48.180ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105282025-04-30T06:26:48.180ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105292025-04-30T06:26:48.180ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105302025-04-30T06:26:48.180ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105312025-04-30T06:26:48.184ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105322025-04-30T06:26:48.184ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
105332025-04-30T06:26:48.184ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
105342025-04-30T06:26:48.184ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
105352025-04-30T06:26:48.186ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
105362025-04-30T06:26:48.188ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105372025-04-30T06:26:48.209ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105382025-04-30T06:26:48.210ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105392025-04-30T06:26:48.210ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105402025-04-30T06:26:48.210ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105412025-04-30T06:26:48.212ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105422025-04-30T06:26:48.218ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105432025-04-30T06:26:48.218ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105442025-04-30T06:26:48.218ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105452025-04-30T06:26:48.219ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105462025-04-30T06:26:48.221ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
105472025-04-30T06:26:48.221ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
105482025-04-30T06:26:48.225ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105492025-04-30T06:26:48.225ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105502025-04-30T06:26:48.226ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105512025-04-30T06:26:48.313ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
105522025-04-30T06:26:48.313ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
105532025-04-30T06:26:48.314ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:44483 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
105542025-04-30T06:26:48.314ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
105552025-04-30T06:26:48.314ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
105562025-04-30T06:26:48.316ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
105572025-04-30T06:26:48.317ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:44483 req_id = 4ed10d11-c50d-4fbb-9bd6-58b0ad509ac4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105582025-04-30T06:26:48.317ZINFOtest_disk_metrics (dropshot): request completed latency_us = 574 local_addr = [::1]:50989 method = GET remote_addr = [::1]:44483 req_id = 4ed10d11-c50d-4fbb-9bd6-58b0ad509ac4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105592025-04-30T06:26:48.318ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ed10d11-c50d-4fbb-9bd6-58b0ad509ac4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
105602025-04-30T06:26:48.324ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105612025-04-30T06:26:48.324ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105622025-04-30T06:26:48.324ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
105632025-04-30T06:26:48.329ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105642025-04-30T06:26:48.329ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105652025-04-30T06:26:48.329ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105662025-04-30T06:26:48.329ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105672025-04-30T06:26:48.330ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105682025-04-30T06:26:48.343ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105692025-04-30T06:26:48.348ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105702025-04-30T06:26:48.348ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105712025-04-30T06:26:48.348ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105722025-04-30T06:26:48.349ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105732025-04-30T06:26:48.406ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
105742025-04-30T06:26:48.431ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
105752025-04-30T06:26:48.431ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
105762025-04-30T06:26:48.431ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 630.09333ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
105772025-04-30T06:26:48.431ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
105782025-04-30T06:26:48.431ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105792025-04-30T06:26:48.431ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105802025-04-30T06:26:48.431ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105812025-04-30T06:26:48.431ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105822025-04-30T06:26:48.434ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105832025-04-30T06:26:48.443ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105842025-04-30T06:26:48.457ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105852025-04-30T06:26:48.457ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105862025-04-30T06:26:48.457ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105872025-04-30T06:26:48.457ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105882025-04-30T06:26:48.459ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105892025-04-30T06:26:48.459ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105902025-04-30T06:26:48.459ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105912025-04-30T06:26:48.459ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105922025-04-30T06:26:48.460ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105932025-04-30T06:26:48.463ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105942025-04-30T06:26:48.470ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105952025-04-30T06:26:48.471ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105962025-04-30T06:26:48.471ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105972025-04-30T06:26:48.472ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105982025-04-30T06:26:48.473ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
105992025-04-30T06:26:48.473ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
106002025-04-30T06:26:48.474ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: eb 78 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .x..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60588 req_id = c864caa8-e865-4d94-b5d2-f986356ddb1a
106012025-04-30T06:26:48.474ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60280,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60588 req_id = c864caa8-e865-4d94-b5d2-f986356ddb1a
106022025-04-30T06:26:48.475ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
106032025-04-30T06:26:48.476ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106042025-04-30T06:26:48.476ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60588 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c864caa8-e865-4d94-b5d2-f986356ddb1a
106052025-04-30T06:26:48.478ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3273, tv_nsec: 277012389 } })
106062025-04-30T06:26:48.478ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
106072025-04-30T06:26:48.478ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 36 bd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 6...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64807 req_id = 65c6e2af-bc2f-4aee-9915-3aee7d7147ad
106082025-04-30T06:26:48.478ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14013,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64807 req_id = 65c6e2af-bc2f-4aee-9915-3aee7d7147ad
106092025-04-30T06:26:48.478ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106102025-04-30T06:26:48.478ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64807 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 65c6e2af-bc2f-4aee-9915-3aee7d7147ad
106112025-04-30T06:26:48.602ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106122025-04-30T06:26:48.602ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106132025-04-30T06:26:48.603ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
106142025-04-30T06:26:48.605ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106152025-04-30T06:26:48.607ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 04 a9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56118 req_id = 4388f433-5d25-4392-a3f8-237e2d3bce45
106162025-04-30T06:26:48.607ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1193,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56118 req_id = 4388f433-5d25-4392-a3f8-237e2d3bce45
106172025-04-30T06:26:48.608ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
106182025-04-30T06:26:48.608ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1193, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56118 req_id = 4388f433-5d25-4392-a3f8-237e2d3bce45
106192025-04-30T06:26:48.610ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
106202025-04-30T06:26:48.610ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
106212025-04-30T06:26:48.610ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
106222025-04-30T06:26:48.614ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f8f8c8-a93c-4e55-b3e8-167abcd75687", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106232025-04-30T06:26:48.615ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
106242025-04-30T06:26:48.627ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fac934aa-16c9-43b6-a4ba-a05891bf4c77", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106252025-04-30T06:26:48.627ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
106262025-04-30T06:26:48.631ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f50ae567-3b8e-484f-bb11-ac7c2be92304", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106272025-04-30T06:26:48.631ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
106282025-04-30T06:26:48.635ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93634e97-f575-462b-a359-ef09693d7bcd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106292025-04-30T06:26:48.635ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
106302025-04-30T06:26:48.637ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db46c17d-2288-4adb-8de7-dd55609c9f6c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106312025-04-30T06:26:48.638ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
106322025-04-30T06:26:48.644ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4823b3b5-70c4-4b04-b7cf-fc7eb2dfd617", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106332025-04-30T06:26:48.644ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
106342025-04-30T06:26:48.649ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8591771-d760-4a6a-aea1-5eaad3245e50", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106352025-04-30T06:26:48.649ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
106362025-04-30T06:26:48.652ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37104950-f79f-41ae-a166-3b1bbbfcefb4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106372025-04-30T06:26:48.654ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
106382025-04-30T06:26:48.656ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0783a8c2-bb11-4df0-b08d-c9ce802ade08", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106392025-04-30T06:26:48.656ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
106402025-04-30T06:26:48.663ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
106412025-04-30T06:26:48.664ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af08d141-7ac7-40af-8c7a-ff84ef3ee473", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106422025-04-30T06:26:48.664ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
106432025-04-30T06:26:48.666ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1c80289-fe21-4cdd-94f9-599961b2dbf7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106442025-04-30T06:26:48.666ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
106452025-04-30T06:26:48.668ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "471bd419-b49d-4bab-b11b-faf7cd6bd39c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106462025-04-30T06:26:48.668ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
106472025-04-30T06:26:48.673ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b761fbc-6a6e-4486-9f57-179b09ef6404", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106482025-04-30T06:26:48.673ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
106492025-04-30T06:26:48.677ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "689817a5-9f1a-49ff-b359-cd0a6127de29", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106502025-04-30T06:26:48.678ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
106512025-04-30T06:26:48.681ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e43d6cbb-181e-4cce-8853-499d0af1dad9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106522025-04-30T06:26:48.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
106532025-04-30T06:26:48.686ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18f282f4-2221-4087-bd2f-a5990ac1fdbc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106542025-04-30T06:26:48.687ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
106552025-04-30T06:26:48.693ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db668433-b54c-4aa6-b32d-3d4604b87ded", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106562025-04-30T06:26:48.693ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
106572025-04-30T06:26:48.698ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db91e2a3-f94c-44be-95ff-f26e98e74517", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106582025-04-30T06:26:48.698ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
106592025-04-30T06:26:48.708ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a41684e6-76e4-41b9-9f2c-b58310e87bf5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106602025-04-30T06:26:48.708ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
106612025-04-30T06:26:48.710ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47759a99-256d-4982-891c-dcabea8fa3da", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106622025-04-30T06:26:48.710ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
106632025-04-30T06:26:48.714ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b8cd672-9e79-4958-a591-c451c44aaab5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106642025-04-30T06:26:48.714ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
106652025-04-30T06:26:48.714ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106662025-04-30T06:26:48.714ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106672025-04-30T06:26:48.715ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106682025-04-30T06:26:48.715ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
106692025-04-30T06:26:48.716ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106702025-04-30T06:26:48.721ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106712025-04-30T06:26:48.736ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106722025-04-30T06:26:48.736ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106732025-04-30T06:26:48.736ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
106742025-04-30T06:26:48.737ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106752025-04-30T06:26:48.737ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49c1b155-cb09-4582-b3e9-b6bbf5de6d0d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106762025-04-30T06:26:48.737ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
106772025-04-30T06:26:48.739ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af62acc-a431-4085-9a78-a16c569379a8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106782025-04-30T06:26:48.739ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
106792025-04-30T06:26:48.741ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82bac213-10bb-488a-8e70-92f99dfa49df", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106802025-04-30T06:26:48.741ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
106812025-04-30T06:26:48.743ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a94d574a-a5c5-486c-a77d-4622eada2a03", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106822025-04-30T06:26:48.743ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
106832025-04-30T06:26:48.745ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9db946-6c48-4ff3-b762-e6a27e9c7326", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106842025-04-30T06:26:48.745ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
106852025-04-30T06:26:48.747ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "852c1811-2c92-4fd4-a3f0-63dedbd66a92", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106862025-04-30T06:26:48.747ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
106872025-04-30T06:26:48.750ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "214f0bbe-ec2e-4226-b70e-66458bebfd29", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106882025-04-30T06:26:48.751ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
106892025-04-30T06:26:48.751ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106902025-04-30T06:26:48.751ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
106912025-04-30T06:26:48.753ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106922025-04-30T06:26:48.757ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ec37794-d48d-46dc-ab6e-bf15b817c99b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106932025-04-30T06:26:48.757ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
106942025-04-30T06:26:48.760ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5126cd3-657c-4bb9-8570-4c8238144cc5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106952025-04-30T06:26:48.760ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
106962025-04-30T06:26:48.761ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f4a126f-ef9f-44e1-a459-851f21a81122", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106972025-04-30T06:26:48.761ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
106982025-04-30T06:26:48.763ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03009f16-af53-4ff3-ae1e-2515efd76454", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
106992025-04-30T06:26:48.763ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
107002025-04-30T06:26:48.764ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccd711d7-5497-448f-aab1-0fa0ac9fdcbb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107012025-04-30T06:26:48.764ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
107022025-04-30T06:26:48.765ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10e0389-82ce-4523-a2b9-1be9b9cc8ea0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107032025-04-30T06:26:48.765ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
107042025-04-30T06:26:48.766ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce2e684e-92dc-45da-81fa-06cb94589aa0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107052025-04-30T06:26:48.770ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
107062025-04-30T06:26:48.773ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3f05a4c-d306-485c-9b5a-285b6e7f809c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107072025-04-30T06:26:48.773ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
107082025-04-30T06:26:48.775ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "185fc9ef-12a4-4fdf-b4eb-7b3d34bd31e0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107092025-04-30T06:26:48.779ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
107102025-04-30T06:26:48.781ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30c0161a-6882-49d7-ade9-9ae180a479d3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107112025-04-30T06:26:48.782ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
107122025-04-30T06:26:48.786ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49c92fe1-4a03-4a78-afd8-439298b119c2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107132025-04-30T06:26:48.787ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
107142025-04-30T06:26:48.788ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1397e63f-5f0b-4223-ba5d-00a9d0b4b5f9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107152025-04-30T06:26:48.789ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
107162025-04-30T06:26:48.790ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b03a05d-adb6-4970-ac90-e33d9a0c591b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107172025-04-30T06:26:48.791ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
107182025-04-30T06:26:48.792ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ce0cbab-29c0-423b-992b-54cf4b1e9bab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107192025-04-30T06:26:48.792ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
107202025-04-30T06:26:48.794ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "738f4eb9-e2af-4ecf-bd23-d613348981ba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107212025-04-30T06:26:48.794ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
107222025-04-30T06:26:48.796ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "710762db-f801-4e10-99e1-519646aa9fca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107232025-04-30T06:26:48.796ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
107242025-04-30T06:26:48.798ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8c2f346-82d2-4fab-b631-9293fbbb468a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
107252025-04-30T06:26:48.798ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107262025-04-30T06:26:48.798ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107272025-04-30T06:26:48.798ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107282025-04-30T06:26:48.798ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107292025-04-30T06:26:48.800ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107302025-04-30T06:26:48.804ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
107312025-04-30T06:26:48.804ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
107322025-04-30T06:26:48.804ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
107332025-04-30T06:26:48.805ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
107342025-04-30T06:26:48.806ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
107352025-04-30T06:26:48.806ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
107362025-04-30T06:26:48.806ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
107372025-04-30T06:26:48.806ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
107382025-04-30T06:26:48.807ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
107392025-04-30T06:26:48.808ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
107402025-04-30T06:26:48.809ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 138.697\u{b5}s iteration = 3 status = {"error":"task disabled"}
107412025-04-30T06:26:48.809ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
107422025-04-30T06:26:48.809ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
107432025-04-30T06:26:48.811ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107442025-04-30T06:26:48.811ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
107452025-04-30T06:26:48.811ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
107462025-04-30T06:26:48.812ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
107472025-04-30T06:26:48.812ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
107482025-04-30T06:26:48.812ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
107492025-04-30T06:26:48.812ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107502025-04-30T06:26:48.812ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
107512025-04-30T06:26:48.812ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107522025-04-30T06:26:48.813ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107532025-04-30T06:26:48.813ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107542025-04-30T06:26:48.822ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107552025-04-30T06:26:48.822ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107562025-04-30T06:26:48.822ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107572025-04-30T06:26:48.823ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107582025-04-30T06:26:48.823ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107592025-04-30T06:26:48.823ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107602025-04-30T06:26:48.826ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
107612025-04-30T06:26:48.826ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
107622025-04-30T06:26:48.826ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107632025-04-30T06:26:48.827ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107642025-04-30T06:26:48.832ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
107652025-04-30T06:26:48.832ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
107662025-04-30T06:26:48.832ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107672025-04-30T06:26:48.832ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
107682025-04-30T06:26:48.832ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
107692025-04-30T06:26:48.833ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
107702025-04-30T06:26:48.833ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
107712025-04-30T06:26:48.833ZINFOtest_disk_metrics (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
107722025-04-30T06:26:48.833ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = instance_updater elapsed = 63.417\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
107732025-04-30T06:26:48.833ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
107742025-04-30T06:26:48.833ZINFOtest_disk_metrics (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
107752025-04-30T06:26:48.833ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
107762025-04-30T06:26:48.833ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107772025-04-30T06:26:48.833ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
107782025-04-30T06:26:48.833ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107792025-04-30T06:26:48.833ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107802025-04-30T06:26:48.834ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
107812025-04-30T06:26:48.835ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 40 52 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 @R..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64679 req_id = 555b94b4-6f4e-42d6-902b-65ac0bd7df81
107822025-04-30T06:26:48.835ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16466,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64679 req_id = 555b94b4-6f4e-42d6-902b-65ac0bd7df81
107832025-04-30T06:26:48.835ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
107842025-04-30T06:26:48.836ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
107852025-04-30T06:26:48.836ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64679 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 555b94b4-6f4e-42d6-902b-65ac0bd7df81
107862025-04-30T06:26:48.837ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 31.642596ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
107872025-04-30T06:26:48.839ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3273, tv_nsec: 638205062 } })
107882025-04-30T06:26:48.839ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
107892025-04-30T06:26:48.842ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
107902025-04-30T06:26:48.844ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107912025-04-30T06:26:48.853ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
107922025-04-30T06:26:48.853ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
107932025-04-30T06:26:48.853ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107942025-04-30T06:26:48.854ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107952025-04-30T06:26:48.856ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107962025-04-30T06:26:48.861ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107972025-04-30T06:26:48.861ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107982025-04-30T06:26:48.861ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107992025-04-30T06:26:48.861ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108002025-04-30T06:26:48.864ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108012025-04-30T06:26:48.870ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
108022025-04-30T06:26:48.870ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108032025-04-30T06:26:48.870ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108042025-04-30T06:26:48.871ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108052025-04-30T06:26:48.877ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
108062025-04-30T06:26:48.878ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 90 02 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58099 req_id = 3b4dbfc3-fe91-45bc-9e97-55e3a77665d6
108072025-04-30T06:26:48.878ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36866,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58099 req_id = 3b4dbfc3-fe91-45bc-9e97-55e3a77665d6
108082025-04-30T06:26:48.878ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108092025-04-30T06:26:48.878ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58099 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3b4dbfc3-fe91-45bc-9e97-55e3a77665d6
108102025-04-30T06:26:48.879ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 58 25 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 X%..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:54971 req_id = ed41ae1f-22fa-40f5-a4ce-1f3291d3cc00
108112025-04-30T06:26:48.879ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22565,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54971 req_id = ed41ae1f-22fa-40f5-a4ce-1f3291d3cc00
108122025-04-30T06:26:48.879ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
108132025-04-30T06:26:48.879ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108142025-04-30T06:26:48.879ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54971 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ed41ae1f-22fa-40f5-a4ce-1f3291d3cc00
108152025-04-30T06:26:48.880ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108162025-04-30T06:26:48.880ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108172025-04-30T06:26:48.886ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
108182025-04-30T06:26:48.886ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
108192025-04-30T06:26:48.886ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108202025-04-30T06:26:48.886ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
108212025-04-30T06:26:48.886ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
108222025-04-30T06:26:48.887ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
108232025-04-30T06:26:48.887ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108242025-04-30T06:26:49.034ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
108252025-04-30T06:26:49.034ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
108262025-04-30T06:26:49.035ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108272025-04-30T06:26:49.054ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
108282025-04-30T06:26:49.054ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
108292025-04-30T06:26:49.054ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
108302025-04-30T06:26:49.055ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
108312025-04-30T06:26:49.056ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108322025-04-30T06:26:49.057ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108332025-04-30T06:26:49.057ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108342025-04-30T06:26:49.058ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
108352025-04-30T06:26:49.058ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
108362025-04-30T06:26:49.058ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
108372025-04-30T06:26:49.059ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108382025-04-30T06:26:49.062ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
108392025-04-30T06:26:49.062ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
108402025-04-30T06:26:49.062ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
108412025-04-30T06:26:49.062ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
108422025-04-30T06:26:49.063ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6c522d1e-16cd-4898-a24f-b664a79c506f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108432025-04-30T06:26:49.064ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:51232 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
108442025-04-30T06:26:49.064ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
108452025-04-30T06:26:49.138ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
108462025-04-30T06:26:49.138ZDEBGtest_disk_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
108472025-04-30T06:26:49.139ZDEBGtest_disk_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
108482025-04-30T06:26:49.139ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
108492025-04-30T06:26:49.140ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
108502025-04-30T06:26:49.141ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3273, tv_nsec: 940389425 } })
108512025-04-30T06:26:49.141ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
108522025-04-30T06:26:49.142ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108532025-04-30T06:26:49.142ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6c522d1e-16cd-4898-a24f-b664a79c506f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108542025-04-30T06:26:49.143ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6c522d1e-16cd-4898-a24f-b664a79c506f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108552025-04-30T06:26:49.143ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6c522d1e-16cd-4898-a24f-b664a79c506f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108562025-04-30T06:26:49.144ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108572025-04-30T06:26:49.150ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:51232 req_id = 3f6f3a91-32a8-4089-a1c0-e9989f2ed7b0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108582025-04-30T06:26:49.150ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 81 8d 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53403 req_id = cd5933bd-a16c-49a4-8cad-6d65baffdb7b
108592025-04-30T06:26:49.150ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33165,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53403 req_id = cd5933bd-a16c-49a4-8cad-6d65baffdb7b
108602025-04-30T06:26:49.150ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
108612025-04-30T06:26:49.151ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108622025-04-30T06:26:49.151ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53403 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cd5933bd-a16c-49a4-8cad-6d65baffdb7b
108632025-04-30T06:26:49.153ZINFOtest_disk_metrics (dropshot): request completed latency_us = 3419 local_addr = [::1]:50989 method = GET remote_addr = [::1]:51232 req_id = 3f6f3a91-32a8-4089-a1c0-e9989f2ed7b0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108642025-04-30T06:26:49.154ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
108652025-04-30T06:26:49.155ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3273, tv_nsec: 954707132 } })
108662025-04-30T06:26:49.156ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
108672025-04-30T06:26:49.156ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e7 46 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .F..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58051 req_id = b8285041-86d3-4f03-b204-dae7377912bf
108682025-04-30T06:26:49.156ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59206,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58051 req_id = b8285041-86d3-4f03-b204-dae7377912bf
108692025-04-30T06:26:49.156ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108702025-04-30T06:26:49.159ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58051 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b8285041-86d3-4f03-b204-dae7377912bf
108712025-04-30T06:26:49.159ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108722025-04-30T06:26:49.161ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108732025-04-30T06:26:49.161ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108742025-04-30T06:26:49.161ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108752025-04-30T06:26:49.161ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f6f3a91-32a8-4089-a1c0-e9989f2ed7b0", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
108762025-04-30T06:26:49.177ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108772025-04-30T06:26:49.177ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108782025-04-30T06:26:49.178ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
108792025-04-30T06:26:49.180ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108802025-04-30T06:26:49.180ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108812025-04-30T06:26:49.180ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108822025-04-30T06:26:49.180ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108832025-04-30T06:26:49.181ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108842025-04-30T06:26:49.253ZINFOtest_disk_metrics (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
108852025-04-30T06:26:49.255ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:50989/vpc-routes
108862025-04-30T06:26:49.256ZINFOtest_disk_metrics (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
108872025-04-30T06:26:49.257ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 424.583576ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
108882025-04-30T06:26:49.258ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108892025-04-30T06:26:49.258ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108902025-04-30T06:26:49.258ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108912025-04-30T06:26:49.258ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
108922025-04-30T06:26:49.260ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108932025-04-30T06:26:49.261ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:50342 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
108942025-04-30T06:26:49.262ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
108952025-04-30T06:26:49.353ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
108962025-04-30T06:26:49.353ZDEBGtest_disk_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
108972025-04-30T06:26:49.353ZDEBGtest_disk_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
108982025-04-30T06:26:49.353ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
108992025-04-30T06:26:49.373ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
109002025-04-30T06:26:49.373ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109012025-04-30T06:26:49.374ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109022025-04-30T06:26:49.374ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
109032025-04-30T06:26:49.375ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
109042025-04-30T06:26:49.389ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109052025-04-30T06:26:49.420ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109062025-04-30T06:26:49.421ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109072025-04-30T06:26:49.421ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109082025-04-30T06:26:49.422ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109092025-04-30T06:26:49.434ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 370797 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 6c522d1e-16cd-4898-a24f-b664a79c506f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
109102025-04-30T06:26:49.435ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
109112025-04-30T06:26:49.435ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109122025-04-30T06:26:49.436ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109132025-04-30T06:26:49.436ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109142025-04-30T06:26:49.438ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109152025-04-30T06:26:49.460ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:50342 req_id = 10279e84-a72b-4b1c-a4e8-e059f6efdb53 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
109162025-04-30T06:26:49.466ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d2 86 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58266 req_id = 3a1f18f3-4ea2-4b44-a9e9-452f82b3120e
109172025-04-30T06:26:49.466ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53894,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58266 req_id = 3a1f18f3-4ea2-4b44-a9e9-452f82b3120e
109182025-04-30T06:26:49.466ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109192025-04-30T06:26:49.466ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58266 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3a1f18f3-4ea2-4b44-a9e9-452f82b3120e
109202025-04-30T06:26:49.466ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c522d1e-16cd-4898-a24f-b664a79c506f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
109212025-04-30T06:26:49.467ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
109222025-04-30T06:26:49.467ZINFOtest_disk_metrics (dropshot): request completed latency_us = 5535 local_addr = [::1]:50989 method = GET remote_addr = [::1]:50342 req_id = 10279e84-a72b-4b1c-a4e8-e059f6efdb53 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
109232025-04-30T06:26:49.467ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109242025-04-30T06:26:49.472ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109252025-04-30T06:26:49.472ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109262025-04-30T06:26:49.472ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109272025-04-30T06:26:49.473ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109282025-04-30T06:26:49.473ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 34070bde-d57b-4766-9e35-a25f49bcaa23 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
109292025-04-30T06:26:49.474ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
109302025-04-30T06:26:49.474ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
109312025-04-30T06:26:49.475ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
109322025-04-30T06:26:49.556ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
109332025-04-30T06:26:49.556ZDEBGtest_disk_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
109342025-04-30T06:26:49.557ZDEBGtest_disk_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109352025-04-30T06:26:49.557ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
109362025-04-30T06:26:49.558ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
109372025-04-30T06:26:49.558ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 34070bde-d57b-4766-9e35-a25f49bcaa23 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
109382025-04-30T06:26:49.558ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 34070bde-d57b-4766-9e35-a25f49bcaa23 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
109392025-04-30T06:26:49.560ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 34070bde-d57b-4766-9e35-a25f49bcaa23 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
109402025-04-30T06:26:49.560ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:50989/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "10279e84-a72b-4b1c-a4e8-e059f6efdb53", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
109412025-04-30T06:26:49.560ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109422025-04-30T06:26:49.560ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109432025-04-30T06:26:49.560ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109442025-04-30T06:26:49.561ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109452025-04-30T06:26:49.587ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
109462025-04-30T06:26:49.587ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
109472025-04-30T06:26:49.587ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109482025-04-30T06:26:49.587ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109492025-04-30T06:26:49.587ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109502025-04-30T06:26:49.588ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109512025-04-30T06:26:49.598ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
109522025-04-30T06:26:49.610ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 136926 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 34070bde-d57b-4766-9e35-a25f49bcaa23 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
109532025-04-30T06:26:49.646ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109542025-04-30T06:26:49.654ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
109552025-04-30T06:26:49.654ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109562025-04-30T06:26:49.654ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109572025-04-30T06:26:49.655ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109582025-04-30T06:26:49.658ZINFOtest_disk_metrics (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
109592025-04-30T06:26:49.658ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:50989/eip-gateways
109602025-04-30T06:26:49.726ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:50989/v2p
109612025-04-30T06:26:49.727ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109622025-04-30T06:26:49.727ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109632025-04-30T06:26:49.729ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109642025-04-30T06:26:49.732ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:43099 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
109652025-04-30T06:26:49.732ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
109662025-04-30T06:26:49.732ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109672025-04-30T06:26:49.733ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = PUT remote_addr = [::1]:50342 req_id = 6659994d-4f69-4674-a648-35ce5b1ced3a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
109682025-04-30T06:26:49.733ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "34070bde-d57b-4766-9e35-a25f49bcaa23", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
109692025-04-30T06:26:49.733ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
109702025-04-30T06:26:49.733ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109712025-04-30T06:26:49.733ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
109722025-04-30T06:26:49.736ZINFOtest_disk_metrics (dropshot): request completed latency_us = 2978 local_addr = [::1]:50989 method = PUT remote_addr = [::1]:50342 req_id = 6659994d-4f69-4674-a648-35ce5b1ced3a response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
109732025-04-30T06:26:49.737ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:43099 req_id = 8d515cbd-fa56-43e8-b89e-e79457ba88f0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
109742025-04-30T06:26:49.742ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:50989/eip-gateways", status: 204, headers: {"x-request-id": "6659994d-4f69-4674-a648-35ce5b1ced3a", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
109752025-04-30T06:26:49.744ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 911.01573ms iteration = 2 status = {}
109762025-04-30T06:26:49.745ZINFOtest_disk_metrics (dropshot): request completed latency_us = 8417 local_addr = [::1]:50989 method = GET remote_addr = [::1]:43099 req_id = 8d515cbd-fa56-43e8-b89e-e79457ba88f0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
109772025-04-30T06:26:49.747ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:50989/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d515cbd-fa56-43e8-b89e-e79457ba88f0", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
109782025-04-30T06:26:49.747ZINFOtest_disk_metrics (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
109792025-04-30T06:26:49.747ZINFOtest_disk_metrics (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
109802025-04-30T06:26:49.747ZINFOtest_disk_metrics (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
109812025-04-30T06:26:49.762ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 935.924378ms iteration = 2 status = {}
109822025-04-30T06:26:49.781ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109832025-04-30T06:26:49.784ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109842025-04-30T06:26:49.785ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109852025-04-30T06:26:49.785ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
109862025-04-30T06:26:49.786ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
109872025-04-30T06:26:49.786ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.355255018s iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
109882025-04-30T06:26:49.786ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
109892025-04-30T06:26:49.786ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109902025-04-30T06:26:49.786ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109912025-04-30T06:26:49.786ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109922025-04-30T06:26:49.786ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109932025-04-30T06:26:49.788ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109942025-04-30T06:26:49.805ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109952025-04-30T06:26:49.813ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109962025-04-30T06:26:49.814ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109972025-04-30T06:26:49.814ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109982025-04-30T06:26:49.814ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109992025-04-30T06:26:49.815ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
110002025-04-30T06:26:49.815ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110012025-04-30T06:26:49.815ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110022025-04-30T06:26:49.815ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110032025-04-30T06:26:49.816ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110042025-04-30T06:26:49.825ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110052025-04-30T06:26:49.833ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
110062025-04-30T06:26:49.833ZERROtest_disk_metrics (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
110072025-04-30T06:26:49.833ZERROtest_disk_metrics (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
110082025-04-30T06:26:49.833ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 1.024185262s iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
110092025-04-30T06:26:49.833ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
110102025-04-30T06:26:49.833ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
110112025-04-30T06:26:49.848ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c4 8f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49607 req_id = bbbda983-ab40-431f-9e8b-1bec2e068058
110122025-04-30T06:26:49.848ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50319,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49607 req_id = bbbda983-ab40-431f-9e8b-1bec2e068058
110132025-04-30T06:26:49.848ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
110142025-04-30T06:26:49.848ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110152025-04-30T06:26:49.848ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49607 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bbbda983-ab40-431f-9e8b-1bec2e068058
110162025-04-30T06:26:49.849ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3274, tv_nsec: 648033780 } })
110172025-04-30T06:26:49.849ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
110182025-04-30T06:26:49.855ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: de 42 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .B..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57206 req_id = fdc2d43e-7537-4478-b7d5-904729a2e62d
110192025-04-30T06:26:49.855ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56898,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57206 req_id = fdc2d43e-7537-4478-b7d5-904729a2e62d
110202025-04-30T06:26:49.856ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110212025-04-30T06:26:49.856ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57206 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fdc2d43e-7537-4478-b7d5-904729a2e62d
110222025-04-30T06:26:49.921ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
110232025-04-30T06:26:49.924ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 36 d1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 6............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61990 req_id = e5b9a781-4aa1-4386-83ca-58bf669dbf5e
110242025-04-30T06:26:49.924ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14033,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61990 req_id = e5b9a781-4aa1-4386-83ca-58bf669dbf5e
110252025-04-30T06:26:49.924ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
110262025-04-30T06:26:49.925ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14033, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61990 req_id = e5b9a781-4aa1-4386-83ca-58bf669dbf5e
110272025-04-30T06:26:49.925ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
110282025-04-30T06:26:49.926ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
110292025-04-30T06:26:49.926ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
110302025-04-30T06:26:49.928ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5c57d36-a66f-4196-bfc1-b83143bed171", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110312025-04-30T06:26:49.928ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
110322025-04-30T06:26:49.940ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
110332025-04-30T06:26:49.948ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01a4f14a-ac53-40c8-a0ea-44157495ee38", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110342025-04-30T06:26:49.949ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
110352025-04-30T06:26:49.986ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110362025-04-30T06:26:49.986ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110372025-04-30T06:26:49.989ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110382025-04-30T06:26:49.990ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110392025-04-30T06:26:49.991ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110402025-04-30T06:26:49.991ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110412025-04-30T06:26:49.991ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110422025-04-30T06:26:49.991ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110432025-04-30T06:26:49.991ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110442025-04-30T06:26:49.991ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
110452025-04-30T06:26:49.992ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110462025-04-30T06:26:49.993ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "893c1255-5fb8-4e5c-8f7f-ef7badb0a4b4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110472025-04-30T06:26:49.994ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
110482025-04-30T06:26:49.995ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110492025-04-30T06:26:50.002ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110502025-04-30T06:26:50.002ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110512025-04-30T06:26:50.003ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
110522025-04-30T06:26:50.004ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110532025-04-30T06:26:50.007ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47f70673-3998-431d-8651-0baf08305889", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110542025-04-30T06:26:50.008ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
110552025-04-30T06:26:50.010ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110562025-04-30T06:26:50.010ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
110572025-04-30T06:26:50.011ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110582025-04-30T06:26:50.017ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7935080-1510-413e-8a6a-aec8472027a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110592025-04-30T06:26:50.020ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
110602025-04-30T06:26:50.027ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcb8a60a-c85e-4726-9c0b-d1e3dd8cd594", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110612025-04-30T06:26:50.027ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
110622025-04-30T06:26:50.032ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b35805b5-c2d9-41e1-8bb2-1ecf7c97ee71", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110632025-04-30T06:26:50.032ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
110642025-04-30T06:26:50.035ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bbeaae4-d434-49fa-96ab-7ae9c12247e9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110652025-04-30T06:26:50.035ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
110662025-04-30T06:26:50.036ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "464628c7-8c07-4667-888d-948a987a9d8d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110672025-04-30T06:26:50.036ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
110682025-04-30T06:26:50.038ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c12ed2-bec2-46a3-8ad5-90f953bbdae4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110692025-04-30T06:26:50.038ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
110702025-04-30T06:26:50.040ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2481349-21f3-489d-abd6-72b9b0096c19", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110712025-04-30T06:26:50.040ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
110722025-04-30T06:26:50.041ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fdff658-3bc6-4c6e-a334-31c10cd70a6a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110732025-04-30T06:26:50.041ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
110742025-04-30T06:26:50.043ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57e1fd33-f97d-40cd-a9a1-20f7d9cdf084", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110752025-04-30T06:26:50.043ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
110762025-04-30T06:26:50.048ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
110772025-04-30T06:26:50.049ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2af0830-f721-4c66-a869-b82e75691a01", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110782025-04-30T06:26:50.050ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
110792025-04-30T06:26:50.051ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "468b884f-f362-4f3d-b2ae-6b1f24e459ce", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110802025-04-30T06:26:50.052ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
110812025-04-30T06:26:50.053ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64c45f23-07c5-4fb8-a8a9-aff8177fd33e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110822025-04-30T06:26:50.054ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
110832025-04-30T06:26:50.055ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6a8bb9d-ddb2-49bf-af99-3dbd34a518c1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110842025-04-30T06:26:50.055ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
110852025-04-30T06:26:50.059ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e06ffec-4b5f-419e-8175-b95d6487b9fb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110862025-04-30T06:26:50.059ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
110872025-04-30T06:26:50.061ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "031bea5e-a0c4-4f2c-bc85-d7cb43e3b4aa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110882025-04-30T06:26:50.061ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
110892025-04-30T06:26:50.062ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ac4d10-3506-4206-98bb-701dac2b0de2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110902025-04-30T06:26:50.062ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
110912025-04-30T06:26:50.064ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d50faa0a-f56d-44c4-b1b7-feeb7649867b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
110922025-04-30T06:26:50.064ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
110932025-04-30T06:26:50.065ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7abb1c16-466d-47ed-895d-a9fc6c3dbe80", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
110942025-04-30T06:26:50.065ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
110952025-04-30T06:26:50.066ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a67edf8c-86b3-4cd3-814f-bec8862cf784", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
110962025-04-30T06:26:50.066ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
110972025-04-30T06:26:50.068ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05152d0e-11b3-438f-9cb5-2abef1fd795f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
110982025-04-30T06:26:50.068ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
110992025-04-30T06:26:50.070ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60afc1b6-4ae9-448f-8fe6-c9b9ad452cdf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111002025-04-30T06:26:50.070ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
111012025-04-30T06:26:50.072ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a23ecda-ba9a-43ba-94b6-f06d121bf5c2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111022025-04-30T06:26:50.072ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
111032025-04-30T06:26:50.074ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b28267c-50f3-44cc-863b-a28d003fb83d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111042025-04-30T06:26:50.074ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
111052025-04-30T06:26:50.078ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a99378c-8b8f-4a6d-9eeb-8a10d80be2a0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111062025-04-30T06:26:50.079ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
111072025-04-30T06:26:50.081ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcfe5e0c-06e8-4983-b767-e9f9afed7d21", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111082025-04-30T06:26:50.081ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
111092025-04-30T06:26:50.083ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ab956e8-ee46-4775-a742-e40bd9e0e954", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111102025-04-30T06:26:50.083ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
111112025-04-30T06:26:50.086ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c6ddc08-6d28-4e69-8abd-24c3a96e7dcd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111122025-04-30T06:26:50.087ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
111132025-04-30T06:26:50.089ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4783ddde-bb42-4bde-8d54-fdef20f8048e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111142025-04-30T06:26:50.089ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
111152025-04-30T06:26:50.092ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a531506-5f67-4541-ae77-41e242f6f15d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111162025-04-30T06:26:50.094ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
111172025-04-30T06:26:50.095ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c0e572d-10eb-4c75-b89b-5290f7f93040", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111182025-04-30T06:26:50.095ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
111192025-04-30T06:26:50.099ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111202025-04-30T06:26:50.100ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111212025-04-30T06:26:50.100ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111222025-04-30T06:26:50.100ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
111232025-04-30T06:26:50.101ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111242025-04-30T06:26:50.103ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c2a3621-864d-4398-b0f3-b16431e17d7f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111252025-04-30T06:26:50.103ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
111262025-04-30T06:26:50.104ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111272025-04-30T06:26:50.115ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111282025-04-30T06:26:50.115ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111292025-04-30T06:26:50.115ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
111302025-04-30T06:26:50.118ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111312025-04-30T06:26:50.123ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111322025-04-30T06:26:50.123ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
111332025-04-30T06:26:50.124ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111342025-04-30T06:26:50.124ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fec125b-1054-41e0-a246-34b26cc4f43f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111352025-04-30T06:26:50.124ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
111362025-04-30T06:26:50.129ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "973c9324-88c8-45a9-bfaa-5ebaa99deefc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111372025-04-30T06:26:50.129ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
111382025-04-30T06:26:50.136ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22c5e0c0-314c-4ad1-a4f2-2394317b261e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111392025-04-30T06:26:50.136ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
111402025-04-30T06:26:50.142ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "267b8e2e-f959-4413-803d-32e499d40160", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111412025-04-30T06:26:50.142ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
111422025-04-30T06:26:50.148ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36327bb5-3f8d-4e62-862f-950f9664964a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111432025-04-30T06:26:50.148ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
111442025-04-30T06:26:50.152ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51661534-1ce4-40f9-a51b-cb15ae912197", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111452025-04-30T06:26:50.152ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
111462025-04-30T06:26:50.153ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec21ae44-5122-4920-8293-9c800894d236", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111472025-04-30T06:26:50.153ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
111482025-04-30T06:26:50.154ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "655415eb-1a16-47b3-a63d-8f6b2bcec00b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111492025-04-30T06:26:50.154ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
111502025-04-30T06:26:50.155ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "321bb761-384e-4c5e-9dcf-f835ce0cd3d2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111512025-04-30T06:26:50.155ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
111522025-04-30T06:26:50.159ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "359dd12a-bb70-44c1-84c1-06fdc2193b27", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111532025-04-30T06:26:50.159ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
111542025-04-30T06:26:50.159ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111552025-04-30T06:26:50.159ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111562025-04-30T06:26:50.159ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111572025-04-30T06:26:50.160ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111582025-04-30T06:26:50.166ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111592025-04-30T06:26:50.174ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
111602025-04-30T06:26:50.174ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
111612025-04-30T06:26:50.174ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111622025-04-30T06:26:50.175ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111632025-04-30T06:26:50.175ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111642025-04-30T06:26:50.175ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111652025-04-30T06:26:50.177ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111662025-04-30T06:26:50.177ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111672025-04-30T06:26:50.178ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
111682025-04-30T06:26:50.180ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
111692025-04-30T06:26:50.187ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111702025-04-30T06:26:50.204ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
111712025-04-30T06:26:50.204ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111722025-04-30T06:26:50.204ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111732025-04-30T06:26:50.204ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111742025-04-30T06:26:50.205ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
111752025-04-30T06:26:50.205ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
111762025-04-30T06:26:50.206ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
111772025-04-30T06:26:50.210ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
111782025-04-30T06:26:50.210ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111792025-04-30T06:26:50.257ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
111802025-04-30T06:26:50.258ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
111812025-04-30T06:26:50.258ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111822025-04-30T06:26:50.258ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111832025-04-30T06:26:50.258ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111842025-04-30T06:26:50.258ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
111852025-04-30T06:26:50.259ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111862025-04-30T06:26:50.259ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:62194 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
111872025-04-30T06:26:50.260ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:62194 req_id = 4ed775b9-49fc-4aa3-ad1a-70b7e0a14cda sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
111882025-04-30T06:26:50.261ZINFOtest_disk_metrics (dropshot): request completed latency_us = 657 local_addr = [::1]:50989 method = GET remote_addr = [::1]:62194 req_id = 4ed775b9-49fc-4aa3-ad1a-70b7e0a14cda response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
111892025-04-30T06:26:50.261ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ed775b9-49fc-4aa3-ad1a-70b7e0a14cda", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
111902025-04-30T06:26:50.271ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
111912025-04-30T06:26:50.271ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
111922025-04-30T06:26:50.271ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
111932025-04-30T06:26:50.273ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
111942025-04-30T06:26:50.274ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111952025-04-30T06:26:50.274ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111962025-04-30T06:26:50.275ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111972025-04-30T06:26:50.275ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111982025-04-30T06:26:50.278ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111992025-04-30T06:26:50.284ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112002025-04-30T06:26:50.285ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112012025-04-30T06:26:50.285ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112022025-04-30T06:26:50.285ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112032025-04-30T06:26:50.285ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
112042025-04-30T06:26:50.286ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112052025-04-30T06:26:50.289ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112062025-04-30T06:26:50.290ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112072025-04-30T06:26:50.290ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112082025-04-30T06:26:50.290ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112092025-04-30T06:26:50.293ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112102025-04-30T06:26:50.293ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112112025-04-30T06:26:50.294ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112122025-04-30T06:26:50.338ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
112132025-04-30T06:26:50.339ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
112142025-04-30T06:26:50.339ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 552.327376ms iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
112152025-04-30T06:26:50.339ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
112162025-04-30T06:26:50.339ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112172025-04-30T06:26:50.339ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112182025-04-30T06:26:50.339ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112192025-04-30T06:26:50.339ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112202025-04-30T06:26:50.339ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112212025-04-30T06:26:50.348ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
112222025-04-30T06:26:50.349ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112232025-04-30T06:26:50.358ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112242025-04-30T06:26:50.358ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112252025-04-30T06:26:50.358ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112262025-04-30T06:26:50.359ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112272025-04-30T06:26:50.363ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
112282025-04-30T06:26:50.363ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
112292025-04-30T06:26:50.366ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ed c9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60188 req_id = 9dcb3564-7255-4f0c-b881-d4074175ab63
112302025-04-30T06:26:50.366ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60873,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60188 req_id = 9dcb3564-7255-4f0c-b881-d4074175ab63
112312025-04-30T06:26:50.366ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
112322025-04-30T06:26:50.366ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112332025-04-30T06:26:50.366ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60188 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9dcb3564-7255-4f0c-b881-d4074175ab63
112342025-04-30T06:26:50.367ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3275, tv_nsec: 166363408 } })
112352025-04-30T06:26:50.367ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
112362025-04-30T06:26:50.368ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 39 3d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 9=..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63319 req_id = c6a61146-a2e8-48b0-bba4-8b866faeac55
112372025-04-30T06:26:50.369ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14653,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63319 req_id = c6a61146-a2e8-48b0-bba4-8b866faeac55
112382025-04-30T06:26:50.370ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112392025-04-30T06:26:50.372ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63319 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c6a61146-a2e8-48b0-bba4-8b866faeac55
112402025-04-30T06:26:50.446ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
112412025-04-30T06:26:50.446ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112422025-04-30T06:26:50.446ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112432025-04-30T06:26:50.446ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112442025-04-30T06:26:50.446ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112452025-04-30T06:26:50.448ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112462025-04-30T06:26:50.448ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2d 3c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c -<..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61630 req_id = d0caddb4-51ae-491d-b344-38feda4fbd4b
112472025-04-30T06:26:50.448ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11580,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61630 req_id = d0caddb4-51ae-491d-b344-38feda4fbd4b
112482025-04-30T06:26:50.448ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
112492025-04-30T06:26:50.448ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11580, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61630 req_id = d0caddb4-51ae-491d-b344-38feda4fbd4b
112502025-04-30T06:26:50.449ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
112512025-04-30T06:26:50.449ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
112522025-04-30T06:26:50.449ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
112532025-04-30T06:26:50.452ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112542025-04-30T06:26:50.467ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112552025-04-30T06:26:50.467ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112562025-04-30T06:26:50.467ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112572025-04-30T06:26:50.468ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112582025-04-30T06:26:50.470ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1188f20f-f069-4e30-9870-f1c8ec4ef0a0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112592025-04-30T06:26:50.470ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
112602025-04-30T06:26:50.472ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112612025-04-30T06:26:50.472ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112622025-04-30T06:26:50.473ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112632025-04-30T06:26:50.475ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "208f902d-cd41-4536-bcfe-d9fcd0596faa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112642025-04-30T06:26:50.475ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
112652025-04-30T06:26:50.480ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80828ba3-1724-4e16-8c53-297b1ba3191c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112662025-04-30T06:26:50.481ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
112672025-04-30T06:26:50.483ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc515bc-0710-466c-b3d4-041aae545ab4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112682025-04-30T06:26:50.483ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
112692025-04-30T06:26:50.486ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80577953-a75f-484b-97b9-f5cfed51805b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112702025-04-30T06:26:50.486ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
112712025-04-30T06:26:50.492ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dad41e51-89a1-4891-afc0-3d15808b0704", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112722025-04-30T06:26:50.492ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
112732025-04-30T06:26:50.496ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f18a8749-de41-4b71-b165-7801d9176f7d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112742025-04-30T06:26:50.497ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
112752025-04-30T06:26:50.499ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbf69113-7cfa-4431-a446-25fda47f47e3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112762025-04-30T06:26:50.499ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
112772025-04-30T06:26:50.501ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "101dc1f7-09b3-47a8-8f4e-5065a358f3a9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112782025-04-30T06:26:50.502ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
112792025-04-30T06:26:50.506ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
112802025-04-30T06:26:50.507ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b956f220-4291-42cb-a92f-801d4643f69c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112812025-04-30T06:26:50.507ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
112822025-04-30T06:26:50.509ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e3db4bd-519d-452d-94d1-38d1ec6b0191", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112832025-04-30T06:26:50.509ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
112842025-04-30T06:26:50.511ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c62d5798-5b39-4312-8909-0347608f4bf4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112852025-04-30T06:26:50.511ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
112862025-04-30T06:26:50.512ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ad21cdd-c6df-46e1-953d-d601802c546a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112872025-04-30T06:26:50.512ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
112882025-04-30T06:26:50.514ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c91cd71-0523-404d-a224-a3f97a7b752f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112892025-04-30T06:26:50.514ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
112902025-04-30T06:26:50.516ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c47365e-a439-4e67-9543-378d619163bc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112912025-04-30T06:26:50.516ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
112922025-04-30T06:26:50.519ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "687bc218-257d-4907-b8ab-2b116d79a425", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112932025-04-30T06:26:50.519ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
112942025-04-30T06:26:50.521ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "396d0ccd-7095-4c45-98e6-5c1b49d48973", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112952025-04-30T06:26:50.521ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
112962025-04-30T06:26:50.523ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db4002e-8f8d-49bf-8556-b7fab9d1e160", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112972025-04-30T06:26:50.523ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
112982025-04-30T06:26:50.525ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bb87c8-641a-4b92-a33b-70bf9012a607", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
112992025-04-30T06:26:50.525ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
113002025-04-30T06:26:50.528ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c958d44-2515-4864-b025-5017233504f2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113012025-04-30T06:26:50.528ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
113022025-04-30T06:26:50.529ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b3854a0-359f-4cce-8ec2-dbb3064cb9a0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113032025-04-30T06:26:50.529ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
113042025-04-30T06:26:50.530ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2667c32-891a-4de0-8f7f-e7b82d427730", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113052025-04-30T06:26:50.530ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
113062025-04-30T06:26:50.531ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a972554-6f12-4245-9bd5-02f57de9226e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113072025-04-30T06:26:50.531ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
113082025-04-30T06:26:50.534ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "241ac2e2-879b-44b9-aed7-78e49adaafde", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113092025-04-30T06:26:50.534ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
113102025-04-30T06:26:50.536ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a40a0ecc-b89e-4045-a8cb-b229d29ce224", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113112025-04-30T06:26:50.536ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
113122025-04-30T06:26:50.540ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfe98081-3658-4795-ae87-d005e17e0f9c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113132025-04-30T06:26:50.540ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
113142025-04-30T06:26:50.547ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4949220-5b10-4e2e-9223-12b2bffd3710", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113152025-04-30T06:26:50.547ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
113162025-04-30T06:26:50.551ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7917035a-e5fe-4876-8520-7c941feab9c7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113172025-04-30T06:26:50.551ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
113182025-04-30T06:26:50.553ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1bf082-c135-46dd-a461-8c73f03d5e2a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113192025-04-30T06:26:50.556ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
113202025-04-30T06:26:50.558ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113212025-04-30T06:26:50.558ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113222025-04-30T06:26:50.558ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113232025-04-30T06:26:50.558ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
113242025-04-30T06:26:50.570ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113252025-04-30T06:26:50.580ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113262025-04-30T06:26:50.598ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113272025-04-30T06:26:50.598ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113282025-04-30T06:26:50.598ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
113292025-04-30T06:26:50.599ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113302025-04-30T06:26:50.601ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca408fc2-f1e2-4574-baf4-264b3c396859", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113312025-04-30T06:26:50.603ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
113322025-04-30T06:26:50.603ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113332025-04-30T06:26:50.603ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
113342025-04-30T06:26:50.604ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113352025-04-30T06:26:50.606ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1308fe5-3cd3-4db3-b0ea-896e0b80b403", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113362025-04-30T06:26:50.606ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
113372025-04-30T06:26:50.609ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33398246-1b1e-4b21-aa0d-2b8069bd6318", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113382025-04-30T06:26:50.609ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
113392025-04-30T06:26:50.610ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ef0670-513a-439a-93f1-f49764d03ffa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113402025-04-30T06:26:50.611ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
113412025-04-30T06:26:50.612ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b78082d-818d-4f36-b644-91cb8b38242d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113422025-04-30T06:26:50.612ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
113432025-04-30T06:26:50.622ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd5d26ba-0bf9-4772-99a2-2e9ba73f090a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113442025-04-30T06:26:50.622ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
113452025-04-30T06:26:50.625ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc3b9131-e28f-42f9-9287-e138ffd7f46a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113462025-04-30T06:26:50.625ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
113472025-04-30T06:26:50.627ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc0dd11-2afd-4ed1-b802-64868daa650e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113482025-04-30T06:26:50.627ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
113492025-04-30T06:26:50.629ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9697141e-529e-44e0-96a0-3e6babf3c938", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113502025-04-30T06:26:50.629ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
113512025-04-30T06:26:50.633ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db28925b-106d-4404-b79d-d4f0d3f5b0bf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113522025-04-30T06:26:50.633ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
113532025-04-30T06:26:50.638ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87127467-2ba5-45d3-bee3-4a3ff5e7e681", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113542025-04-30T06:26:50.638ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
113552025-04-30T06:26:50.640ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dfadaf1-50d1-4774-9064-90d61345d3a2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113562025-04-30T06:26:50.641ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
113572025-04-30T06:26:50.643ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "befb6a38-264c-4345-8a38-d43c701d746b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113582025-04-30T06:26:50.644ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
113592025-04-30T06:26:50.645ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
113602025-04-30T06:26:50.645ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f13bf5f8-62c1-4288-8324-1dd67e965543", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113612025-04-30T06:26:50.645ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
113622025-04-30T06:26:50.647ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d985ed88-d85a-46fb-9d40-34e995e9e267", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113632025-04-30T06:26:50.647ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
113642025-04-30T06:26:50.648ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c738cb5-ea09-4e75-9d1a-d83adfc3b2e0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113652025-04-30T06:26:50.648ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113662025-04-30T06:26:50.649ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113672025-04-30T06:26:50.649ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113682025-04-30T06:26:50.649ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113692025-04-30T06:26:50.650ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113702025-04-30T06:26:50.652ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113712025-04-30T06:26:50.656ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113722025-04-30T06:26:50.656ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113732025-04-30T06:26:50.656ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113742025-04-30T06:26:50.656ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113752025-04-30T06:26:50.656ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113762025-04-30T06:26:50.657ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113772025-04-30T06:26:50.660ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113782025-04-30T06:26:50.663ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113792025-04-30T06:26:50.664ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113802025-04-30T06:26:50.664ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113812025-04-30T06:26:50.664ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113822025-04-30T06:26:50.753ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
113832025-04-30T06:26:50.754ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
113842025-04-30T06:26:50.754ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113852025-04-30T06:26:50.755ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113862025-04-30T06:26:50.755ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113872025-04-30T06:26:50.755ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
113882025-04-30T06:26:50.758ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113892025-04-30T06:26:50.758ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:48972 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113902025-04-30T06:26:50.759ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:48972 req_id = f100f8fa-9125-486d-b2d2-a118387cfb55 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113912025-04-30T06:26:50.761ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1218 local_addr = [::1]:50989 method = GET remote_addr = [::1]:48972 req_id = f100f8fa-9125-486d-b2d2-a118387cfb55 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113922025-04-30T06:26:50.761ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f100f8fa-9125-486d-b2d2-a118387cfb55", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
113932025-04-30T06:26:50.767ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113942025-04-30T06:26:50.767ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113952025-04-30T06:26:50.767ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
113962025-04-30T06:26:50.772ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113972025-04-30T06:26:50.772ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113982025-04-30T06:26:50.772ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113992025-04-30T06:26:50.772ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114002025-04-30T06:26:50.772ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114012025-04-30T06:26:50.773ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114022025-04-30T06:26:50.799ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114032025-04-30T06:26:50.800ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114042025-04-30T06:26:50.800ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114052025-04-30T06:26:50.800ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114062025-04-30T06:26:50.804ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114072025-04-30T06:26:50.809ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114082025-04-30T06:26:50.809ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114092025-04-30T06:26:50.809ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114102025-04-30T06:26:50.810ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114112025-04-30T06:26:50.811ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114122025-04-30T06:26:50.811ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114132025-04-30T06:26:50.812ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114142025-04-30T06:26:50.850ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
114152025-04-30T06:26:50.850ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
114162025-04-30T06:26:50.851ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
114172025-04-30T06:26:50.851ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 512.056589ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
114182025-04-30T06:26:50.851ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
114192025-04-30T06:26:50.851ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114202025-04-30T06:26:50.851ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114212025-04-30T06:26:50.851ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114222025-04-30T06:26:50.851ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114232025-04-30T06:26:50.852ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114242025-04-30T06:26:50.857ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
114252025-04-30T06:26:50.858ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114262025-04-30T06:26:50.859ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114272025-04-30T06:26:50.865ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114282025-04-30T06:26:50.865ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114292025-04-30T06:26:50.865ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114302025-04-30T06:26:50.865ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114312025-04-30T06:26:50.866ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 049813ed-cb45-4119-b537-6d936e68948a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114322025-04-30T06:26:50.867ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 049813ed-cb45-4119-b537-6d936e68948a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114332025-04-30T06:26:50.867ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 049813ed-cb45-4119-b537-6d936e68948a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114342025-04-30T06:26:50.867ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 049813ed-cb45-4119-b537-6d936e68948a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114352025-04-30T06:26:50.871ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
114362025-04-30T06:26:50.871ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
114372025-04-30T06:26:50.872ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 6123 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 049813ed-cb45-4119-b537-6d936e68948a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114382025-04-30T06:26:50.873ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "049813ed-cb45-4119-b537-6d936e68948a", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
114392025-04-30T06:26:50.873ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
114402025-04-30T06:26:50.873ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 58a9bd38-a4ac-409c-bdab-8cb122842ef7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
114412025-04-30T06:26:50.874ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 91 ab 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54730 req_id = 7fe5cc15-9a8e-405b-8668-08255f16bc84
114422025-04-30T06:26:50.874ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37291,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54730 req_id = 7fe5cc15-9a8e-405b-8668-08255f16bc84
114432025-04-30T06:26:50.874ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
114442025-04-30T06:26:50.874ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
114452025-04-30T06:26:50.875ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54730 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7fe5cc15-9a8e-405b-8668-08255f16bc84
114462025-04-30T06:26:50.875ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 58a9bd38-a4ac-409c-bdab-8cb122842ef7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
114472025-04-30T06:26:50.875ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 58a9bd38-a4ac-409c-bdab-8cb122842ef7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
114482025-04-30T06:26:50.876ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 58a9bd38-a4ac-409c-bdab-8cb122842ef7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
114492025-04-30T06:26:50.877ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3275, tv_nsec: 676060476 } })
114502025-04-30T06:26:50.877ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
114512025-04-30T06:26:50.877ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 45 3e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e E>..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52977 req_id = 301fdda5-bbee-4463-9a26-ee7b71980eaa
114522025-04-30T06:26:50.878ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17726,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52977 req_id = 301fdda5-bbee-4463-9a26-ee7b71980eaa
114532025-04-30T06:26:50.878ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
114542025-04-30T06:26:50.879ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52977 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 301fdda5-bbee-4463-9a26-ee7b71980eaa
114552025-04-30T06:26:50.879ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 5363 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 58a9bd38-a4ac-409c-bdab-8cb122842ef7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
114562025-04-30T06:26:50.963ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
114572025-04-30T06:26:50.964ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a9bd38-a4ac-409c-bdab-8cb122842ef7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
114582025-04-30T06:26:50.964ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114592025-04-30T06:26:50.964ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114602025-04-30T06:26:50.964ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
114612025-04-30T06:26:50.965ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114622025-04-30T06:26:50.965ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114632025-04-30T06:26:50.965ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114642025-04-30T06:26:50.965ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114652025-04-30T06:26:50.968ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114662025-04-30T06:26:50.968ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114672025-04-30T06:26:50.968ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114682025-04-30T06:26:50.970ZTRACtest_disk_metrics (dropshot): incoming request local_addr = 127.0.0.1:39352 method = GET remote_addr = 127.0.0.1:51857 req_id = b8f17eec-7151-43b9-8e62-6d15eeaa18cd uri = /06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114692025-04-30T06:26:50.970ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e6 4b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .K..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50339 req_id = 25ef0661-d758-4749-915c-6b6fc53ba9b8
114702025-04-30T06:26:50.970ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58955,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50339 req_id = 25ef0661-d758-4749-915c-6b6fc53ba9b8
114712025-04-30T06:26:50.971ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
114722025-04-30T06:26:50.971ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58955, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50339 req_id = 25ef0661-d758-4749-915c-6b6fc53ba9b8
114732025-04-30T06:26:50.976ZINFOtest_disk_metrics (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 6003 local_addr = 127.0.0.1:39352 method = GET remote_addr = 127.0.0.1:51857 req_id = b8f17eec-7151-43b9-8e62-6d15eeaa18cd response_code = 200 uri = /06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114742025-04-30T06:26:50.977ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
114752025-04-30T06:26:50.977ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
114762025-04-30T06:26:50.977ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
114772025-04-30T06:26:50.978ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114782025-04-30T06:26:50.984ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114792025-04-30T06:26:50.984ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114802025-04-30T06:26:50.984ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
114812025-04-30T06:26:50.985ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114822025-04-30T06:26:50.991ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114832025-04-30T06:26:50.991ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114842025-04-30T06:26:50.991ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114852025-04-30T06:26:50.991ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114862025-04-30T06:26:50.993ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114872025-04-30T06:26:50.993ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114882025-04-30T06:26:51.003ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114892025-04-30T06:26:51.003ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114902025-04-30T06:26:51.003ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114912025-04-30T06:26:51.004ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114922025-04-30T06:26:51.004ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114932025-04-30T06:26:51.004ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114942025-04-30T06:26:51.006ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114952025-04-30T06:26:51.014ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "16d122d2-01bc-4859-858f-0e15f66a5517", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
114962025-04-30T06:26:51.015ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
114972025-04-30T06:26:51.031ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa717527-05e8-4794-80eb-2383e0377f25", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
114982025-04-30T06:26:51.031ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
114992025-04-30T06:26:51.044ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06c68b5e-623b-4306-8407-759ce39cdb66", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
115002025-04-30T06:26:51.046ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
115012025-04-30T06:26:51.049ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce6fe93-82bf-4266-a619-2859f6c7affc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
115022025-04-30T06:26:51.049ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
115032025-04-30T06:26:51.052ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98ebf97c-b5a7-4f6f-9e3c-83b032bf38e2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
115042025-04-30T06:26:51.052ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
115052025-04-30T06:26:51.065ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
115062025-04-30T06:26:51.066ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d64d91-988f-4454-98e0-787f15a16447", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115072025-04-30T06:26:51.066ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
115082025-04-30T06:26:51.070ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ced10d-c0ee-40a3-ac32-9fadc3d071fe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115092025-04-30T06:26:51.071ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
115102025-04-30T06:26:51.074ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c0b286f-fffa-41ca-8633-6b2747bd6a73", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115112025-04-30T06:26:51.074ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
115122025-04-30T06:26:51.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "954b14d3-3cc5-4f5c-bd92-1218fb47763c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115132025-04-30T06:26:51.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
115142025-04-30T06:26:51.077ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8af1838-4127-499b-bf74-843cc1f9d75b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115152025-04-30T06:26:51.078ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
115162025-04-30T06:26:51.079ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "baa5b1d4-4797-484c-bb3b-b32952cc32dc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115172025-04-30T06:26:51.079ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
115182025-04-30T06:26:51.081ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3282f5-4a61-48b9-b8e1-95f948a575ce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115192025-04-30T06:26:51.081ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
115202025-04-30T06:26:51.082ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "210f8c3c-1bbc-46b2-a40d-be6a5c0fd30f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115212025-04-30T06:26:51.083ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
115222025-04-30T06:26:51.087ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d84e859-b1c7-4403-9572-1e7a9ff099d1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115232025-04-30T06:26:51.088ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
115242025-04-30T06:26:51.089ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4f8cd58-f2a0-475c-9fbd-81b02363eb44", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115252025-04-30T06:26:51.090ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
115262025-04-30T06:26:51.096ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb8db039-8370-4ea1-923e-44353ebca379", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115272025-04-30T06:26:51.096ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
115282025-04-30T06:26:51.100ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9111858-3b55-4d57-9ed7-b7bd2500a92c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115292025-04-30T06:26:51.100ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
115302025-04-30T06:26:51.102ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4292ac04-1dc8-484b-8350-c9b6e2a737a7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115312025-04-30T06:26:51.102ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
115322025-04-30T06:26:51.105ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0ab6a27-07c5-42db-93fe-189db12d1b48", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115332025-04-30T06:26:51.105ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
115342025-04-30T06:26:51.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5397d69e-65e2-4379-a7c9-1dbda56f37b8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115352025-04-30T06:26:51.108ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
115362025-04-30T06:26:51.108ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3521a589-7003-4c8a-9386-72c0d358582d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115372025-04-30T06:26:51.109ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
115382025-04-30T06:26:51.112ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e163035-8bc5-48ea-92dc-f2ef2b38486f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115392025-04-30T06:26:51.112ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
115402025-04-30T06:26:51.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb4893df-880c-47ba-8790-9ac63625febb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115412025-04-30T06:26:51.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
115422025-04-30T06:26:51.116ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aeeb184-caa2-482a-a3c4-9585281218b5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115432025-04-30T06:26:51.116ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
115442025-04-30T06:26:51.116ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115452025-04-30T06:26:51.117ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115462025-04-30T06:26:51.117ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115472025-04-30T06:26:51.117ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115482025-04-30T06:26:51.118ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115492025-04-30T06:26:51.121ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115502025-04-30T06:26:51.130ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115512025-04-30T06:26:51.130ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115522025-04-30T06:26:51.130ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115532025-04-30T06:26:51.133ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115542025-04-30T06:26:51.134ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b750b8b4-b1c6-4238-a491-edcdcbac9f7c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115552025-04-30T06:26:51.134ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
115562025-04-30T06:26:51.138ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115572025-04-30T06:26:51.138ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115582025-04-30T06:26:51.139ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115592025-04-30T06:26:51.140ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "809a4ffa-8c5a-443f-945d-cbaf6f4b99c7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115602025-04-30T06:26:51.141ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
115612025-04-30T06:26:51.141ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
115622025-04-30T06:26:51.142ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 74 7b 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 t{..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53667 req_id = a222a761-d4d0-4f4e-8619-8b12586de3b1
115632025-04-30T06:26:51.143ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29819,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53667 req_id = a222a761-d4d0-4f4e-8619-8b12586de3b1
115642025-04-30T06:26:51.143ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
115652025-04-30T06:26:51.143ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115662025-04-30T06:26:51.144ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53667 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a222a761-d4d0-4f4e-8619-8b12586de3b1
115672025-04-30T06:26:51.144ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3275, tv_nsec: 943652175 } })
115682025-04-30T06:26:51.145ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
115692025-04-30T06:26:51.146ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 03 73 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .s..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49876 req_id = bf09ad26-edf4-49a2-af91-54531760eae0
115702025-04-30T06:26:51.146ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 883,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49876 req_id = bf09ad26-edf4-49a2-af91-54531760eae0
115712025-04-30T06:26:51.146ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115722025-04-30T06:26:51.146ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49876 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bf09ad26-edf4-49a2-af91-54531760eae0
115732025-04-30T06:26:51.147ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
115742025-04-30T06:26:51.205ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
115752025-04-30T06:26:51.205ZDEBGtest_disk_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
115762025-04-30T06:26:51.206ZDEBGtest_disk_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
115772025-04-30T06:26:51.206ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
115782025-04-30T06:26:51.218ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
115792025-04-30T06:26:51.218ZDEBGtest_disk_metrics (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115802025-04-30T06:26:51.220ZDEBGtest_disk_metrics (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115812025-04-30T06:26:51.221ZDEBGtest_disk_metrics (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
115822025-04-30T06:26:51.226ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
115832025-04-30T06:26:51.230ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4bbdba8-c62b-40bc-9cec-cbb74649482a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115842025-04-30T06:26:51.230ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
115852025-04-30T06:26:51.231ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 12 table_name = fields_string
115862025-04-30T06:26:51.231ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
115872025-04-30T06:26:51.235ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 5 table_name = fields_u16
115882025-04-30T06:26:51.235ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
115892025-04-30T06:26:51.237ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36f58687-2409-409f-a997-512339ad22c3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115902025-04-30T06:26:51.237ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
115912025-04-30T06:26:51.239ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 5 table_name = fields_uuid
115922025-04-30T06:26:51.240ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
115932025-04-30T06:26:51.245ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 5 table_name = measurements_histogramu64
115942025-04-30T06:26:51.245ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
115952025-04-30T06:26:51.249ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
115962025-04-30T06:26:51.249ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115972025-04-30T06:26:51.252ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38565b6a-7cb0-4ddc-b732-f19edfd035d3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
115982025-04-30T06:26:51.252ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
115992025-04-30T06:26:51.255ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
116002025-04-30T06:26:51.259ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "262fdddc-995a-4206-b7e0-b4da61393093", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116012025-04-30T06:26:51.259ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
116022025-04-30T06:26:51.260ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6a3c0b6-44e5-4bfa-8bc3-fc87b9f89417", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116032025-04-30T06:26:51.260ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
116042025-04-30T06:26:51.266ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7b2edbc-465f-4f9a-9cad-c11d91d13bd9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116052025-04-30T06:26:51.267ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
116062025-04-30T06:26:51.269ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "138e358b-0375-4a79-8499-d21a0aaffc36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116072025-04-30T06:26:51.269ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
116082025-04-30T06:26:51.272ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebfeb815-c496-47e2-9099-e3af593d8910", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116092025-04-30T06:26:51.273ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
116102025-04-30T06:26:51.276ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "459d9dff-a96d-4cbd-af80-4301b6f99330", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116112025-04-30T06:26:51.277ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
116122025-04-30T06:26:51.281ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f231601-1779-4697-846b-b8c299ce6fe4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116132025-04-30T06:26:51.282ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
116142025-04-30T06:26:51.285ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6b445a1-fa85-4b1d-9170-ddf337215fec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116152025-04-30T06:26:51.285ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
116162025-04-30T06:26:51.290ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8624c94d-ca2e-42da-9833-e85a2310e328", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116172025-04-30T06:26:51.291ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
116182025-04-30T06:26:51.294ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d09c6615-e23f-4e4d-b0f3-8b43200ea976", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116192025-04-30T06:26:51.294ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
116202025-04-30T06:26:51.297ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e47e11-f877-4798-8814-693a1b47a0b3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116212025-04-30T06:26:51.297ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
116222025-04-30T06:26:51.302ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1184e710-2d7f-474f-a15e-24d18a187ede", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116232025-04-30T06:26:51.302ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
116242025-04-30T06:26:51.303ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48bebba-6933-4bab-b698-26b0d6e399f1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116252025-04-30T06:26:51.304ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
116262025-04-30T06:26:51.305ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2059f702-1ad8-4974-aa8c-003fe7823ee2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116272025-04-30T06:26:51.305ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
116282025-04-30T06:26:51.306ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116292025-04-30T06:26:51.306ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116302025-04-30T06:26:51.306ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116312025-04-30T06:26:51.306ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
116322025-04-30T06:26:51.307ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116332025-04-30T06:26:51.308ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41752cf1-7b8a-4795-b552-080a2a71b988", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116342025-04-30T06:26:51.308ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
116352025-04-30T06:26:51.311ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21de5391-61e8-4feb-a0d9-48a07da54817", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116362025-04-30T06:26:51.312ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
116372025-04-30T06:26:51.312ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116382025-04-30T06:26:51.312ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116392025-04-30T06:26:51.312ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116402025-04-30T06:26:51.313ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116412025-04-30T06:26:51.313ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116422025-04-30T06:26:51.319ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116432025-04-30T06:26:51.319ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116442025-04-30T06:26:51.319ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
116452025-04-30T06:26:51.319ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116462025-04-30T06:26:51.321ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116472025-04-30T06:26:51.326ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
116482025-04-30T06:26:51.326ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
116492025-04-30T06:26:51.326ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116502025-04-30T06:26:51.326ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116512025-04-30T06:26:51.326ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116522025-04-30T06:26:51.327ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116532025-04-30T06:26:51.327ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
116542025-04-30T06:26:51.328ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116552025-04-30T06:26:51.328ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
116562025-04-30T06:26:51.328ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116572025-04-30T06:26:51.330ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116582025-04-30T06:26:51.334ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
116592025-04-30T06:26:51.334ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116602025-04-30T06:26:51.334ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116612025-04-30T06:26:51.335ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116622025-04-30T06:26:51.411ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
116632025-04-30T06:26:51.411ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
116642025-04-30T06:26:51.412ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:57102 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
116652025-04-30T06:26:51.412ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
116662025-04-30T06:26:51.413ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:57102 req_id = 483775b5-c3d1-4add-bbb2-4039a371703d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
116672025-04-30T06:26:51.414ZINFOtest_disk_metrics (dropshot): request completed latency_us = 957 local_addr = [::1]:50989 method = GET remote_addr = [::1]:57102 req_id = 483775b5-c3d1-4add-bbb2-4039a371703d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
116682025-04-30T06:26:51.415ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 6f 41 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 oA..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56194 req_id = 22c9a6a5-50e8-4309-baa9-bac87e3d016b
116692025-04-30T06:26:51.418ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28481,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56194 req_id = 22c9a6a5-50e8-4309-baa9-bac87e3d016b
116702025-04-30T06:26:51.418ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
116712025-04-30T06:26:51.418ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116722025-04-30T06:26:51.418ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56194 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 22c9a6a5-50e8-4309-baa9-bac87e3d016b
116732025-04-30T06:26:51.429ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "483775b5-c3d1-4add-bbb2-4039a371703d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
116742025-04-30T06:26:51.436ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
116752025-04-30T06:26:51.436ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
116762025-04-30T06:26:51.436ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
116772025-04-30T06:26:51.438ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
116782025-04-30T06:26:51.438ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116792025-04-30T06:26:51.438ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116802025-04-30T06:26:51.438ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116812025-04-30T06:26:51.439ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116822025-04-30T06:26:51.440ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3276, tv_nsec: 238898164 } })
116832025-04-30T06:26:51.441ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
116842025-04-30T06:26:51.441ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 72 b9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e r...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51313 req_id = a9bdfe54-1101-479a-9382-93a18496c21b
116852025-04-30T06:26:51.441ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29369,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51313 req_id = a9bdfe54-1101-479a-9382-93a18496c21b
116862025-04-30T06:26:51.442ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116872025-04-30T06:26:51.442ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51313 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a9bdfe54-1101-479a-9382-93a18496c21b
116882025-04-30T06:26:51.442ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
116892025-04-30T06:26:51.500ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
116902025-04-30T06:26:51.500ZDEBGtest_disk_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
116912025-04-30T06:26:51.500ZDEBGtest_disk_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116922025-04-30T06:26:51.500ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116932025-04-30T06:26:51.500ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
116942025-04-30T06:26:51.501ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116952025-04-30T06:26:51.504ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
116962025-04-30T06:26:51.505ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116972025-04-30T06:26:51.505ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116982025-04-30T06:26:51.505ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116992025-04-30T06:26:51.562ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
117002025-04-30T06:26:51.563ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
117012025-04-30T06:26:51.564ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 3f 15 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ?............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56425 req_id = 3185905f-128b-46a5-afde-ddc5e576e0fd
117022025-04-30T06:26:51.565ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16149,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56425 req_id = 3185905f-128b-46a5-afde-ddc5e576e0fd
117032025-04-30T06:26:51.565ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
117042025-04-30T06:26:51.565ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117052025-04-30T06:26:51.565ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56425 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3185905f-128b-46a5-afde-ddc5e576e0fd
117062025-04-30T06:26:51.566ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3276, tv_nsec: 365134271 } })
117072025-04-30T06:26:51.566ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
117082025-04-30T06:26:51.566ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 11 19 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63224 req_id = 11e976dc-e725-407c-bf53-0999a1c50fa5
117092025-04-30T06:26:51.566ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4377,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63224 req_id = 11e976dc-e725-407c-bf53-0999a1c50fa5
117102025-04-30T06:26:51.566ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117112025-04-30T06:26:51.566ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63224 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 11e976dc-e725-407c-bf53-0999a1c50fa5
117122025-04-30T06:26:51.567ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
117132025-04-30T06:26:51.636ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
117142025-04-30T06:26:51.636ZDEBGtest_disk_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
117152025-04-30T06:26:51.637ZDEBGtest_disk_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
117162025-04-30T06:26:51.637ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
117172025-04-30T06:26:51.638ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
117182025-04-30T06:26:51.638ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117192025-04-30T06:26:51.638ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117202025-04-30T06:26:51.638ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117212025-04-30T06:26:51.639ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117222025-04-30T06:26:51.639ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117232025-04-30T06:26:51.642ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117242025-04-30T06:26:51.649ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117252025-04-30T06:26:51.649ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117262025-04-30T06:26:51.649ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117272025-04-30T06:26:51.650ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117282025-04-30T06:26:51.656ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
117292025-04-30T06:26:51.656ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
117302025-04-30T06:26:51.656ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 805.550297ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
117312025-04-30T06:26:51.656ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
117322025-04-30T06:26:51.657ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117332025-04-30T06:26:51.657ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117342025-04-30T06:26:51.657ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117352025-04-30T06:26:51.657ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117362025-04-30T06:26:51.657ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117372025-04-30T06:26:51.660ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117382025-04-30T06:26:51.666ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117392025-04-30T06:26:51.666ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117402025-04-30T06:26:51.666ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117412025-04-30T06:26:51.668ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117422025-04-30T06:26:51.668ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117432025-04-30T06:26:51.668ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117442025-04-30T06:26:51.669ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117452025-04-30T06:26:51.673ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
117462025-04-30T06:26:51.673ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
117472025-04-30T06:26:51.674ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 11 ee 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55769 req_id = ae225b4c-3f76-4968-becb-e6730640f76f
117482025-04-30T06:26:51.674ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4590,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55769 req_id = ae225b4c-3f76-4968-becb-e6730640f76f
117492025-04-30T06:26:51.675ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
117502025-04-30T06:26:51.675ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117512025-04-30T06:26:51.675ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55769 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ae225b4c-3f76-4968-becb-e6730640f76f
117522025-04-30T06:26:51.678ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3276, tv_nsec: 476888734 } })
117532025-04-30T06:26:51.678ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
117542025-04-30T06:26:51.678ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 82 30 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .0..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51721 req_id = 59ac0284-4f59-48d3-ae66-59d389884842
117552025-04-30T06:26:51.679ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33328,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51721 req_id = 59ac0284-4f59-48d3-ae66-59d389884842
117562025-04-30T06:26:51.679ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117572025-04-30T06:26:51.679ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51721 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 59ac0284-4f59-48d3-ae66-59d389884842
117582025-04-30T06:26:51.753ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
117592025-04-30T06:26:51.755ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e0 a7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63274 req_id = f46ef17f-dcfe-42d6-9bf3-b47f35160468
117602025-04-30T06:26:51.755ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63274 req_id = f46ef17f-dcfe-42d6-9bf3-b47f35160468
117612025-04-30T06:26:51.756ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
117622025-04-30T06:26:51.756ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57511, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63274 req_id = f46ef17f-dcfe-42d6-9bf3-b47f35160468
117632025-04-30T06:26:51.756ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
117642025-04-30T06:26:51.756ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
117652025-04-30T06:26:51.756ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
117662025-04-30T06:26:51.759ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "598fd7fb-0834-4c2e-bd0f-395e7719c08e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117672025-04-30T06:26:51.760ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
117682025-04-30T06:26:51.761ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "933fda57-ef28-4624-a8ee-8690a2cebc34", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117692025-04-30T06:26:51.762ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
117702025-04-30T06:26:51.767ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da2f0335-855c-4221-b943-1b14ed516381", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117712025-04-30T06:26:51.767ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
117722025-04-30T06:26:51.773ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99042ee-d2bd-4db7-9e11-174aefa3ca34", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117732025-04-30T06:26:51.773ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
117742025-04-30T06:26:51.777ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
117752025-04-30T06:26:51.779ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f66e3b97-60b8-426e-be2d-d582775991b3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117762025-04-30T06:26:51.779ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
117772025-04-30T06:26:51.781ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec67e125-4093-4be6-bfd4-3336cd7f8b00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117782025-04-30T06:26:51.782ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
117792025-04-30T06:26:51.783ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8dbaeb0-05f3-4d65-aa22-953f410118bf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117802025-04-30T06:26:51.783ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
117812025-04-30T06:26:51.785ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a56445ef-21a2-4374-aaaa-95779bd8af24", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117822025-04-30T06:26:51.786ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
117832025-04-30T06:26:51.787ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "108805c3-41d2-4353-88e7-f7bcc9b4c7ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117842025-04-30T06:26:51.787ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
117852025-04-30T06:26:51.789ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7189b1a9-529f-4810-a4c5-053b98b58eae", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117862025-04-30T06:26:51.789ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
117872025-04-30T06:26:51.790ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce9d0ff-c2fb-4c9b-9767-61817e9c545e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117882025-04-30T06:26:51.790ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
117892025-04-30T06:26:51.792ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9ad243-25b6-4ac7-a0fe-2f73f7ca2e43", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117902025-04-30T06:26:51.792ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
117912025-04-30T06:26:51.793ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d81f1e98-4c49-46bd-ad2b-52af8a9a374c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117922025-04-30T06:26:51.793ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
117932025-04-30T06:26:51.796ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c8f707f-828b-44d5-85cc-8490016c6742", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117942025-04-30T06:26:51.796ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
117952025-04-30T06:26:51.800ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "887e5985-cfb0-4bd8-b179-5ca72ee52959", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117962025-04-30T06:26:51.800ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
117972025-04-30T06:26:51.802ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e98cb26-f3df-466b-8b02-1852f0f1df7e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
117982025-04-30T06:26:51.802ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
117992025-04-30T06:26:51.804ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96d864fe-d6aa-4ef4-8f9d-317bd38f128f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118002025-04-30T06:26:51.804ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
118012025-04-30T06:26:51.806ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a980eb5-69a3-4002-9700-cc7ee06febcf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118022025-04-30T06:26:51.806ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
118032025-04-30T06:26:51.808ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e6ab2ec-b16f-4cd8-901f-a29eea3dc843", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118042025-04-30T06:26:51.808ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
118052025-04-30T06:26:51.809ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b29b8bfc-6ad4-47ac-bafc-d2c94754610f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118062025-04-30T06:26:51.809ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
118072025-04-30T06:26:51.810ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd006157-3d61-4ac8-8fae-10e19b7bd6ed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118082025-04-30T06:26:51.811ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
118092025-04-30T06:26:51.812ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5698106a-ff40-4334-a529-5d9ba10ccc4a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118102025-04-30T06:26:51.812ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
118112025-04-30T06:26:51.813ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6570247-d5e3-4ecf-892b-a548d10c653e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118122025-04-30T06:26:51.813ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
118132025-04-30T06:26:51.815ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "079cace7-a19c-4e95-8e23-c976bcd5a0b2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118142025-04-30T06:26:51.815ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
118152025-04-30T06:26:51.820ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc0b90d5-c480-4d9b-bc47-f397ae1a069d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118162025-04-30T06:26:51.820ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
118172025-04-30T06:26:51.826ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e25f288b-5680-4488-b18b-37f88365f9d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118182025-04-30T06:26:51.826ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
118192025-04-30T06:26:51.828ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118202025-04-30T06:26:51.828ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118212025-04-30T06:26:51.828ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118222025-04-30T06:26:51.828ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118232025-04-30T06:26:51.829ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118242025-04-30T06:26:51.834ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118252025-04-30T06:26:51.844ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118262025-04-30T06:26:51.844ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118272025-04-30T06:26:51.844ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118282025-04-30T06:26:51.844ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118292025-04-30T06:26:51.845ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8f91807-3779-4f3d-b043-96e50c3210d3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118302025-04-30T06:26:51.845ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
118312025-04-30T06:26:51.848ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118322025-04-30T06:26:51.848ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118332025-04-30T06:26:51.848ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118342025-04-30T06:26:51.855ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33232c62-d97f-4a5a-a2e5-c7ea13caa8f3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118352025-04-30T06:26:51.855ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
118362025-04-30T06:26:51.864ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa51ba94-83bd-420b-85f3-ece895792bfb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118372025-04-30T06:26:51.864ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
118382025-04-30T06:26:51.880ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00f32026-705c-4c46-8453-0232a8149770", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118392025-04-30T06:26:51.880ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
118402025-04-30T06:26:51.885ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c077757-97a2-4840-a92a-27f729a5830c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118412025-04-30T06:26:51.885ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
118422025-04-30T06:26:51.890ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35a53ba5-fd93-473e-99df-010237385d34", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118432025-04-30T06:26:51.890ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
118442025-04-30T06:26:51.890ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
118452025-04-30T06:26:51.893ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9e15595-6fb3-4328-8155-ba097a798a5d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118462025-04-30T06:26:51.893ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
118472025-04-30T06:26:51.895ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0815d2a0-5706-4c61-bdd2-d09d6da73cbf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118482025-04-30T06:26:51.895ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
118492025-04-30T06:26:51.898ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd3cd68e-e535-44a0-b7d5-f6485528664d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118502025-04-30T06:26:51.899ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
118512025-04-30T06:26:51.903ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61d74c1d-cf7a-434d-830f-f7ba63314a61", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118522025-04-30T06:26:51.903ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
118532025-04-30T06:26:51.906ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce619155-30ac-4552-9dec-8960d4854b43", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118542025-04-30T06:26:51.907ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
118552025-04-30T06:26:51.910ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a169107-352a-472d-83ad-0a5c3009a6c8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118562025-04-30T06:26:51.910ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
118572025-04-30T06:26:51.913ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f95be8b0-612f-464a-b21e-87a9295d4483", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118582025-04-30T06:26:51.913ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
118592025-04-30T06:26:51.916ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19f91696-024d-4f2c-af4f-84c12af99dd9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118602025-04-30T06:26:51.917ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
118612025-04-30T06:26:51.920ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dee6ae18-e62d-4fa5-ab61-7562f4d2fc0e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118622025-04-30T06:26:51.920ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
118632025-04-30T06:26:51.921ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "617f662c-a2ad-4681-b746-f87de10a2fe8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118642025-04-30T06:26:51.921ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
118652025-04-30T06:26:51.923ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8ef9ecf-313b-444e-b5c8-3551fff524e4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118662025-04-30T06:26:51.923ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
118672025-04-30T06:26:51.924ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "601ae90d-bc26-420d-a5b6-ff5a529fe092", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118682025-04-30T06:26:51.925ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
118692025-04-30T06:26:51.926ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "937bd0d3-324a-459a-ab79-24e7d079922c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
118702025-04-30T06:26:51.926ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
118712025-04-30T06:26:51.927ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118722025-04-30T06:26:51.927ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118732025-04-30T06:26:51.927ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118742025-04-30T06:26:51.927ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118752025-04-30T06:26:51.931ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118762025-04-30T06:26:51.936ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
118772025-04-30T06:26:51.936ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
118782025-04-30T06:26:51.936ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118792025-04-30T06:26:51.936ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118802025-04-30T06:26:51.936ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118812025-04-30T06:26:51.936ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118822025-04-30T06:26:51.942ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118832025-04-30T06:26:51.945ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
118842025-04-30T06:26:51.945ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118852025-04-30T06:26:51.945ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118862025-04-30T06:26:51.946ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118872025-04-30T06:26:51.947ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118882025-04-30T06:26:51.947ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118892025-04-30T06:26:51.947ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118902025-04-30T06:26:51.947ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118912025-04-30T06:26:51.948ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118922025-04-30T06:26:52.045ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
118932025-04-30T06:26:52.045ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
118942025-04-30T06:26:52.045ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118952025-04-30T06:26:52.054ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118962025-04-30T06:26:52.054ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118972025-04-30T06:26:52.054ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118982025-04-30T06:26:52.055ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118992025-04-30T06:26:52.055ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:33446 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
119002025-04-30T06:26:52.056ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119012025-04-30T06:26:52.060ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119022025-04-30T06:26:52.061ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:33446 req_id = 0437c742-dc76-4c31-812b-9e1ecba82b3e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119032025-04-30T06:26:52.062ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1199 local_addr = [::1]:50989 method = GET remote_addr = [::1]:33446 req_id = 0437c742-dc76-4c31-812b-9e1ecba82b3e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119042025-04-30T06:26:52.067ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0437c742-dc76-4c31-812b-9e1ecba82b3e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
119052025-04-30T06:26:52.073ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119062025-04-30T06:26:52.073ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119072025-04-30T06:26:52.073ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
119082025-04-30T06:26:52.078ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119092025-04-30T06:26:52.078ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119102025-04-30T06:26:52.079ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119112025-04-30T06:26:52.079ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119122025-04-30T06:26:52.082ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119132025-04-30T06:26:52.083ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119142025-04-30T06:26:52.088ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119152025-04-30T06:26:52.088ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119162025-04-30T06:26:52.088ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119172025-04-30T06:26:52.089ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119182025-04-30T06:26:52.089ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119192025-04-30T06:26:52.091ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119202025-04-30T06:26:52.095ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119212025-04-30T06:26:52.100ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119222025-04-30T06:26:52.100ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119232025-04-30T06:26:52.101ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119242025-04-30T06:26:52.101ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119252025-04-30T06:26:52.166ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
119262025-04-30T06:26:52.167ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
119272025-04-30T06:26:52.167ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 510.194649ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
119282025-04-30T06:26:52.167ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
119292025-04-30T06:26:52.167ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119302025-04-30T06:26:52.167ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119312025-04-30T06:26:52.167ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119322025-04-30T06:26:52.167ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119332025-04-30T06:26:52.168ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119342025-04-30T06:26:52.177ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119352025-04-30T06:26:52.178ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119362025-04-30T06:26:52.179ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
119372025-04-30T06:26:52.180ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
119382025-04-30T06:26:52.180ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119392025-04-30T06:26:52.189ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119402025-04-30T06:26:52.189ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119412025-04-30T06:26:52.189ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119422025-04-30T06:26:52.189ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119432025-04-30T06:26:52.190ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
119442025-04-30T06:26:52.195ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
119452025-04-30T06:26:52.195ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
119462025-04-30T06:26:52.196ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
119472025-04-30T06:26:52.196ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
119482025-04-30T06:26:52.196ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d0 40 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .@..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63942 req_id = 16ae5d08-276c-4223-afbf-c587d7eb2383
119492025-04-30T06:26:52.196ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53312,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63942 req_id = 16ae5d08-276c-4223-afbf-c587d7eb2383
119502025-04-30T06:26:52.196ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
119512025-04-30T06:26:52.197ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
119522025-04-30T06:26:52.197ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63942 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 16ae5d08-276c-4223-afbf-c587d7eb2383
119532025-04-30T06:26:52.197ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3276, tv_nsec: 996660746 } })
119542025-04-30T06:26:52.197ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
119552025-04-30T06:26:52.201ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a5 88 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49853 req_id = 470fb147-e33c-4d4b-91f5-406447b23ee4
119562025-04-30T06:26:52.201ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42376,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49853 req_id = 470fb147-e33c-4d4b-91f5-406447b23ee4
119572025-04-30T06:26:52.203ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
119582025-04-30T06:26:52.203ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49853 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 470fb147-e33c-4d4b-91f5-406447b23ee4
119592025-04-30T06:26:52.270ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
119602025-04-30T06:26:52.271ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
119612025-04-30T06:26:52.271ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119622025-04-30T06:26:52.271ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119632025-04-30T06:26:52.271ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119642025-04-30T06:26:52.271ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119652025-04-30T06:26:52.271ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119662025-04-30T06:26:52.271ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119672025-04-30T06:26:52.272ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7d b3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c }............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58223 req_id = 81262135-168e-4218-a22f-c2ef63c3d09f
119682025-04-30T06:26:52.272ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32179,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58223 req_id = 81262135-168e-4218-a22f-c2ef63c3d09f
119692025-04-30T06:26:52.272ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
119702025-04-30T06:26:52.272ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32179, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58223 req_id = 81262135-168e-4218-a22f-c2ef63c3d09f
119712025-04-30T06:26:52.272ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
119722025-04-30T06:26:52.272ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
119732025-04-30T06:26:52.272ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
119742025-04-30T06:26:52.274ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119752025-04-30T06:26:52.280ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119762025-04-30T06:26:52.280ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119772025-04-30T06:26:52.280ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119782025-04-30T06:26:52.281ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119792025-04-30T06:26:52.282ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "798826e4-d111-4b0d-abc3-ae6e12ff9336", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119802025-04-30T06:26:52.283ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
119812025-04-30T06:26:52.283ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
119822025-04-30T06:26:52.290ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119832025-04-30T06:26:52.290ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119842025-04-30T06:26:52.290ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119852025-04-30T06:26:52.290ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51506f7d-7a60-4ccd-b73d-875346300660", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119862025-04-30T06:26:52.290ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
119872025-04-30T06:26:52.292ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "494a0392-e1c3-46cd-b91a-54905c5cebca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119882025-04-30T06:26:52.292ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
119892025-04-30T06:26:52.298ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb7f4725-3170-4655-b737-b804034975c4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119902025-04-30T06:26:52.298ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
119912025-04-30T06:26:52.301ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29e6ea7b-1d19-4731-8add-c2fcfa863e01", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119922025-04-30T06:26:52.301ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
119932025-04-30T06:26:52.313ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1887be49-7244-446b-82ab-1bc458a65e3d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119942025-04-30T06:26:52.313ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
119952025-04-30T06:26:52.316ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a3efcd9-174f-48c0-9729-a4c9db50c176", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119962025-04-30T06:26:52.316ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
119972025-04-30T06:26:52.319ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f053f7cb-5519-45b8-8d9d-20b0d03606b7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
119982025-04-30T06:26:52.324ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
119992025-04-30T06:26:52.327ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
120002025-04-30T06:26:52.328ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e062b6b7-2af4-4b10-9909-e8ec3e9a5320", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120012025-04-30T06:26:52.328ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
120022025-04-30T06:26:52.331ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac76112-3c1d-4190-a70b-993c6daac972", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120032025-04-30T06:26:52.331ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
120042025-04-30T06:26:52.333ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f74e2697-be15-4dfb-9925-d038426cd8cd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120052025-04-30T06:26:52.333ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
120062025-04-30T06:26:52.335ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e32a2fb-cb25-43da-ad3d-31c7c67d8832", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120072025-04-30T06:26:52.335ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
120082025-04-30T06:26:52.337ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95e1719b-e5a9-4455-83be-9fba480f31ac", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120092025-04-30T06:26:52.337ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
120102025-04-30T06:26:52.340ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d21459e1-d91e-4d37-b5a7-836a6aa507b1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120112025-04-30T06:26:52.340ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
120122025-04-30T06:26:52.344ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3f3d537-5daf-4351-8f4a-1dc1628dc9b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120132025-04-30T06:26:52.344ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
120142025-04-30T06:26:52.347ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52335ab4-5559-46b8-a1bf-4cb5464259e5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120152025-04-30T06:26:52.347ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
120162025-04-30T06:26:52.353ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bcea6b3-d358-4e16-b284-3a0324594529", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120172025-04-30T06:26:52.354ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
120182025-04-30T06:26:52.357ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8acf04f8-cfa0-4695-a563-6dc731c1a8c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120192025-04-30T06:26:52.357ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
120202025-04-30T06:26:52.360ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23cff346-8f3a-481b-a7fb-962f55164b56", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120212025-04-30T06:26:52.361ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
120222025-04-30T06:26:52.364ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4056dee4-ecbc-4187-b467-3951f0b23197", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120232025-04-30T06:26:52.365ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
120242025-04-30T06:26:52.368ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a1815d-d10e-4a8d-ab71-f0fc22fd63c0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120252025-04-30T06:26:52.368ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
120262025-04-30T06:26:52.370ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ace6671-7763-448d-a992-46fa1a7c76c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120272025-04-30T06:26:52.370ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
120282025-04-30T06:26:52.372ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac42eac5-f7ad-4260-a854-d51cf62cc376", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120292025-04-30T06:26:52.372ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
120302025-04-30T06:26:52.379ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120312025-04-30T06:26:52.379ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120322025-04-30T06:26:52.379ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120332025-04-30T06:26:52.380ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
120342025-04-30T06:26:52.381ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120352025-04-30T06:26:52.385ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83c12996-200a-42a7-aedb-9e04fa147d64", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120362025-04-30T06:26:52.387ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
120372025-04-30T06:26:52.387ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120382025-04-30T06:26:52.397ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120392025-04-30T06:26:52.397ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120402025-04-30T06:26:52.397ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
120412025-04-30T06:26:52.398ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120422025-04-30T06:26:52.400ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a0e151d-2e9c-44fd-978c-34166a3d3f45", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120432025-04-30T06:26:52.400ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
120442025-04-30T06:26:52.401ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120452025-04-30T06:26:52.401ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
120462025-04-30T06:26:52.401ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120472025-04-30T06:26:52.404ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31b99864-7e97-47e2-8c91-7cfaa3b0dadf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120482025-04-30T06:26:52.404ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
120492025-04-30T06:26:52.407ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db453724-43d1-4112-884b-27f39604ef64", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120502025-04-30T06:26:52.407ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
120512025-04-30T06:26:52.409ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1264171-94c6-4909-8456-47fbdd851a03", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120522025-04-30T06:26:52.409ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
120532025-04-30T06:26:52.413ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc1228d-e3be-4538-a3c0-a1148219a9a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120542025-04-30T06:26:52.413ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
120552025-04-30T06:26:52.416ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e592a3d3-eee4-4188-b1a0-a1196449a897", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120562025-04-30T06:26:52.416ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
120572025-04-30T06:26:52.417ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9271ebd4-85d1-45be-a910-553ec890d566", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120582025-04-30T06:26:52.417ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
120592025-04-30T06:26:52.418ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a05df230-195f-4f93-9ebf-2e05e3f9ba83", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120602025-04-30T06:26:52.419ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
120612025-04-30T06:26:52.420ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bac8e25b-4c0a-43fe-85b1-70c4bd9f43e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120622025-04-30T06:26:52.420ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
120632025-04-30T06:26:52.421ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb912b8c-66ed-4cf2-bdc0-b8b41dc19eb9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120642025-04-30T06:26:52.421ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
120652025-04-30T06:26:52.423ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7f97a05-40a1-499c-bf71-14f70e8b4527", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120662025-04-30T06:26:52.424ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
120672025-04-30T06:26:52.426ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e29b0255-9012-48f6-91ec-46d58cf3e291", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120682025-04-30T06:26:52.426ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
120692025-04-30T06:26:52.430ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
120702025-04-30T06:26:52.434ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "138d44fb-1279-4e9d-ac17-43956f2f10cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120712025-04-30T06:26:52.434ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
120722025-04-30T06:26:52.437ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41f8d33e-07be-41be-a346-8d446c35f6b8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120732025-04-30T06:26:52.437ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
120742025-04-30T06:26:52.441ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8916424c-76a2-4e46-85aa-fb5b37dc1b7e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120752025-04-30T06:26:52.441ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
120762025-04-30T06:26:52.444ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fefa76e-d4c9-4346-ae92-2264baed1385", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120772025-04-30T06:26:52.444ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
120782025-04-30T06:26:52.452ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f03ffc52-6482-4fe5-b10f-414a439e08b2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120792025-04-30T06:26:52.452ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
120802025-04-30T06:26:52.453ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e128446f-6ae9-4369-8a08-f9fb1f2b3e6c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120812025-04-30T06:26:52.454ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
120822025-04-30T06:26:52.455ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cded5861-3f25-44f8-a18e-ad5a1600266c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120832025-04-30T06:26:52.455ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
120842025-04-30T06:26:52.456ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b6358d-e79f-4f58-9474-76b2b665adfa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120852025-04-30T06:26:52.457ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
120862025-04-30T06:26:52.458ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d2ab0c5-12c1-4cf1-bc19-10bef2f975e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
120872025-04-30T06:26:52.458ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120882025-04-30T06:26:52.458ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120892025-04-30T06:26:52.458ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120902025-04-30T06:26:52.458ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120912025-04-30T06:26:52.459ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120922025-04-30T06:26:52.461ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120932025-04-30T06:26:52.475ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120942025-04-30T06:26:52.475ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120952025-04-30T06:26:52.475ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120962025-04-30T06:26:52.475ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120972025-04-30T06:26:52.476ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120982025-04-30T06:26:52.478ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120992025-04-30T06:26:52.482ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121002025-04-30T06:26:52.482ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121012025-04-30T06:26:52.482ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121022025-04-30T06:26:52.482ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121032025-04-30T06:26:52.482ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121042025-04-30T06:26:52.483ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121052025-04-30T06:26:52.488ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121062025-04-30T06:26:52.488ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121072025-04-30T06:26:52.488ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121082025-04-30T06:26:52.489ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121092025-04-30T06:26:52.490ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121102025-04-30T06:26:52.497ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121112025-04-30T06:26:52.497ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121122025-04-30T06:26:52.497ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121132025-04-30T06:26:52.497ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121142025-04-30T06:26:52.556ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
121152025-04-30T06:26:52.556ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
121162025-04-30T06:26:52.557ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:40305 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
121172025-04-30T06:26:52.557ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:40305 req_id = 3991822f-5fd4-4156-b640-2c33ba3d2358 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121182025-04-30T06:26:52.558ZINFOtest_disk_metrics (dropshot): request completed latency_us = 741 local_addr = [::1]:50989 method = GET remote_addr = [::1]:40305 req_id = 3991822f-5fd4-4156-b640-2c33ba3d2358 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121192025-04-30T06:26:52.558ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121202025-04-30T06:26:52.558ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121212025-04-30T06:26:52.560ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121222025-04-30T06:26:52.560ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3991822f-5fd4-4156-b640-2c33ba3d2358", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121232025-04-30T06:26:52.564ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
121242025-04-30T06:26:52.564ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
121252025-04-30T06:26:52.564ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
121262025-04-30T06:26:52.569ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
121272025-04-30T06:26:52.569ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121282025-04-30T06:26:52.569ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121292025-04-30T06:26:52.569ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121302025-04-30T06:26:52.571ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121312025-04-30T06:26:52.577ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121322025-04-30T06:26:52.582ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
121332025-04-30T06:26:52.583ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121342025-04-30T06:26:52.583ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121352025-04-30T06:26:52.584ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121362025-04-30T06:26:52.638ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
121372025-04-30T06:26:52.645ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
121382025-04-30T06:26:52.645ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
121392025-04-30T06:26:52.646ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 478.859446ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
121402025-04-30T06:26:52.646ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
121412025-04-30T06:26:52.646ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
121422025-04-30T06:26:52.646ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121432025-04-30T06:26:52.646ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121442025-04-30T06:26:52.646ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121452025-04-30T06:26:52.647ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121462025-04-30T06:26:52.650ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121472025-04-30T06:26:52.655ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
121482025-04-30T06:26:52.656ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121492025-04-30T06:26:52.656ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121502025-04-30T06:26:52.657ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121512025-04-30T06:26:52.662ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
121522025-04-30T06:26:52.662ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
121532025-04-30T06:26:52.663ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: dc 26 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .&..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62901 req_id = 7c24c9cb-4103-42bd-8a70-84a0a43ae722
121542025-04-30T06:26:52.663ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56358,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62901 req_id = 7c24c9cb-4103-42bd-8a70-84a0a43ae722
121552025-04-30T06:26:52.665ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
121562025-04-30T06:26:52.665ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121572025-04-30T06:26:52.665ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62901 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7c24c9cb-4103-42bd-8a70-84a0a43ae722
121582025-04-30T06:26:52.667ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3277, tv_nsec: 466158019 } })
121592025-04-30T06:26:52.667ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
121602025-04-30T06:26:52.668ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 60 9b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e `...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55110 req_id = 3aba080e-61bf-40bf-a002-97287546caca
121612025-04-30T06:26:52.668ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24731,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55110 req_id = 3aba080e-61bf-40bf-a002-97287546caca
121622025-04-30T06:26:52.669ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121632025-04-30T06:26:52.669ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55110 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3aba080e-61bf-40bf-a002-97287546caca
121642025-04-30T06:26:52.740ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
121652025-04-30T06:26:52.741ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121662025-04-30T06:26:52.741ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121672025-04-30T06:26:52.741ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121682025-04-30T06:26:52.741ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121692025-04-30T06:26:52.742ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121702025-04-30T06:26:52.742ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8c 79 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .y..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53358 req_id = 550c0520-9461-4363-9738-60aceea84370
121712025-04-30T06:26:52.742ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35961,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53358 req_id = 550c0520-9461-4363-9738-60aceea84370
121722025-04-30T06:26:52.742ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
121732025-04-30T06:26:52.742ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35961, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53358 req_id = 550c0520-9461-4363-9738-60aceea84370
121742025-04-30T06:26:52.743ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
121752025-04-30T06:26:52.743ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
121762025-04-30T06:26:52.743ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
121772025-04-30T06:26:52.745ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b20127bb-e42a-4523-8f84-4f309c8e3239", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121782025-04-30T06:26:52.745ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
121792025-04-30T06:26:52.745ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121802025-04-30T06:26:52.750ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121812025-04-30T06:26:52.750ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121822025-04-30T06:26:52.750ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121832025-04-30T06:26:52.750ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121842025-04-30T06:26:52.753ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121852025-04-30T06:26:52.753ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121862025-04-30T06:26:52.753ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121872025-04-30T06:26:52.755ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "177ca96a-51dd-4915-9f13-d5b99554a2c0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121882025-04-30T06:26:52.755ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
121892025-04-30T06:26:52.761ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c13c564c-ae49-4162-b6ac-09367904fb29", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121902025-04-30T06:26:52.762ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
121912025-04-30T06:26:52.764ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9099a9d1-84d7-4da7-997c-ebb78b2acda1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121922025-04-30T06:26:52.764ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
121932025-04-30T06:26:52.767ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16a55f41-c60a-4320-99fb-42f5443cb285", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121942025-04-30T06:26:52.767ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
121952025-04-30T06:26:52.770ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e77ae92-5c70-4f88-8e02-1654fd346b30", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121962025-04-30T06:26:52.770ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
121972025-04-30T06:26:52.774ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7de3f534-8c26-43d9-a6f1-2fe7d199dbe8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
121982025-04-30T06:26:52.774ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
121992025-04-30T06:26:52.785ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fe35c1b-3730-44d2-be89-f133ca341953", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122002025-04-30T06:26:52.785ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
122012025-04-30T06:26:52.789ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
122022025-04-30T06:26:52.790ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51cf79ca-76d7-45b4-8b35-ebd54a98b420", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122032025-04-30T06:26:52.790ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
122042025-04-30T06:26:52.793ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd59d38b-d91e-48fa-8053-a073df8fa479", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122052025-04-30T06:26:52.793ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
122062025-04-30T06:26:52.794ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c5dab40-263a-4778-b26c-f54ea7db104b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122072025-04-30T06:26:52.794ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
122082025-04-30T06:26:52.798ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "839093dc-b5ab-477d-a5b5-b9b83712ce1d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122092025-04-30T06:26:52.798ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
122102025-04-30T06:26:52.801ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3dded42-cd90-4ac5-b3af-3c08b5bbd298", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122112025-04-30T06:26:52.801ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
122122025-04-30T06:26:52.803ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fe38518-0478-4f76-8033-5490008d348b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122132025-04-30T06:26:52.805ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
122142025-04-30T06:26:52.812ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af26a8fd-44ce-45f2-bcbc-5725cca41450", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122152025-04-30T06:26:52.812ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
122162025-04-30T06:26:52.815ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf204034-ca91-4759-bdc1-d24c1bbcd85e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122172025-04-30T06:26:52.815ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
122182025-04-30T06:26:52.819ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bf567c6-7b71-402d-b7a4-f10e902b83f0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122192025-04-30T06:26:52.819ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
122202025-04-30T06:26:52.823ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e23bb7-af13-4254-bc85-4995eab9918e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122212025-04-30T06:26:52.823ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
122222025-04-30T06:26:52.825ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c3bde6-80b5-4b10-ac8b-29adca5ff491", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122232025-04-30T06:26:52.825ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
122242025-04-30T06:26:52.826ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "959b8b46-0afd-48dd-8d01-e0ef17f10d09", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122252025-04-30T06:26:52.826ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
122262025-04-30T06:26:52.828ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e9a03eb-4dda-46f9-8249-650009ebe06d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122272025-04-30T06:26:52.828ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
122282025-04-30T06:26:52.831ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee9d2441-66e6-4595-8824-138a0da2a1a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122292025-04-30T06:26:52.831ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
122302025-04-30T06:26:52.832ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39d243e4-62bd-4c87-8516-43db050c4591", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122312025-04-30T06:26:52.833ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
122322025-04-30T06:26:52.837ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01ce8657-c0b0-4920-9432-e2be88f13824", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122332025-04-30T06:26:52.837ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
122342025-04-30T06:26:52.840ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "539122ad-4077-400e-a097-9cca9fb7e26a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122352025-04-30T06:26:52.840ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
122362025-04-30T06:26:52.841ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122372025-04-30T06:26:52.841ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122382025-04-30T06:26:52.841ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122392025-04-30T06:26:52.841ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122402025-04-30T06:26:52.842ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122412025-04-30T06:26:52.846ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122422025-04-30T06:26:52.854ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122432025-04-30T06:26:52.854ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122442025-04-30T06:26:52.854ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122452025-04-30T06:26:52.854ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122462025-04-30T06:26:52.857ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122472025-04-30T06:26:52.857ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122482025-04-30T06:26:52.857ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122492025-04-30T06:26:52.857ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
122502025-04-30T06:26:52.858ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13493e68-3881-469e-b951-4de95d947777", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122512025-04-30T06:26:52.858ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
122522025-04-30T06:26:52.859ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
122532025-04-30T06:26:52.860ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 97658e1f-48e5-42cd-acd2-76c77b46ae60 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
122542025-04-30T06:26:52.862ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 97658e1f-48e5-42cd-acd2-76c77b46ae60 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
122552025-04-30T06:26:52.862ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 97658e1f-48e5-42cd-acd2-76c77b46ae60 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
122562025-04-30T06:26:52.863ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 97658e1f-48e5-42cd-acd2-76c77b46ae60 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
122572025-04-30T06:26:52.867ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 6038 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 97658e1f-48e5-42cd-acd2-76c77b46ae60 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
122582025-04-30T06:26:52.867ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "97658e1f-48e5-42cd-acd2-76c77b46ae60", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122592025-04-30T06:26:52.867ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
122602025-04-30T06:26:52.869ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eb387222-1074-4757-b479-22cccab07318 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
122612025-04-30T06:26:52.869ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eb387222-1074-4757-b479-22cccab07318 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
122622025-04-30T06:26:52.870ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eb387222-1074-4757-b479-22cccab07318 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
122632025-04-30T06:26:52.870ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eb387222-1074-4757-b479-22cccab07318 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
122642025-04-30T06:26:52.872ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b381d825-9073-4c2f-a07e-8152920ef217", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122652025-04-30T06:26:52.873ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
122662025-04-30T06:26:52.874ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 5247 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eb387222-1074-4757-b479-22cccab07318 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
122672025-04-30T06:26:52.886ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb387222-1074-4757-b479-22cccab07318", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122682025-04-30T06:26:52.886ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
122692025-04-30T06:26:52.886ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
122702025-04-30T06:26:52.886ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
122712025-04-30T06:26:52.890ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03b5aa1f-0c2a-4d4f-8a7b-80dc3218d158", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122722025-04-30T06:26:52.891ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
122732025-04-30T06:26:52.894ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65fec677-bbd9-4398-9829-97bf47296e68", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122742025-04-30T06:26:52.894ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
122752025-04-30T06:26:52.897ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "640e6e0d-d24d-4417-82ac-65fae14bfe6a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122762025-04-30T06:26:52.897ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
122772025-04-30T06:26:52.899ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81e281b4-a82c-4338-b663-e4836604b35b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122782025-04-30T06:26:52.899ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
122792025-04-30T06:26:52.900ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fa7af06-e9d2-4d65-8210-1e8822097366", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122802025-04-30T06:26:52.900ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
122812025-04-30T06:26:52.901ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6a40478-d0a7-4f96-b5f6-785e846aa313", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122822025-04-30T06:26:52.901ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
122832025-04-30T06:26:52.903ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a2d1b26-cffa-46ea-870a-62afea31f485", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122842025-04-30T06:26:52.903ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
122852025-04-30T06:26:52.907ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ee6f18-d61d-4797-8556-44c89c69a345", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122862025-04-30T06:26:52.907ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
122872025-04-30T06:26:52.908ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
122882025-04-30T06:26:52.910ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0fdfd4b-c26b-4cf7-9b2d-8845ab2d2acc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122892025-04-30T06:26:52.910ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
122902025-04-30T06:26:52.912ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c69758c-c679-4e6c-b9f5-b7f1f2576811", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122912025-04-30T06:26:52.912ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
122922025-04-30T06:26:52.915ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93489e28-17ed-42ed-806a-47513f489438", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122932025-04-30T06:26:52.915ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
122942025-04-30T06:26:52.917ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9c39db9-0696-4ea9-afd8-1119c98d5f97", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122952025-04-30T06:26:52.918ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
122962025-04-30T06:26:52.922ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d44162a1-cf0f-48c4-8ff9-8aa085abc547", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122972025-04-30T06:26:52.922ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
122982025-04-30T06:26:52.945ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bee56a87-5781-41bf-86a9-a2a2c74e8f3b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
122992025-04-30T06:26:52.946ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
123002025-04-30T06:26:52.960ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123012025-04-30T06:26:52.960ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123022025-04-30T06:26:52.960ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123032025-04-30T06:26:52.960ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
123042025-04-30T06:26:52.961ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123052025-04-30T06:26:52.962ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7aa9fa8-2751-4756-9246-cf1d64252682", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
123062025-04-30T06:26:52.962ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
123072025-04-30T06:26:52.969ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123082025-04-30T06:26:52.975ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123092025-04-30T06:26:52.975ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123102025-04-30T06:26:52.975ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
123112025-04-30T06:26:52.976ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123122025-04-30T06:26:52.978ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c86d2f08-dd82-452c-a3ef-672caf1ac103", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
123132025-04-30T06:26:52.978ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
123142025-04-30T06:26:52.984ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123152025-04-30T06:26:52.984ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
123162025-04-30T06:26:52.986ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123172025-04-30T06:26:52.986ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123182025-04-30T06:26:52.986ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123192025-04-30T06:26:52.990ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123202025-04-30T06:26:52.991ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123212025-04-30T06:26:52.991ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123222025-04-30T06:26:52.991ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123232025-04-30T06:26:52.992ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9781b453-5ab1-490b-a6cc-2f8f2a69d6a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
123242025-04-30T06:26:52.992ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
123252025-04-30T06:26:52.999ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3adb39cb-49b9-4638-a546-0eb85b013eb5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
123262025-04-30T06:26:52.999ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
123272025-04-30T06:26:52.999ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123282025-04-30T06:26:52.999ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123292025-04-30T06:26:52.999ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123302025-04-30T06:26:53.004ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123312025-04-30T06:26:53.008ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123322025-04-30T06:26:53.013ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
123332025-04-30T06:26:53.013ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
123342025-04-30T06:26:53.013ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123352025-04-30T06:26:53.013ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123362025-04-30T06:26:53.013ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123372025-04-30T06:26:53.014ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123382025-04-30T06:26:53.016ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123392025-04-30T06:26:53.027ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
123402025-04-30T06:26:53.027ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123412025-04-30T06:26:53.027ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123422025-04-30T06:26:53.028ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123432025-04-30T06:26:53.105ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
123442025-04-30T06:26:53.106ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
123452025-04-30T06:26:53.107ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:55267 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
123462025-04-30T06:26:53.107ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:55267 req_id = 3cb28319-4c85-4067-95a1-abe579230e15 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
123472025-04-30T06:26:53.108ZINFOtest_disk_metrics (dropshot): request completed latency_us = 499 local_addr = [::1]:50989 method = GET remote_addr = [::1]:55267 req_id = 3cb28319-4c85-4067-95a1-abe579230e15 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
123482025-04-30T06:26:53.108ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cb28319-4c85-4067-95a1-abe579230e15", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
123492025-04-30T06:26:53.113ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
123502025-04-30T06:26:53.113ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
123512025-04-30T06:26:53.113ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
123522025-04-30T06:26:53.124ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
123532025-04-30T06:26:53.124ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123542025-04-30T06:26:53.124ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123552025-04-30T06:26:53.124ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123562025-04-30T06:26:53.127ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123572025-04-30T06:26:53.131ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123582025-04-30T06:26:53.136ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123592025-04-30T06:26:53.136ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123602025-04-30T06:26:53.136ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123612025-04-30T06:26:53.137ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123622025-04-30T06:26:53.167ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
123632025-04-30T06:26:53.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
123642025-04-30T06:26:53.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
123652025-04-30T06:26:53.176ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
123662025-04-30T06:26:53.177ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
123672025-04-30T06:26:53.183ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
123682025-04-30T06:26:53.183ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
123692025-04-30T06:26:53.186ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
123702025-04-30T06:26:53.186ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
123712025-04-30T06:26:53.190ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
123722025-04-30T06:26:53.190ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
123732025-04-30T06:26:53.191ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 545.004897ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
123742025-04-30T06:26:53.191ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
123752025-04-30T06:26:53.191ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
123762025-04-30T06:26:53.191ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123772025-04-30T06:26:53.191ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123782025-04-30T06:26:53.191ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123792025-04-30T06:26:53.192ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123802025-04-30T06:26:53.194ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123812025-04-30T06:26:53.201ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123822025-04-30T06:26:53.201ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123832025-04-30T06:26:53.201ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123842025-04-30T06:26:53.201ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123852025-04-30T06:26:53.205ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
123862025-04-30T06:26:53.205ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
123872025-04-30T06:26:53.206ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c4 f2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53073 req_id = 5d77d5a2-fbc7-42b4-aebb-428b51c092f6
123882025-04-30T06:26:53.206ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50418,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53073 req_id = 5d77d5a2-fbc7-42b4-aebb-428b51c092f6
123892025-04-30T06:26:53.206ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
123902025-04-30T06:26:53.207ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123912025-04-30T06:26:53.207ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53073 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5d77d5a2-fbc7-42b4-aebb-428b51c092f6
123922025-04-30T06:26:53.208ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 6964440 } })
123932025-04-30T06:26:53.208ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
123942025-04-30T06:26:53.208ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c0 f8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58032 req_id = c3a38911-3347-4848-a756-39ac18564880
123952025-04-30T06:26:53.208ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49400,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58032 req_id = c3a38911-3347-4848-a756-39ac18564880
123962025-04-30T06:26:53.209ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123972025-04-30T06:26:53.209ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58032 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c3a38911-3347-4848-a756-39ac18564880
123982025-04-30T06:26:53.256ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
123992025-04-30T06:26:53.257ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124002025-04-30T06:26:53.257ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124012025-04-30T06:26:53.257ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124022025-04-30T06:26:53.257ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124032025-04-30T06:26:53.258ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124042025-04-30T06:26:53.258ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
124052025-04-30T06:26:53.259ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a7 46 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .F..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60032 req_id = f9252b5e-d6a5-4c14-b546-e3030524c5be
124062025-04-30T06:26:53.259ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42822,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60032 req_id = f9252b5e-d6a5-4c14-b546-e3030524c5be
124072025-04-30T06:26:53.259ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
124082025-04-30T06:26:53.259ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42822, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60032 req_id = f9252b5e-d6a5-4c14-b546-e3030524c5be
124092025-04-30T06:26:53.260ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
124102025-04-30T06:26:53.260ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
124112025-04-30T06:26:53.260ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
124122025-04-30T06:26:53.260ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 6d 94 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 m............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59726 req_id = e1989165-3a25-4641-b001-cab971a0eaad
124132025-04-30T06:26:53.260ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28052,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59726 req_id = e1989165-3a25-4641-b001-cab971a0eaad
124142025-04-30T06:26:53.260ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
124152025-04-30T06:26:53.260ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124162025-04-30T06:26:53.260ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59726 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e1989165-3a25-4641-b001-cab971a0eaad
124172025-04-30T06:26:53.261ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 60520157 } })
124182025-04-30T06:26:53.261ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
124192025-04-30T06:26:53.262ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124202025-04-30T06:26:53.269ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124212025-04-30T06:26:53.269ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124222025-04-30T06:26:53.269ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124232025-04-30T06:26:53.270ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124242025-04-30T06:26:53.271ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f9 fe 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62515 req_id = 223b27c2-4a29-44a2-b3df-dd9a1a2476a6
124252025-04-30T06:26:53.272ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63998,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62515 req_id = 223b27c2-4a29-44a2-b3df-dd9a1a2476a6
124262025-04-30T06:26:53.272ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124272025-04-30T06:26:53.272ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62515 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 223b27c2-4a29-44a2-b3df-dd9a1a2476a6
124282025-04-30T06:26:53.274ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
124292025-04-30T06:26:53.351ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
124302025-04-30T06:26:53.351ZDEBGtest_disk_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
124312025-04-30T06:26:53.352ZDEBGtest_disk_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
124322025-04-30T06:26:53.352ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
124332025-04-30T06:26:53.353ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
124342025-04-30T06:26:53.354ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124352025-04-30T06:26:53.354ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124362025-04-30T06:26:53.356ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124372025-04-30T06:26:53.356ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7797895f-d26d-4a06-bd38-3efe847f18ea", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124382025-04-30T06:26:53.356ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
124392025-04-30T06:26:53.356ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
124402025-04-30T06:26:53.359ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fbd4971-ec38-48f3-b789-612d0254295c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124412025-04-30T06:26:53.360ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
124422025-04-30T06:26:53.363ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c28aa44-648e-4409-b190-828e57684caf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124432025-04-30T06:26:53.363ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
124442025-04-30T06:26:53.367ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a552c24d-d0d6-4cfd-91fc-150ce1cebc57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124452025-04-30T06:26:53.368ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
124462025-04-30T06:26:53.372ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec56fe8-c1d8-4531-919a-a7c62eb75584", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124472025-04-30T06:26:53.373ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
124482025-04-30T06:26:53.377ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdb7a165-f826-4a7d-b4a7-095583b69519", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124492025-04-30T06:26:53.377ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
124502025-04-30T06:26:53.382ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "340c5f1c-156f-4a54-bc2e-e897d2738124", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124512025-04-30T06:26:53.382ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
124522025-04-30T06:26:53.385ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c56878b-a439-4faf-87e5-0d5682ba366e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124532025-04-30T06:26:53.386ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
124542025-04-30T06:26:53.388ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca84f90-a633-4a1d-afeb-178c0a708c50", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124552025-04-30T06:26:53.389ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
124562025-04-30T06:26:53.390ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
124572025-04-30T06:26:53.391ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ff7c6a2-bb44-4e38-bba1-5b0d32606b47", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124582025-04-30T06:26:53.391ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
124592025-04-30T06:26:53.392ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26d06639-bc13-427b-8375-258975beeaa3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124602025-04-30T06:26:53.392ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
124612025-04-30T06:26:53.393ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "813d60f5-d350-4d8e-95e6-4a9923c54d92", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124622025-04-30T06:26:53.394ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
124632025-04-30T06:26:53.395ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da57e1c1-f7af-4ecd-93be-de985c8f946b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124642025-04-30T06:26:53.395ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
124652025-04-30T06:26:53.397ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3543f16-9847-4e69-8508-e1254de0274d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124662025-04-30T06:26:53.397ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
124672025-04-30T06:26:53.399ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04a8a1e0-ed7a-44df-87b7-ada423e464c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124682025-04-30T06:26:53.399ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
124692025-04-30T06:26:53.402ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9322a561-3aaa-4746-91a6-2c7c387a2b7a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124702025-04-30T06:26:53.402ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
124712025-04-30T06:26:53.404ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a2bd397-9fff-459b-9965-e0f6427ca236", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124722025-04-30T06:26:53.404ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
124732025-04-30T06:26:53.408ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17a60b51-7eed-413b-87cb-74bc830178c9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124742025-04-30T06:26:53.409ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
124752025-04-30T06:26:53.411ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "076e06a2-145c-4bf7-bc14-07088159b09a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124762025-04-30T06:26:53.411ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
124772025-04-30T06:26:53.412ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "297ba598-09da-426b-92a1-3610dc521e3d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124782025-04-30T06:26:53.412ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
124792025-04-30T06:26:53.413ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20d89165-1433-402e-a214-c0b8cf2f7899", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124802025-04-30T06:26:53.413ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
124812025-04-30T06:26:53.414ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "052eaf79-2972-4bd5-88e9-7915f5d12266", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124822025-04-30T06:26:53.414ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
124832025-04-30T06:26:53.416ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0080dd2-e794-425d-b52d-4e7188a34a92", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124842025-04-30T06:26:53.417ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
124852025-04-30T06:26:53.420ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9682e53-a212-436c-85ba-65fca132c60c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124862025-04-30T06:26:53.420ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
124872025-04-30T06:26:53.422ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf26e0de-ea76-47f7-8ba7-974f57aa748a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124882025-04-30T06:26:53.422ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
124892025-04-30T06:26:53.424ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a6cf201-2b52-48bf-a1f5-ccbce1589145", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124902025-04-30T06:26:53.424ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
124912025-04-30T06:26:53.426ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "694c5cf1-08fd-4d91-bc9e-12d6568e3727", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124922025-04-30T06:26:53.426ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
124932025-04-30T06:26:53.429ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12cb3d03-9c27-4575-b03f-bcad760a039c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124942025-04-30T06:26:53.429ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
124952025-04-30T06:26:53.431ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a13c0791-c821-40df-aa5d-b7cbc3e8447a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124962025-04-30T06:26:53.431ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
124972025-04-30T06:26:53.432ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37a38849-b34f-4458-93a9-b687afa384ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
124982025-04-30T06:26:53.434ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
124992025-04-30T06:26:53.434ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a95ff34d-084a-4108-b69d-9824e18ed9eb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125002025-04-30T06:26:53.434ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
125012025-04-30T06:26:53.434ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5bf0631-4cab-42fc-89e0-0b3bc9d4483d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125022025-04-30T06:26:53.434ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
125032025-04-30T06:26:53.435ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "998e98df-2f02-4077-940c-c65a8cbbe161", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125042025-04-30T06:26:53.436ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
125052025-04-30T06:26:53.436ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d55297a2-4272-4af5-840c-a1035016d20e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125062025-04-30T06:26:53.437ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
125072025-04-30T06:26:53.438ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9c03647-5e04-401c-b617-f11db49fe00b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125082025-04-30T06:26:53.438ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
125092025-04-30T06:26:53.440ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4540707c-7593-452f-bf8e-26f845f3a68b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125102025-04-30T06:26:53.440ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
125112025-04-30T06:26:53.441ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125122025-04-30T06:26:53.441ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125132025-04-30T06:26:53.441ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125142025-04-30T06:26:53.441ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125152025-04-30T06:26:53.442ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125162025-04-30T06:26:53.445ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f93a9f3-967e-405b-9ca3-2b1c6dd14fcb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125172025-04-30T06:26:53.445ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
125182025-04-30T06:26:53.446ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125192025-04-30T06:26:53.454ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125202025-04-30T06:26:53.454ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125212025-04-30T06:26:53.454ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125222025-04-30T06:26:53.455ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125232025-04-30T06:26:53.458ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125242025-04-30T06:26:53.458ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125252025-04-30T06:26:53.459ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125262025-04-30T06:26:53.461ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9882a80c-b693-4721-9180-830525e02694", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125272025-04-30T06:26:53.461ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
125282025-04-30T06:26:53.467ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f23f3329-a0a7-445f-88e3-4097b9ff4bd8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125292025-04-30T06:26:53.467ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
125302025-04-30T06:26:53.472ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1c5fec0-40a8-49c8-b3c9-6af4eadbf85e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125312025-04-30T06:26:53.472ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
125322025-04-30T06:26:53.478ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88fbb048-01e0-4c3a-a4eb-d60f543558e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125332025-04-30T06:26:53.478ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
125342025-04-30T06:26:53.480ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38ee95e0-6489-4b18-a9e6-f5693183742d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125352025-04-30T06:26:53.482ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
125362025-04-30T06:26:53.484ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e90dfd3-2693-4342-b880-46cca3d4b5cc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125372025-04-30T06:26:53.484ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
125382025-04-30T06:26:53.486ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83cd74f6-a279-464f-a14c-80ae61bbab3a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125392025-04-30T06:26:53.487ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
125402025-04-30T06:26:53.493ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b9ea9c1-c896-4be3-ae28-0b60bbf4a855", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125412025-04-30T06:26:53.494ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
125422025-04-30T06:26:53.494ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125432025-04-30T06:26:53.494ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125442025-04-30T06:26:53.495ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125452025-04-30T06:26:53.496ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125462025-04-30T06:26:53.496ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
125472025-04-30T06:26:53.501ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125482025-04-30T06:26:53.509ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
125492025-04-30T06:26:53.509ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
125502025-04-30T06:26:53.510ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125512025-04-30T06:26:53.510ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125522025-04-30T06:26:53.510ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125532025-04-30T06:26:53.510ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125542025-04-30T06:26:53.511ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
125552025-04-30T06:26:53.512ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: be 88 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56589 req_id = de573564-9829-48a6-b9e2-31b4157c8ac8
125562025-04-30T06:26:53.512ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48776,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56589 req_id = de573564-9829-48a6-b9e2-31b4157c8ac8
125572025-04-30T06:26:53.513ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
125582025-04-30T06:26:53.513ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125592025-04-30T06:26:53.513ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56589 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = de573564-9829-48a6-b9e2-31b4157c8ac8
125602025-04-30T06:26:53.514ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125612025-04-30T06:26:53.517ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
125622025-04-30T06:26:53.518ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125632025-04-30T06:26:53.518ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125642025-04-30T06:26:53.519ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125652025-04-30T06:26:53.519ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 318573190 } })
125662025-04-30T06:26:53.519ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
125672025-04-30T06:26:53.520ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 15 be 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65060 req_id = 31536bc5-bbec-4332-9849-28d1e9fc118e
125682025-04-30T06:26:53.520ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5566,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65060 req_id = 31536bc5-bbec-4332-9849-28d1e9fc118e
125692025-04-30T06:26:53.520ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125702025-04-30T06:26:53.521ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65060 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 31536bc5-bbec-4332-9849-28d1e9fc118e
125712025-04-30T06:26:53.521ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
125722025-04-30T06:26:53.588ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
125732025-04-30T06:26:53.588ZDEBGtest_disk_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
125742025-04-30T06:26:53.589ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125752025-04-30T06:26:53.589ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125762025-04-30T06:26:53.589ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125772025-04-30T06:26:53.589ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125782025-04-30T06:26:53.592ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125792025-04-30T06:26:53.647ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
125802025-04-30T06:26:53.647ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
125812025-04-30T06:26:53.647ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:44059 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
125822025-04-30T06:26:53.647ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
125832025-04-30T06:26:53.648ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:44059 req_id = da517302-edd5-4365-8640-b6a0b25eea6f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
125842025-04-30T06:26:53.649ZINFOtest_disk_metrics (dropshot): request completed latency_us = 687 local_addr = [::1]:50989 method = GET remote_addr = [::1]:44059 req_id = da517302-edd5-4365-8640-b6a0b25eea6f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
125852025-04-30T06:26:53.649ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125862025-04-30T06:26:53.654ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125872025-04-30T06:26:53.654ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125882025-04-30T06:26:53.655ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125892025-04-30T06:26:53.655ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125902025-04-30T06:26:53.656ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 05 1e 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:65252 req_id = ee315faf-0bc0-47d2-a49e-b93dc33d4077
125912025-04-30T06:26:53.656ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1310,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65252 req_id = ee315faf-0bc0-47d2-a49e-b93dc33d4077
125922025-04-30T06:26:53.656ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
125932025-04-30T06:26:53.657ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125942025-04-30T06:26:53.657ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65252 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ee315faf-0bc0-47d2-a49e-b93dc33d4077
125952025-04-30T06:26:53.657ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "da517302-edd5-4365-8640-b6a0b25eea6f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
125962025-04-30T06:26:53.659ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
125972025-04-30T06:26:53.659ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
125982025-04-30T06:26:53.660ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
125992025-04-30T06:26:53.662ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126002025-04-30T06:26:53.662ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126012025-04-30T06:26:53.663ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126022025-04-30T06:26:53.663ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126032025-04-30T06:26:53.663ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126042025-04-30T06:26:53.664ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 463782551 } })
126052025-04-30T06:26:53.665ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
126062025-04-30T06:26:53.665ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 77 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e w...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54010 req_id = 53812476-a239-4e14-b5f0-cdbb15df26b0
126072025-04-30T06:26:53.666ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30487,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54010 req_id = 53812476-a239-4e14-b5f0-cdbb15df26b0
126082025-04-30T06:26:53.666ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126092025-04-30T06:26:53.666ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54010 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 53812476-a239-4e14-b5f0-cdbb15df26b0
126102025-04-30T06:26:53.667ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126112025-04-30T06:26:53.667ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126122025-04-30T06:26:53.667ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126132025-04-30T06:26:53.668ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
126142025-04-30T06:26:53.732ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
126152025-04-30T06:26:53.732ZDEBGtest_disk_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
126162025-04-30T06:26:53.733ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126172025-04-30T06:26:53.738ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126182025-04-30T06:26:53.739ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126192025-04-30T06:26:53.739ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126202025-04-30T06:26:53.741ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126212025-04-30T06:26:53.784ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
126222025-04-30T06:26:53.786ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
126232025-04-30T06:26:53.786ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
126242025-04-30T06:26:53.786ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 595.288512ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
126252025-04-30T06:26:53.786ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
126262025-04-30T06:26:53.786ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126272025-04-30T06:26:53.786ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126282025-04-30T06:26:53.786ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126292025-04-30T06:26:53.786ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126302025-04-30T06:26:53.787ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126312025-04-30T06:26:53.791ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126322025-04-30T06:26:53.802ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126332025-04-30T06:26:53.802ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126342025-04-30T06:26:53.803ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126352025-04-30T06:26:53.803ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126362025-04-30T06:26:53.807ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
126372025-04-30T06:26:53.807ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
126382025-04-30T06:26:53.808ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: aa 58 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .X..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49670 req_id = 240b9d5b-c903-4c08-a1f1-75ef3a837b37
126392025-04-30T06:26:53.809ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43608,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49670 req_id = 240b9d5b-c903-4c08-a1f1-75ef3a837b37
126402025-04-30T06:26:53.809ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
126412025-04-30T06:26:53.809ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126422025-04-30T06:26:53.809ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49670 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 240b9d5b-c903-4c08-a1f1-75ef3a837b37
126432025-04-30T06:26:53.810ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 609094244 } })
126442025-04-30T06:26:53.810ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
126452025-04-30T06:26:53.810ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 90 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53764 req_id = 5bbc17f4-308e-4ecb-ae2e-d6124bbf6645
126462025-04-30T06:26:53.811ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36887,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53764 req_id = 5bbc17f4-308e-4ecb-ae2e-d6124bbf6645
126472025-04-30T06:26:53.814ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126482025-04-30T06:26:53.814ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53764 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5bbc17f4-308e-4ecb-ae2e-d6124bbf6645
126492025-04-30T06:26:53.876ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
126502025-04-30T06:26:53.877ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126512025-04-30T06:26:53.877ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126522025-04-30T06:26:53.877ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126532025-04-30T06:26:53.877ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126542025-04-30T06:26:53.880ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126552025-04-30T06:26:53.881ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1b ab 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51759 req_id = 39ba36ba-250f-4386-afed-11ed10980671
126562025-04-30T06:26:53.881ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7083,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51759 req_id = 39ba36ba-250f-4386-afed-11ed10980671
126572025-04-30T06:26:53.881ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
126582025-04-30T06:26:53.881ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7083, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51759 req_id = 39ba36ba-250f-4386-afed-11ed10980671
126592025-04-30T06:26:53.882ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
126602025-04-30T06:26:53.882ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
126612025-04-30T06:26:53.882ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
126622025-04-30T06:26:53.883ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126632025-04-30T06:26:53.895ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126642025-04-30T06:26:53.895ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126652025-04-30T06:26:53.895ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126662025-04-30T06:26:53.897ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126672025-04-30T06:26:53.899ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "84fc58b3-264e-49a4-ab22-e076d9f2db4b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126682025-04-30T06:26:53.900ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
126692025-04-30T06:26:53.900ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126702025-04-30T06:26:53.900ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126712025-04-30T06:26:53.901ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126722025-04-30T06:26:53.903ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2d0f223-ae7c-45e0-8022-31dd413341df", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126732025-04-30T06:26:53.903ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
126742025-04-30T06:26:53.906ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "059f4a5d-3158-4ce2-ac77-8bf7ee3dc096", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126752025-04-30T06:26:53.907ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
126762025-04-30T06:26:53.909ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2402527-ce20-437a-a988-96b55bbb54d2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126772025-04-30T06:26:53.909ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
126782025-04-30T06:26:53.911ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "463f3493-30c6-4390-944c-177fca4db053", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126792025-04-30T06:26:53.911ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
126802025-04-30T06:26:53.913ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40b303d5-b781-489c-82bb-bfc88b504fa2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126812025-04-30T06:26:53.914ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
126822025-04-30T06:26:53.916ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02af514a-5f31-4592-95a3-5c628ed7b63d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126832025-04-30T06:26:53.917ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
126842025-04-30T06:26:53.920ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca52fcc4-3ec5-4c2b-a7d1-b6ed9afdab40", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126852025-04-30T06:26:53.920ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
126862025-04-30T06:26:53.921ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6c06a1-684f-4a62-a7be-29840bb9e6b2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126872025-04-30T06:26:53.921ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
126882025-04-30T06:26:53.923ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b542a8ca-6889-4231-82c3-35fb3ddbc0ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126892025-04-30T06:26:53.923ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
126902025-04-30T06:26:53.924ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b3b1e66-3e5a-40bf-ab74-a3bda9993f97", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126912025-04-30T06:26:53.924ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
126922025-04-30T06:26:53.925ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "005e07bd-1b4d-4e09-ba6e-214e3aba62d7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126932025-04-30T06:26:53.925ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
126942025-04-30T06:26:53.927ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "507372de-7502-499c-a7de-3bce67f893f4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126952025-04-30T06:26:53.927ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
126962025-04-30T06:26:53.931ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
126972025-04-30T06:26:53.945ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "583d5693-10a6-47cc-94af-db123c86e6ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
126982025-04-30T06:26:53.945ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
126992025-04-30T06:26:53.961ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41e08780-b80c-4147-bef2-5b1cdc9319d0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127002025-04-30T06:26:53.961ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
127012025-04-30T06:26:53.965ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65b742f4-f738-4967-b5e9-e3f90cf211b2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127022025-04-30T06:26:53.965ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
127032025-04-30T06:26:53.968ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "712993ef-409e-4258-b95c-3bbccc1960a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127042025-04-30T06:26:53.968ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
127052025-04-30T06:26:53.970ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b61f6af-b790-4627-8e08-2f52af19c5df", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127062025-04-30T06:26:53.970ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
127072025-04-30T06:26:53.973ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc23313e-3deb-4de2-bf77-a78100da8e37", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127082025-04-30T06:26:53.973ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
127092025-04-30T06:26:53.975ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d80be92-b6db-4563-b76a-9f54bc088235", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127102025-04-30T06:26:53.976ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
127112025-04-30T06:26:53.977ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2de3dc8-51e1-4f9f-a74d-0ddaf803da6d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127122025-04-30T06:26:53.978ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
127132025-04-30T06:26:53.979ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "108eafbf-4b7b-47ff-936d-af8a47b2b315", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127142025-04-30T06:26:53.979ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
127152025-04-30T06:26:53.982ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "015f08c2-05eb-4862-a39e-1cc6e89cc256", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127162025-04-30T06:26:53.982ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
127172025-04-30T06:26:53.982ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b14de07b-1d1a-4d92-9b6e-d88a97c4696c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127182025-04-30T06:26:53.982ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
127192025-04-30T06:26:53.982ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127202025-04-30T06:26:53.982ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127212025-04-30T06:26:53.982ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127222025-04-30T06:26:53.982ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127232025-04-30T06:26:53.983ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127242025-04-30T06:26:53.984ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127252025-04-30T06:26:53.985ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127262025-04-30T06:26:53.986ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127272025-04-30T06:26:53.997ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127282025-04-30T06:26:53.997ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127292025-04-30T06:26:53.997ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127302025-04-30T06:26:53.997ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127312025-04-30T06:26:53.998ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127322025-04-30T06:26:54.000ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127332025-04-30T06:26:54.000ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127342025-04-30T06:26:54.001ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127352025-04-30T06:26:54.004ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127362025-04-30T06:26:54.004ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127372025-04-30T06:26:54.006ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127382025-04-30T06:26:54.007ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "326ab10b-6645-40c0-811c-30bedfeb11ff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127392025-04-30T06:26:54.008ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
127402025-04-30T06:26:54.016ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4d40c2c-a92d-41f9-aa0c-196f3c001841", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127412025-04-30T06:26:54.016ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
127422025-04-30T06:26:54.021ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d8e1b5b-ad48-42ef-92c7-1d09dbab679d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127432025-04-30T06:26:54.022ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
127442025-04-30T06:26:54.030ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "915cc9c9-5443-4803-b5b0-2635a2218f09", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127452025-04-30T06:26:54.030ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
127462025-04-30T06:26:54.034ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a176272e-f083-497f-978a-2e4a0b86b04d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127472025-04-30T06:26:54.037ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
127482025-04-30T06:26:54.051ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb4a4a54-6dcf-4204-9d5e-648a579a5a7d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127492025-04-30T06:26:54.051ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
127502025-04-30T06:26:54.055ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1171351-9ad9-4850-85b7-bec0bac95dbb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127512025-04-30T06:26:54.055ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
127522025-04-30T06:26:54.057ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
127532025-04-30T06:26:54.061ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e0b0c8d-c4c8-418f-b7e5-115131ddc619", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127542025-04-30T06:26:54.062ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
127552025-04-30T06:26:54.063ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "501af22c-3dc1-4589-8b5e-5353902b007d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127562025-04-30T06:26:54.063ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
127572025-04-30T06:26:54.064ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb393a5f-ee25-4ad2-9de1-cbc6bed9b467", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127582025-04-30T06:26:54.064ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
127592025-04-30T06:26:54.067ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "336ed0b5-e3df-477b-931a-880f15df267d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127602025-04-30T06:26:54.067ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
127612025-04-30T06:26:54.070ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9194e554-cb04-4fee-a921-b802afbe913f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127622025-04-30T06:26:54.070ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
127632025-04-30T06:26:54.071ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb75857-c34b-43dc-93bf-f453d75b6ff5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127642025-04-30T06:26:54.071ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
127652025-04-30T06:26:54.083ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f8c4867-5f80-44ad-b788-ddd6f054dda8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127662025-04-30T06:26:54.083ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
127672025-04-30T06:26:54.093ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e66b4c7-557e-4111-98c9-a7db82b9c9f5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127682025-04-30T06:26:54.095ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
127692025-04-30T06:26:54.097ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59c2c897-ea3a-4303-836c-680b2c73f0ec", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127702025-04-30T06:26:54.097ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
127712025-04-30T06:26:54.101ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87783464-4c8b-4d36-aed3-fa5862359dff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127722025-04-30T06:26:54.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
127732025-04-30T06:26:54.102ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9b1d71a-b3d9-46d1-b3a0-85e01c763e38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127742025-04-30T06:26:54.103ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
127752025-04-30T06:26:54.104ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aceed049-138d-4130-9241-6ec08c8a99f0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127762025-04-30T06:26:54.105ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
127772025-04-30T06:26:54.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d43d97a-4c48-4c75-89f5-3079db49dbf6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127782025-04-30T06:26:54.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
127792025-04-30T06:26:54.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "979758f3-a89b-40c1-a8ef-6bb8488ba941", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
127802025-04-30T06:26:54.107ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
127812025-04-30T06:26:54.107ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127822025-04-30T06:26:54.107ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127832025-04-30T06:26:54.108ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127842025-04-30T06:26:54.121ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127852025-04-30T06:26:54.121ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127862025-04-30T06:26:54.121ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127872025-04-30T06:26:54.121ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127882025-04-30T06:26:54.121ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127892025-04-30T06:26:54.122ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127902025-04-30T06:26:54.135ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127912025-04-30T06:26:54.156ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127922025-04-30T06:26:54.156ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127932025-04-30T06:26:54.156ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127942025-04-30T06:26:54.157ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127952025-04-30T06:26:54.158ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127962025-04-30T06:26:54.160ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
127972025-04-30T06:26:54.161ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
127982025-04-30T06:26:54.161ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127992025-04-30T06:26:54.161ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128002025-04-30T06:26:54.161ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128012025-04-30T06:26:54.161ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128022025-04-30T06:26:54.179ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128032025-04-30T06:26:54.179ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128042025-04-30T06:26:54.180ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128052025-04-30T06:26:54.180ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128062025-04-30T06:26:54.180ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128072025-04-30T06:26:54.181ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
128082025-04-30T06:26:54.181ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
128092025-04-30T06:26:54.184ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128102025-04-30T06:26:54.189ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128112025-04-30T06:26:54.189ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128122025-04-30T06:26:54.189ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128132025-04-30T06:26:54.190ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128142025-04-30T06:26:54.192ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
128152025-04-30T06:26:54.192ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
128162025-04-30T06:26:54.196ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
128172025-04-30T06:26:54.196ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128182025-04-30T06:26:54.261ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
128192025-04-30T06:26:54.261ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
128202025-04-30T06:26:54.262ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:52267 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
128212025-04-30T06:26:54.263ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:52267 req_id = 1fec30fe-9748-48d2-b2da-5eaef98c063b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128222025-04-30T06:26:54.263ZINFOtest_disk_metrics (dropshot): request completed latency_us = 634 local_addr = [::1]:50989 method = GET remote_addr = [::1]:52267 req_id = 1fec30fe-9748-48d2-b2da-5eaef98c063b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128232025-04-30T06:26:54.263ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fec30fe-9748-48d2-b2da-5eaef98c063b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
128242025-04-30T06:26:54.266ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128252025-04-30T06:26:54.266ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128262025-04-30T06:26:54.266ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
128272025-04-30T06:26:54.269ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128282025-04-30T06:26:54.269ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128292025-04-30T06:26:54.269ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128302025-04-30T06:26:54.269ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128312025-04-30T06:26:54.270ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128322025-04-30T06:26:54.275ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128332025-04-30T06:26:54.284ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128342025-04-30T06:26:54.284ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128352025-04-30T06:26:54.284ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128362025-04-30T06:26:54.285ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128372025-04-30T06:26:54.286ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
128382025-04-30T06:26:54.329ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
128392025-04-30T06:26:54.349ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
128402025-04-30T06:26:54.349ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
128412025-04-30T06:26:54.349ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 563.20515ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
128422025-04-30T06:26:54.350ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
128432025-04-30T06:26:54.350ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128442025-04-30T06:26:54.350ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128452025-04-30T06:26:54.350ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128462025-04-30T06:26:54.350ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128472025-04-30T06:26:54.350ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128482025-04-30T06:26:54.353ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128492025-04-30T06:26:54.358ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128502025-04-30T06:26:54.359ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128512025-04-30T06:26:54.359ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128522025-04-30T06:26:54.359ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128532025-04-30T06:26:54.362ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
128542025-04-30T06:26:54.362ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
128552025-04-30T06:26:54.365ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 44 d8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 D............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54725 req_id = a374ba73-31f5-4b56-8ebc-ff8cb53f1a8a
128562025-04-30T06:26:54.365ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17624,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54725 req_id = a374ba73-31f5-4b56-8ebc-ff8cb53f1a8a
128572025-04-30T06:26:54.365ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
128582025-04-30T06:26:54.367ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
128592025-04-30T06:26:54.367ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54725 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a374ba73-31f5-4b56-8ebc-ff8cb53f1a8a
128602025-04-30T06:26:54.367ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3279, tv_nsec: 166752552 } })
128612025-04-30T06:26:54.367ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
128622025-04-30T06:26:54.368ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a6 a1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55024 req_id = 3fbc3a93-b34b-4058-a808-d3e7883321e3
128632025-04-30T06:26:54.368ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42657,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55024 req_id = 3fbc3a93-b34b-4058-a808-d3e7883321e3
128642025-04-30T06:26:54.368ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
128652025-04-30T06:26:54.368ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55024 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3fbc3a93-b34b-4058-a808-d3e7883321e3
128662025-04-30T06:26:54.502ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
128672025-04-30T06:26:54.502ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128682025-04-30T06:26:54.502ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128692025-04-30T06:26:54.502ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128702025-04-30T06:26:54.502ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128712025-04-30T06:26:54.503ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128722025-04-30T06:26:54.503ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f1 e1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56710 req_id = 149095af-7033-4d63-989e-30d44b76aa2e
128732025-04-30T06:26:54.503ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61921,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56710 req_id = 149095af-7033-4d63-989e-30d44b76aa2e
128742025-04-30T06:26:54.504ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
128752025-04-30T06:26:54.504ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61921, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56710 req_id = 149095af-7033-4d63-989e-30d44b76aa2e
128762025-04-30T06:26:54.504ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
128772025-04-30T06:26:54.504ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
128782025-04-30T06:26:54.504ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
128792025-04-30T06:26:54.506ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128802025-04-30T06:26:54.512ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128812025-04-30T06:26:54.512ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128822025-04-30T06:26:54.512ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128832025-04-30T06:26:54.513ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128842025-04-30T06:26:54.513ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "36f59085-82f5-443d-80d0-93f86a8b8ff9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
128852025-04-30T06:26:54.514ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
128862025-04-30T06:26:54.517ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1739eff2-b58f-491f-8790-e4131374765a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
128872025-04-30T06:26:54.518ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
128882025-04-30T06:26:54.518ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128892025-04-30T06:26:54.518ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128902025-04-30T06:26:54.518ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128912025-04-30T06:26:54.521ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "666c39f0-6a55-4638-95ac-a797cc438b73", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
128922025-04-30T06:26:54.521ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
128932025-04-30T06:26:54.525ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "962961fe-f99f-41a1-ae7b-37373ed08222", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
128942025-04-30T06:26:54.525ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
128952025-04-30T06:26:54.527ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fc80337-1fa6-4a2d-8ef7-602aa032a9e0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
128962025-04-30T06:26:54.527ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
128972025-04-30T06:26:54.532ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c4b788c-b28b-41d0-bc60-6f0bb04452b7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
128982025-04-30T06:26:54.532ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
128992025-04-30T06:26:54.535ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3b31e8f-4088-453e-8040-78918a16760a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129002025-04-30T06:26:54.536ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
129012025-04-30T06:26:54.538ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92e274ff-402d-4b7d-a3fc-661f9d266bf2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129022025-04-30T06:26:54.539ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
129032025-04-30T06:26:54.540ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5faf435a-ecce-402d-8e61-cb0f583f22a8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129042025-04-30T06:26:54.540ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
129052025-04-30T06:26:54.543ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd6468a3-28f3-49f6-861c-f01e9ca2853d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129062025-04-30T06:26:54.544ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
129072025-04-30T06:26:54.546ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0db2bf1d-33d0-45aa-baa4-a17629d45db9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129082025-04-30T06:26:54.547ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
129092025-04-30T06:26:54.548ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
129102025-04-30T06:26:54.548ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52fa6672-4480-4bde-8ccd-4a124e0f01e7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129112025-04-30T06:26:54.549ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
129122025-04-30T06:26:54.550ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b79887-1773-46af-a042-e7e21828b9fb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129132025-04-30T06:26:54.550ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
129142025-04-30T06:26:54.552ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e8c7fc8-cb18-428d-abe1-e2d5c8853b41", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129152025-04-30T06:26:54.553ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
129162025-04-30T06:26:54.555ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51e07eab-f56e-4e3d-ab5c-06e9e8201d60", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129172025-04-30T06:26:54.555ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
129182025-04-30T06:26:54.558ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce1c3406-88ff-43a1-8769-798f666a5470", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129192025-04-30T06:26:54.558ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
129202025-04-30T06:26:54.559ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db46f05f-4557-4dbe-8483-d9f9b7c91a1b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129212025-04-30T06:26:54.560ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
129222025-04-30T06:26:54.562ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9821888d-8f03-4972-9be2-680d96cb5a33", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129232025-04-30T06:26:54.562ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
129242025-04-30T06:26:54.564ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "312556d0-f6d2-4373-9fdd-1962d7d2c345", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129252025-04-30T06:26:54.564ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
129262025-04-30T06:26:54.565ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cc8900c-a548-492a-8d4c-21c5ef92d396", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129272025-04-30T06:26:54.565ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
129282025-04-30T06:26:54.568ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5053dc78-c1f8-490d-a40d-ae70f33da5b2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129292025-04-30T06:26:54.568ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
129302025-04-30T06:26:54.569ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11675d92-e05a-4a0b-b1f4-876534e8927f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129312025-04-30T06:26:54.569ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
129322025-04-30T06:26:54.570ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9337d06f-a23b-41cf-b535-80c0fa56e8d3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129332025-04-30T06:26:54.570ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
129342025-04-30T06:26:54.573ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "071c17a5-76a9-475d-a55e-743899769493", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129352025-04-30T06:26:54.573ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
129362025-04-30T06:26:54.575ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "886abfbd-65d3-44c8-801a-1ef45234dc1a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129372025-04-30T06:26:54.575ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
129382025-04-30T06:26:54.579ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd0a73cc-6889-4d55-934d-75b391dae16b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129392025-04-30T06:26:54.579ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
129402025-04-30T06:26:54.579ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ac60d80-11ea-4681-a879-89e003e57de3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129412025-04-30T06:26:54.579ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
129422025-04-30T06:26:54.587ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f3c01c1-ef35-4908-982e-a6bf2b3d31b1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129432025-04-30T06:26:54.589ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
129442025-04-30T06:26:54.593ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d043943b-2773-482d-bb6d-ea2a7b45b590", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129452025-04-30T06:26:54.593ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
129462025-04-30T06:26:54.597ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b31ba38-2a56-4312-9b1b-7b214da9d027", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129472025-04-30T06:26:54.597ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
129482025-04-30T06:26:54.599ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5aea9f6-078a-4d02-8a64-17c2290efa6d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129492025-04-30T06:26:54.599ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
129502025-04-30T06:26:54.599ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129512025-04-30T06:26:54.599ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129522025-04-30T06:26:54.599ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129532025-04-30T06:26:54.600ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
129542025-04-30T06:26:54.601ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129552025-04-30T06:26:54.604ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7faad00e-1bf5-4c3a-9ffa-5823db88f488", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129562025-04-30T06:26:54.604ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
129572025-04-30T06:26:54.607ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129582025-04-30T06:26:54.613ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129592025-04-30T06:26:54.614ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129602025-04-30T06:26:54.614ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
129612025-04-30T06:26:54.614ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129622025-04-30T06:26:54.616ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3cd81ff-3340-45f3-8abf-4f47644f4605", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129632025-04-30T06:26:54.616ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
129642025-04-30T06:26:54.618ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129652025-04-30T06:26:54.618ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
129662025-04-30T06:26:54.619ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129672025-04-30T06:26:54.620ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2aa188-d58c-4be2-ada6-fb616980e33e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129682025-04-30T06:26:54.620ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
129692025-04-30T06:26:54.621ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ab0e11-31f7-4ad2-a165-eedb34b03094", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129702025-04-30T06:26:54.621ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
129712025-04-30T06:26:54.623ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a4f6835-93ae-4538-82c1-a42f8b2aa687", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129722025-04-30T06:26:54.623ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
129732025-04-30T06:26:54.626ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd8f9641-3c7f-4a9e-8a22-b053e3310e7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129742025-04-30T06:26:54.626ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
129752025-04-30T06:26:54.628ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bb8190f-464b-4651-8733-84ff0f6d6ff9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129762025-04-30T06:26:54.628ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
129772025-04-30T06:26:54.630ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7917f07-1fac-43f2-bfd3-e538eb372528", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129782025-04-30T06:26:54.631ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
129792025-04-30T06:26:54.632ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa010161-a9f5-4932-beb1-a1ef596c34ed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129802025-04-30T06:26:54.632ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
129812025-04-30T06:26:54.634ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "953693de-c621-453c-b0b1-b6f3b8e119ee", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129822025-04-30T06:26:54.634ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
129832025-04-30T06:26:54.635ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee1ebd45-1de1-4be9-b333-9f4d654e9d87", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129842025-04-30T06:26:54.635ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
129852025-04-30T06:26:54.637ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f580b1-0cdc-43fd-874a-90e7a556d80c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129862025-04-30T06:26:54.637ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
129872025-04-30T06:26:54.637ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f52c656-d885-4341-a49c-7a51d6c9fdfd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129882025-04-30T06:26:54.638ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
129892025-04-30T06:26:54.638ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46c18339-af5d-4103-8e8e-cecfc8671f65", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
129902025-04-30T06:26:54.639ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129912025-04-30T06:26:54.639ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129922025-04-30T06:26:54.639ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129932025-04-30T06:26:54.639ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129942025-04-30T06:26:54.639ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129952025-04-30T06:26:54.646ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129962025-04-30T06:26:54.650ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
129972025-04-30T06:26:54.650ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129982025-04-30T06:26:54.650ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129992025-04-30T06:26:54.650ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130002025-04-30T06:26:54.650ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130012025-04-30T06:26:54.650ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130022025-04-30T06:26:54.652ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130032025-04-30T06:26:54.656ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130042025-04-30T06:26:54.656ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130052025-04-30T06:26:54.656ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130062025-04-30T06:26:54.656ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130072025-04-30T06:26:54.660ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
130082025-04-30T06:26:54.720ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
130092025-04-30T06:26:54.720ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
130102025-04-30T06:26:54.720ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130112025-04-30T06:26:54.720ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130122025-04-30T06:26:54.720ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130132025-04-30T06:26:54.721ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130142025-04-30T06:26:54.721ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130152025-04-30T06:26:54.721ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:57948 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130162025-04-30T06:26:54.722ZDEBGtest_disk_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
130172025-04-30T06:26:54.722ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
130182025-04-30T06:26:54.724ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
130192025-04-30T06:26:54.724ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:57948 req_id = 38b62d0c-0d42-441c-af41-a27c2cbbc22a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130202025-04-30T06:26:54.725ZINFOtest_disk_metrics (dropshot): request completed latency_us = 470 local_addr = [::1]:50989 method = GET remote_addr = [::1]:57948 req_id = 38b62d0c-0d42-441c-af41-a27c2cbbc22a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130212025-04-30T06:26:54.725ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "38b62d0c-0d42-441c-af41-a27c2cbbc22a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
130222025-04-30T06:26:54.730ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
130232025-04-30T06:26:54.730ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
130242025-04-30T06:26:54.730ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
130252025-04-30T06:26:54.732ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
130262025-04-30T06:26:54.732ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130272025-04-30T06:26:54.732ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130282025-04-30T06:26:54.732ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130292025-04-30T06:26:54.733ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130302025-04-30T06:26:54.733ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130312025-04-30T06:26:54.739ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130322025-04-30T06:26:54.739ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130332025-04-30T06:26:54.740ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130342025-04-30T06:26:54.740ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130352025-04-30T06:26:54.742ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130362025-04-30T06:26:54.751ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
130372025-04-30T06:26:54.752ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130382025-04-30T06:26:54.752ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130392025-04-30T06:26:54.754ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130402025-04-30T06:26:54.756ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130412025-04-30T06:26:54.756ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130422025-04-30T06:26:54.759ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130432025-04-30T06:26:54.801ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
130442025-04-30T06:26:54.803ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
130452025-04-30T06:26:54.803ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 453.438646ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
130462025-04-30T06:26:54.803ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
130472025-04-30T06:26:54.803ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
130482025-04-30T06:26:54.804ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130492025-04-30T06:26:54.804ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130502025-04-30T06:26:54.804ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130512025-04-30T06:26:54.806ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130522025-04-30T06:26:54.812ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
130532025-04-30T06:26:54.812ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130542025-04-30T06:26:54.817ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
130552025-04-30T06:26:54.818ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130562025-04-30T06:26:54.818ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130572025-04-30T06:26:54.819ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130582025-04-30T06:26:54.823ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
130592025-04-30T06:26:54.823ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
130602025-04-30T06:26:54.824ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d1 61 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .a..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50061 req_id = c6e9d962-ac7c-4502-949e-214759fd146a
130612025-04-30T06:26:54.824ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53601,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50061 req_id = c6e9d962-ac7c-4502-949e-214759fd146a
130622025-04-30T06:26:54.824ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
130632025-04-30T06:26:54.824ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130642025-04-30T06:26:54.824ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50061 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c6e9d962-ac7c-4502-949e-214759fd146a
130652025-04-30T06:26:54.825ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3279, tv_nsec: 624133723 } })
130662025-04-30T06:26:54.825ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
130672025-04-30T06:26:54.825ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 94 ac 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57841 req_id = 4d6f0f68-8fb1-44b8-a8f2-5256cef8a011
130682025-04-30T06:26:54.825ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38060,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57841 req_id = 4d6f0f68-8fb1-44b8-a8f2-5256cef8a011
130692025-04-30T06:26:54.825ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130702025-04-30T06:26:54.825ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57841 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4d6f0f68-8fb1-44b8-a8f2-5256cef8a011
130712025-04-30T06:26:54.889ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
130722025-04-30T06:26:54.890ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130732025-04-30T06:26:54.890ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130742025-04-30T06:26:54.890ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130752025-04-30T06:26:54.890ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130762025-04-30T06:26:54.891ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130772025-04-30T06:26:54.891ZDEBGtest_disk_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
130782025-04-30T06:26:54.892ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
130792025-04-30T06:26:54.893ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
130802025-04-30T06:26:54.893ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
130812025-04-30T06:26:54.894ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130822025-04-30T06:26:54.894ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = adebe201-5696-4193-a363-e0ed4c520466 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130832025-04-30T06:26:54.895ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 38 1d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 8............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60944 req_id = b7c530dd-6e6a-49b9-a449-caefa3d15cf2
130842025-04-30T06:26:54.895ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14365,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60944 req_id = b7c530dd-6e6a-49b9-a449-caefa3d15cf2
130852025-04-30T06:26:54.895ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
130862025-04-30T06:26:54.896ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14365, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60944 req_id = b7c530dd-6e6a-49b9-a449-caefa3d15cf2
130872025-04-30T06:26:54.897ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = adebe201-5696-4193-a363-e0ed4c520466 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130882025-04-30T06:26:54.897ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = adebe201-5696-4193-a363-e0ed4c520466 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130892025-04-30T06:26:54.899ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = adebe201-5696-4193-a363-e0ed4c520466 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130902025-04-30T06:26:54.899ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
130912025-04-30T06:26:54.899ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
130922025-04-30T06:26:54.900ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
130932025-04-30T06:26:54.900ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130942025-04-30T06:26:54.907ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130952025-04-30T06:26:54.907ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130962025-04-30T06:26:54.907ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130972025-04-30T06:26:54.907ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130982025-04-30T06:26:54.908ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 14058 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = adebe201-5696-4193-a363-e0ed4c520466 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130992025-04-30T06:26:54.909ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "adebe201-5696-4193-a363-e0ed4c520466", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131002025-04-30T06:26:54.910ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131012025-04-30T06:26:54.910ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131022025-04-30T06:26:54.910ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
131032025-04-30T06:26:54.910ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131042025-04-30T06:26:54.910ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eeea55ac-040e-44d3-8577-67d6c90db25f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131052025-04-30T06:26:54.911ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eeea55ac-040e-44d3-8577-67d6c90db25f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131062025-04-30T06:26:54.911ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eeea55ac-040e-44d3-8577-67d6c90db25f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131072025-04-30T06:26:54.912ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eeea55ac-040e-44d3-8577-67d6c90db25f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131082025-04-30T06:26:54.912ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "28d8628a-28c1-4a7b-9084-a6357a142858", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131092025-04-30T06:26:54.913ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
131102025-04-30T06:26:54.915ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97a9b509-b7b5-4c93-a53f-803aaa67b19b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131112025-04-30T06:26:54.915ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
131122025-04-30T06:26:54.916ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 5225 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = eeea55ac-040e-44d3-8577-67d6c90db25f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131132025-04-30T06:26:54.916ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeea55ac-040e-44d3-8577-67d6c90db25f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131142025-04-30T06:26:54.916ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
131152025-04-30T06:26:54.916ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131162025-04-30T06:26:54.916ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
131172025-04-30T06:26:54.919ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9b959f-b82c-40ac-82a6-c4f8bf556aef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131182025-04-30T06:26:54.919ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
131192025-04-30T06:26:54.922ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b97c7545-e48e-4e83-afe6-3611aee5b87a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131202025-04-30T06:26:54.922ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
131212025-04-30T06:26:54.924ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b646a6fd-d317-416c-a2ac-bfa61ba9f9f4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131222025-04-30T06:26:54.924ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
131232025-04-30T06:26:54.927ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5da97800-0cf1-4192-944a-8560f2be8e53", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131242025-04-30T06:26:54.927ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
131252025-04-30T06:26:54.953ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6976643-4ec0-48d8-91d8-e22ed8a65e85", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131262025-04-30T06:26:54.953ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
131272025-04-30T06:26:54.965ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
131282025-04-30T06:26:54.980ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fa17008-a95d-4221-9b7b-cc1b23325b3e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131292025-04-30T06:26:54.980ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
131302025-04-30T06:26:54.982ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02547fe5-e396-4764-a0ee-c70514fe2979", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131312025-04-30T06:26:54.982ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
131322025-04-30T06:26:54.984ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "177bde47-9891-407d-90cd-6577f2453a19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131332025-04-30T06:26:54.984ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
131342025-04-30T06:26:54.985ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131352025-04-30T06:26:54.985ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131362025-04-30T06:26:54.990ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131372025-04-30T06:26:54.992ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131382025-04-30T06:26:54.992ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131392025-04-30T06:26:54.992ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131402025-04-30T06:26:54.992ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9b4bab-dfef-4a5e-b3c2-c18fcd4d9755", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131412025-04-30T06:26:54.992ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
131422025-04-30T06:26:54.994ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca3e57e1-ab6b-4056-8d34-7fd2f48a3215", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131432025-04-30T06:26:54.994ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
131442025-04-30T06:26:54.999ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9134447f-e454-4730-8836-a37689811bcf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131452025-04-30T06:26:54.999ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
131462025-04-30T06:26:55.003ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80469186-15ad-4dd4-a808-e582bce6210d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131472025-04-30T06:26:55.003ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
131482025-04-30T06:26:55.005ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f82a204f-0e25-4c27-a774-ab188e976c5f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131492025-04-30T06:26:55.006ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
131502025-04-30T06:26:55.009ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6388eeb-369b-4e60-8476-f17f0df3c9b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131512025-04-30T06:26:55.009ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
131522025-04-30T06:26:55.013ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "736853ad-291a-4e14-8cc6-41c6c45e17ed", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131532025-04-30T06:26:55.013ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
131542025-04-30T06:26:55.022ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131552025-04-30T06:26:55.022ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131562025-04-30T06:26:55.022ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131572025-04-30T06:26:55.022ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
131582025-04-30T06:26:55.024ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131592025-04-30T06:26:55.030ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131602025-04-30T06:26:55.037ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131612025-04-30T06:26:55.037ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131622025-04-30T06:26:55.037ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
131632025-04-30T06:26:55.038ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131642025-04-30T06:26:55.040ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131652025-04-30T06:26:55.040ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
131662025-04-30T06:26:55.041ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131672025-04-30T06:26:55.041ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdf6c0bf-0008-44a2-8346-ca37c62251a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131682025-04-30T06:26:55.041ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
131692025-04-30T06:26:55.046ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e870d99-b703-4c23-96e9-69fa4c3f1844", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131702025-04-30T06:26:55.047ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
131712025-04-30T06:26:55.050ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f9bccd2-09e0-4af4-8510-192f666d15c9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131722025-04-30T06:26:55.050ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
131732025-04-30T06:26:55.051ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16c0cc5f-f344-430e-960c-3aa812b3a92c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131742025-04-30T06:26:55.052ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
131752025-04-30T06:26:55.053ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f99be561-717e-454f-893f-d76b96c07936", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131762025-04-30T06:26:55.053ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
131772025-04-30T06:26:55.057ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4512c546-ab14-4e1a-a5ec-de3a1ff7130d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131782025-04-30T06:26:55.057ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
131792025-04-30T06:26:55.058ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62d54a4e-720a-462c-893c-4e19447e4588", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131802025-04-30T06:26:55.058ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
131812025-04-30T06:26:55.068ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf17ed28-8074-425f-98d4-19770faf55b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131822025-04-30T06:26:55.068ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
131832025-04-30T06:26:55.071ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00c84145-9373-40ba-a4ac-228e867eb24f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131842025-04-30T06:26:55.072ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
131852025-04-30T06:26:55.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9a73b5e-c986-4fee-a674-bd1f4be355e4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131862025-04-30T06:26:55.076ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
131872025-04-30T06:26:55.083ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
131882025-04-30T06:26:55.085ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55484043-6843-4cc1-a799-cb6dd7e54e72", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131892025-04-30T06:26:55.085ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
131902025-04-30T06:26:55.088ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a0137d8-cc54-4ce1-9527-de7a433f8ceb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131912025-04-30T06:26:55.088ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
131922025-04-30T06:26:55.091ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "613553a4-a91a-45fd-9ffb-a30a5096c2cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131932025-04-30T06:26:55.091ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
131942025-04-30T06:26:55.094ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd8cdcf8-9417-4d86-b590-4f271dc314df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131952025-04-30T06:26:55.094ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
131962025-04-30T06:26:55.095ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f440b094-f762-4409-b0ea-6914adc18261", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131972025-04-30T06:26:55.095ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
131982025-04-30T06:26:55.096ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "758c2f77-1248-477a-ab5e-c8cc6c7ab831", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
131992025-04-30T06:26:55.097ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
132002025-04-30T06:26:55.099ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3b12db-7105-44b9-8d58-b07dabaf7e53", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132012025-04-30T06:26:55.100ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
132022025-04-30T06:26:55.101ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad9bb18-490c-471f-b365-20d809ee32df", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132032025-04-30T06:26:55.102ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
132042025-04-30T06:26:55.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "416f09db-079b-4a1f-879c-1c900a9ce9dc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132052025-04-30T06:26:55.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
132062025-04-30T06:26:55.109ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dedb791e-0f19-463e-913b-8f8f5086d788", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132072025-04-30T06:26:55.109ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
132082025-04-30T06:26:55.111ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "680047d0-537d-4b1a-ac26-d42f10a0a57f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132092025-04-30T06:26:55.111ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
132102025-04-30T06:26:55.113ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c20104-6c28-46e7-86d6-819b3991e43e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132112025-04-30T06:26:55.113ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
132122025-04-30T06:26:55.120ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1b5873b-4163-49c7-9184-60b731759647", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132132025-04-30T06:26:55.121ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
132142025-04-30T06:26:55.123ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d855f7-1343-452b-9946-76d1ca233478", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132152025-04-30T06:26:55.123ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
132162025-04-30T06:26:55.124ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53ccb23c-fbaf-4dfb-8ffe-5cf313090562", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132172025-04-30T06:26:55.124ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
132182025-04-30T06:26:55.126ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a6630eb-7662-4072-aa2e-99ceeb9d707f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132192025-04-30T06:26:55.127ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
132202025-04-30T06:26:55.128ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0be7d5a4-403b-42eb-b42c-8975371b7f17", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132212025-04-30T06:26:55.128ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
132222025-04-30T06:26:55.129ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea96e31e-3e4e-4025-b7be-aa497a254606", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
132232025-04-30T06:26:55.130ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
132242025-04-30T06:26:55.130ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132252025-04-30T06:26:55.130ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132262025-04-30T06:26:55.130ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132272025-04-30T06:26:55.130ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132282025-04-30T06:26:55.134ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132292025-04-30T06:26:55.134ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132302025-04-30T06:26:55.134ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132312025-04-30T06:26:55.135ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132322025-04-30T06:26:55.136ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132332025-04-30T06:26:55.136ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132342025-04-30T06:26:55.140ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
132352025-04-30T06:26:55.140ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
132362025-04-30T06:26:55.140ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132372025-04-30T06:26:55.140ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132382025-04-30T06:26:55.140ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132392025-04-30T06:26:55.141ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132402025-04-30T06:26:55.142ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132412025-04-30T06:26:55.148ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132422025-04-30T06:26:55.148ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132432025-04-30T06:26:55.148ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132442025-04-30T06:26:55.149ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132452025-04-30T06:26:55.150ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132462025-04-30T06:26:55.153ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
132472025-04-30T06:26:55.153ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132482025-04-30T06:26:55.153ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132492025-04-30T06:26:55.154ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132502025-04-30T06:26:55.156ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132512025-04-30T06:26:55.156ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132522025-04-30T06:26:55.156ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132532025-04-30T06:26:55.266ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
132542025-04-30T06:26:55.267ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
132552025-04-30T06:26:55.275ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:60681 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
132562025-04-30T06:26:55.276ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132572025-04-30T06:26:55.276ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132582025-04-30T06:26:55.281ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
132592025-04-30T06:26:55.284ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
132602025-04-30T06:26:55.286ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:60681 req_id = a7231488-df16-44e6-9b2b-84ffe1d15c57 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
132612025-04-30T06:26:55.287ZINFOtest_disk_metrics (dropshot): request completed latency_us = 840 local_addr = [::1]:50989 method = GET remote_addr = [::1]:60681 req_id = a7231488-df16-44e6-9b2b-84ffe1d15c57 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
132622025-04-30T06:26:55.287ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7231488-df16-44e6-9b2b-84ffe1d15c57", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
132632025-04-30T06:26:55.290ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
132642025-04-30T06:26:55.290ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
132652025-04-30T06:26:55.290ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
132662025-04-30T06:26:55.294ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132672025-04-30T06:26:55.294ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132682025-04-30T06:26:55.294ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132692025-04-30T06:26:55.294ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132702025-04-30T06:26:55.296ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132712025-04-30T06:26:55.297ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
132722025-04-30T06:26:55.301ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
132732025-04-30T06:26:55.309ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132742025-04-30T06:26:55.315ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132752025-04-30T06:26:55.315ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132762025-04-30T06:26:55.316ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132772025-04-30T06:26:55.316ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132782025-04-30T06:26:55.318ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
132792025-04-30T06:26:55.318ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132802025-04-30T06:26:55.353ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
132812025-04-30T06:26:55.354ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
132822025-04-30T06:26:55.355ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c7 08 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:60194 req_id = cd1a97d5-68e5-4761-8215-025c88a6446d
132832025-04-30T06:26:55.355ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50952,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60194 req_id = cd1a97d5-68e5-4761-8215-025c88a6446d
132842025-04-30T06:26:55.355ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
132852025-04-30T06:26:55.356ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132862025-04-30T06:26:55.356ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60194 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cd1a97d5-68e5-4761-8215-025c88a6446d
132872025-04-30T06:26:55.357ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3280, tv_nsec: 155918650 } })
132882025-04-30T06:26:55.357ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
132892025-04-30T06:26:55.357ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c1 58 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .X..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59154 req_id = 950907f5-92a9-46b8-8d58-de6b6da2a481
132902025-04-30T06:26:55.357ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49496,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59154 req_id = 950907f5-92a9-46b8-8d58-de6b6da2a481
132912025-04-30T06:26:55.357ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132922025-04-30T06:26:55.357ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59154 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 950907f5-92a9-46b8-8d58-de6b6da2a481
132932025-04-30T06:26:55.358ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
132942025-04-30T06:26:55.421ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
132952025-04-30T06:26:55.421ZDEBGtest_disk_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
132962025-04-30T06:26:55.421ZDEBGtest_disk_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
132972025-04-30T06:26:55.421ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
132982025-04-30T06:26:55.422ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
132992025-04-30T06:26:55.423ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133002025-04-30T06:26:55.423ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133012025-04-30T06:26:55.423ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133022025-04-30T06:26:55.423ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133032025-04-30T06:26:55.423ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133042025-04-30T06:26:55.424ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
133052025-04-30T06:26:55.424ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
133062025-04-30T06:26:55.424ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
133072025-04-30T06:26:55.424ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 621.042487ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
133082025-04-30T06:26:55.424ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
133092025-04-30T06:26:55.424ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133102025-04-30T06:26:55.424ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133112025-04-30T06:26:55.424ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133122025-04-30T06:26:55.424ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133132025-04-30T06:26:55.425ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133142025-04-30T06:26:55.427ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133152025-04-30T06:26:55.435ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133162025-04-30T06:26:55.435ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133172025-04-30T06:26:55.435ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133182025-04-30T06:26:55.439ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133192025-04-30T06:26:55.439ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133202025-04-30T06:26:55.452ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133212025-04-30T06:26:55.452ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133222025-04-30T06:26:55.452ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133232025-04-30T06:26:55.452ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133242025-04-30T06:26:55.457ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133252025-04-30T06:26:55.457ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133262025-04-30T06:26:55.458ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133272025-04-30T06:26:55.461ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
133282025-04-30T06:26:55.461ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
133292025-04-30T06:26:55.463ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5f 5c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 _\\..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59179 req_id = 71636525-205f-4e4e-af72-f48c8ebaa8bb
133302025-04-30T06:26:55.464ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24412,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59179 req_id = 71636525-205f-4e4e-af72-f48c8ebaa8bb
133312025-04-30T06:26:55.464ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
133322025-04-30T06:26:55.464ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133332025-04-30T06:26:55.465ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59179 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 71636525-205f-4e4e-af72-f48c8ebaa8bb
133342025-04-30T06:26:55.469ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3280, tv_nsec: 268166427 } })
133352025-04-30T06:26:55.469ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
133362025-04-30T06:26:55.471ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 78 18 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e x...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65454 req_id = 9ca740a7-d86f-41bb-8b6e-7059f7edfbe9
133372025-04-30T06:26:55.471ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30744,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65454 req_id = 9ca740a7-d86f-41bb-8b6e-7059f7edfbe9
133382025-04-30T06:26:55.471ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133392025-04-30T06:26:55.471ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65454 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9ca740a7-d86f-41bb-8b6e-7059f7edfbe9
133402025-04-30T06:26:55.527ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
133412025-04-30T06:26:55.528ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c2 1f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60200 req_id = fcf6e884-8964-4312-924e-90afa3a71197
133422025-04-30T06:26:55.528ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49695,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60200 req_id = fcf6e884-8964-4312-924e-90afa3a71197
133432025-04-30T06:26:55.528ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
133442025-04-30T06:26:55.528ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49695, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60200 req_id = fcf6e884-8964-4312-924e-90afa3a71197
133452025-04-30T06:26:55.528ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
133462025-04-30T06:26:55.528ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
133472025-04-30T06:26:55.528ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
133482025-04-30T06:26:55.530ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "040d440d-16d3-4e5c-80fd-765ded939247", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133492025-04-30T06:26:55.530ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
133502025-04-30T06:26:55.532ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02466444-9bd1-4094-a666-ed4f68feebe2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133512025-04-30T06:26:55.532ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
133522025-04-30T06:26:55.534ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecef006a-87fc-4356-aa27-9aeee04cc911", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133532025-04-30T06:26:55.534ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
133542025-04-30T06:26:55.537ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d77220d-06d6-4d15-ac4c-cae4b7d353a8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133552025-04-30T06:26:55.537ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
133562025-04-30T06:26:55.539ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa80a953-740d-4ab7-8d3a-2d789e7cf64c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133572025-04-30T06:26:55.539ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
133582025-04-30T06:26:55.543ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56e60f36-0055-41a9-aa78-1c5bcf9cda69", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133592025-04-30T06:26:55.547ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
133602025-04-30T06:26:55.556ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
133612025-04-30T06:26:55.558ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7ba510c-bbfe-4259-ad44-53edc5773e7b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133622025-04-30T06:26:55.558ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
133632025-04-30T06:26:55.561ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7654b6ee-111a-478b-a4a1-7a3d6ae6d557", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133642025-04-30T06:26:55.562ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
133652025-04-30T06:26:55.564ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98841828-5f96-4191-a4d8-46b8f14a103f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133662025-04-30T06:26:55.564ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
133672025-04-30T06:26:55.566ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6995af1e-5179-4a64-a294-2d5a539e2c4f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133682025-04-30T06:26:55.566ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
133692025-04-30T06:26:55.568ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75dc4db9-3888-4141-bd23-a414d8ceb532", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133702025-04-30T06:26:55.568ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
133712025-04-30T06:26:55.569ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c733521e-b700-4297-83cc-8daf1d5a210a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133722025-04-30T06:26:55.569ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
133732025-04-30T06:26:55.571ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "783342e6-71b3-4fed-b1d5-080c2140527d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133742025-04-30T06:26:55.571ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
133752025-04-30T06:26:55.575ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fb9ae5c-93af-4c7d-a28e-e1662dc35a04", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133762025-04-30T06:26:55.587ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
133772025-04-30T06:26:55.597ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd4b646-9f85-4f7d-9e5b-419b4cb334be", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133782025-04-30T06:26:55.598ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
133792025-04-30T06:26:55.607ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133802025-04-30T06:26:55.607ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133812025-04-30T06:26:55.607ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133822025-04-30T06:26:55.607ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133832025-04-30T06:26:55.608ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133842025-04-30T06:26:55.609ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dbae3d3-5f76-4a7d-99bc-9393acaf9bbf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133852025-04-30T06:26:55.609ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
133862025-04-30T06:26:55.611ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94afeb06-fca1-430f-8d71-7f69fd1c4789", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133872025-04-30T06:26:55.612ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
133882025-04-30T06:26:55.612ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133892025-04-30T06:26:55.619ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133902025-04-30T06:26:55.619ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133912025-04-30T06:26:55.619ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133922025-04-30T06:26:55.619ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133932025-04-30T06:26:55.623ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133942025-04-30T06:26:55.623ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133952025-04-30T06:26:55.623ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133962025-04-30T06:26:55.624ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a58f796-e9a5-4b7f-a2a6-a14122c190d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133972025-04-30T06:26:55.624ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
133982025-04-30T06:26:55.628ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ffba9f-c8b4-43fa-8011-1736e4a32e86", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
133992025-04-30T06:26:55.628ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
134002025-04-30T06:26:55.629ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8096e9f-c3dc-41ba-a278-8646426bb4d6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134012025-04-30T06:26:55.629ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
134022025-04-30T06:26:55.631ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34aac3ad-4d23-4624-8a14-007a8bc7fed2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134032025-04-30T06:26:55.631ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
134042025-04-30T06:26:55.633ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "874e02be-fcfc-41e6-9781-dff5a563ac8f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134052025-04-30T06:26:55.633ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
134062025-04-30T06:26:55.635ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b20b80ef-4f30-4e26-879b-703ff7fa8f55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134072025-04-30T06:26:55.635ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
134082025-04-30T06:26:55.636ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68095dfe-014b-4d90-91a5-8c2a0f0d1c46", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134092025-04-30T06:26:55.637ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
134102025-04-30T06:26:55.639ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "385519ae-c280-4c29-8e23-870b6a02eb91", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134112025-04-30T06:26:55.639ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
134122025-04-30T06:26:55.642ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37d0ed37-4591-4f49-8fcd-7662e9438d09", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134132025-04-30T06:26:55.642ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
134142025-04-30T06:26:55.645ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28c972a4-af08-4cb9-8a42-8c62bcd83195", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134152025-04-30T06:26:55.645ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
134162025-04-30T06:26:55.648ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45cd3b9e-3ba3-4de2-a9c7-9bcab59e7e82", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134172025-04-30T06:26:55.648ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
134182025-04-30T06:26:55.651ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f46f4eac-6f51-4444-ad04-9f9d282ab52d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134192025-04-30T06:26:55.652ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
134202025-04-30T06:26:55.653ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e99eca8-1061-4e0b-8666-7322ec938eee", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134212025-04-30T06:26:55.654ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
134222025-04-30T06:26:55.655ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe457813-ce79-4ff5-93b5-2818fdd50c0e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134232025-04-30T06:26:55.655ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
134242025-04-30T06:26:55.656ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "327da7de-0471-4a86-a393-d6cd3bc3c5a3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134252025-04-30T06:26:55.656ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
134262025-04-30T06:26:55.658ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "776f2ff9-21b9-432d-839c-e686df6d566f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134272025-04-30T06:26:55.658ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
134282025-04-30T06:26:55.659ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d7501cc-e404-421e-bd8b-768a0936ed28", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134292025-04-30T06:26:55.659ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
134302025-04-30T06:26:55.666ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
134312025-04-30T06:26:55.667ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0171081f-ca1e-4cd3-a7fd-72afb5441144", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134322025-04-30T06:26:55.667ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
134332025-04-30T06:26:55.676ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "170e833b-45fc-44ec-a42a-5b11c2d8c142", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134342025-04-30T06:26:55.677ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
134352025-04-30T06:26:55.681ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eade0ce-1772-4fe7-8271-a987511ef869", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134362025-04-30T06:26:55.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
134372025-04-30T06:26:55.683ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17ba88eb-bb72-4ffe-82b7-0b164d88c291", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134382025-04-30T06:26:55.684ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
134392025-04-30T06:26:55.689ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d5ad25-4319-4d1f-895b-09a84c887fb1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134402025-04-30T06:26:55.689ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
134412025-04-30T06:26:55.694ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dddd4ae-2c3a-4bc3-83e0-1bedcbbab7bb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134422025-04-30T06:26:55.694ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
134432025-04-30T06:26:55.695ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea8c752b-b086-4aae-9d14-0087716ed516", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134442025-04-30T06:26:55.696ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
134452025-04-30T06:26:55.697ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e5323ae-9f6d-41c8-91c6-7e11384de444", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134462025-04-30T06:26:55.697ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
134472025-04-30T06:26:55.698ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08643688-0754-4d9f-992c-3bafba4bf2ef", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134482025-04-30T06:26:55.698ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
134492025-04-30T06:26:55.701ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "314b4f70-b05c-4446-a4dd-5dcae21cdf8a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134502025-04-30T06:26:55.701ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
134512025-04-30T06:26:55.702ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58df4426-67ce-4a1c-b799-ad7bd2aedac9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134522025-04-30T06:26:55.702ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134532025-04-30T06:26:55.703ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134542025-04-30T06:26:55.703ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134552025-04-30T06:26:55.703ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134562025-04-30T06:26:55.704ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134572025-04-30T06:26:55.707ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134582025-04-30T06:26:55.713ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134592025-04-30T06:26:55.714ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134602025-04-30T06:26:55.714ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134612025-04-30T06:26:55.714ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134622025-04-30T06:26:55.714ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134632025-04-30T06:26:55.714ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134642025-04-30T06:26:55.717ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134652025-04-30T06:26:55.723ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134662025-04-30T06:26:55.723ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134672025-04-30T06:26:55.723ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134682025-04-30T06:26:55.724ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134692025-04-30T06:26:55.724ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134702025-04-30T06:26:55.724ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134712025-04-30T06:26:55.724ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134722025-04-30T06:26:55.724ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134732025-04-30T06:26:55.725ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134742025-04-30T06:26:55.790ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
134752025-04-30T06:26:55.791ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
134762025-04-30T06:26:55.792ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134772025-04-30T06:26:55.813ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134782025-04-30T06:26:55.814ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134792025-04-30T06:26:55.814ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134802025-04-30T06:26:55.818ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134812025-04-30T06:26:55.818ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:63950 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
134822025-04-30T06:26:55.819ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:63950 req_id = 0ccbd230-0f07-42d8-87e2-f00180a8e075 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134832025-04-30T06:26:55.820ZINFOtest_disk_metrics (dropshot): request completed latency_us = 963 local_addr = [::1]:50989 method = GET remote_addr = [::1]:63950 req_id = 0ccbd230-0f07-42d8-87e2-f00180a8e075 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134842025-04-30T06:26:55.821ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ccbd230-0f07-42d8-87e2-f00180a8e075", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
134852025-04-30T06:26:55.827ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134862025-04-30T06:26:55.827ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134872025-04-30T06:26:55.827ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
134882025-04-30T06:26:55.829ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134892025-04-30T06:26:55.829ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134902025-04-30T06:26:55.829ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134912025-04-30T06:26:55.829ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134922025-04-30T06:26:55.830ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134932025-04-30T06:26:55.831ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134942025-04-30T06:26:55.831ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134952025-04-30T06:26:55.832ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134962025-04-30T06:26:55.836ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134972025-04-30T06:26:55.842ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134982025-04-30T06:26:55.842ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134992025-04-30T06:26:55.842ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135002025-04-30T06:26:55.843ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135012025-04-30T06:26:55.883ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
135022025-04-30T06:26:55.883ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
135032025-04-30T06:26:55.883ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 459.212241ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
135042025-04-30T06:26:55.884ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
135052025-04-30T06:26:55.884ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135062025-04-30T06:26:55.884ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135072025-04-30T06:26:55.884ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135082025-04-30T06:26:55.884ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135092025-04-30T06:26:55.885ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135102025-04-30T06:26:55.890ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135112025-04-30T06:26:55.896ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135122025-04-30T06:26:55.896ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135132025-04-30T06:26:55.896ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135142025-04-30T06:26:55.897ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135152025-04-30T06:26:55.898ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
135162025-04-30T06:26:55.902ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
135172025-04-30T06:26:55.902ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
135182025-04-30T06:26:55.902ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 44 a8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 D............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62778 req_id = 896bc170-eee9-4010-b2a4-46033d2cc648
135192025-04-30T06:26:55.902ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17576,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62778 req_id = 896bc170-eee9-4010-b2a4-46033d2cc648
135202025-04-30T06:26:55.902ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
135212025-04-30T06:26:55.903ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
135222025-04-30T06:26:55.903ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62778 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 896bc170-eee9-4010-b2a4-46033d2cc648
135232025-04-30T06:26:55.903ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3280, tv_nsec: 702321713 } })
135242025-04-30T06:26:55.903ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
135252025-04-30T06:26:55.903ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0b 75 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .u..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65112 req_id = 0f0b3bb9-ffa4-4b58-a890-51defb7bbfdb
135262025-04-30T06:26:55.903ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2933,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65112 req_id = 0f0b3bb9-ffa4-4b58-a890-51defb7bbfdb
135272025-04-30T06:26:55.904ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
135282025-04-30T06:26:55.904ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65112 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0f0b3bb9-ffa4-4b58-a890-51defb7bbfdb
135292025-04-30T06:26:55.967ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
135302025-04-30T06:26:55.968ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135312025-04-30T06:26:55.968ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135322025-04-30T06:26:55.968ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135332025-04-30T06:26:55.969ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
135342025-04-30T06:26:55.971ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135352025-04-30T06:26:55.974ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f8 ee 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57548 req_id = 8cfa727a-3a92-4c58-9281-d86d8f2e1f23
135362025-04-30T06:26:55.974ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63726,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57548 req_id = 8cfa727a-3a92-4c58-9281-d86d8f2e1f23
135372025-04-30T06:26:55.974ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
135382025-04-30T06:26:55.974ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63726, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57548 req_id = 8cfa727a-3a92-4c58-9281-d86d8f2e1f23
135392025-04-30T06:26:55.975ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
135402025-04-30T06:26:55.975ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
135412025-04-30T06:26:55.975ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
135422025-04-30T06:26:55.976ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135432025-04-30T06:26:55.982ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135442025-04-30T06:26:55.982ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135452025-04-30T06:26:55.982ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
135462025-04-30T06:26:55.983ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135472025-04-30T06:26:55.987ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135482025-04-30T06:26:55.987ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135492025-04-30T06:26:55.991ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135502025-04-30T06:26:55.991ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135512025-04-30T06:26:55.991ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135522025-04-30T06:26:55.991ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135532025-04-30T06:26:55.992ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135542025-04-30T06:26:55.992ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
135552025-04-30T06:26:55.992ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135562025-04-30T06:26:56.001ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "122243b6-f6dc-4cd2-988c-d32ed800794d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135572025-04-30T06:26:56.001ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
135582025-04-30T06:26:56.015ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a33f86f5-f677-4156-98f1-825e8115851d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135592025-04-30T06:26:56.015ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
135602025-04-30T06:26:56.028ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc7a7b6b-a0f6-4c7a-a5a8-38dd5e985404", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135612025-04-30T06:26:56.028ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
135622025-04-30T06:26:56.033ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "202604de-b8de-4fee-8556-0c86d58c2cb7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135632025-04-30T06:26:56.033ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
135642025-04-30T06:26:56.036ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "270802e0-ca28-4a4a-9b3d-1ff85df8327e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135652025-04-30T06:26:56.036ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
135662025-04-30T06:26:56.039ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91068fe5-9bce-4205-8427-51f8c37e6b14", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135672025-04-30T06:26:56.039ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
135682025-04-30T06:26:56.041ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "917d72f9-3637-4f52-982d-77246c656302", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135692025-04-30T06:26:56.041ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
135702025-04-30T06:26:56.046ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22df4919-0194-4784-9eb3-fcd0dc5a5293", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135712025-04-30T06:26:56.046ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
135722025-04-30T06:26:56.047ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
135732025-04-30T06:26:56.050ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2852b70-1c73-49f7-b01d-b025484fb6c7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135742025-04-30T06:26:56.050ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
135752025-04-30T06:26:56.051ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a977448e-af69-44f1-92bc-d6bfb4652a70", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135762025-04-30T06:26:56.051ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
135772025-04-30T06:26:56.052ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91ce17ce-0c1f-4020-804f-a7be00683464", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135782025-04-30T06:26:56.052ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
135792025-04-30T06:26:56.053ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eadb82f5-f87a-461e-a3b5-9c769468742e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135802025-04-30T06:26:56.053ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
135812025-04-30T06:26:56.055ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de652843-1612-44aa-b6f4-b6f6e1fe4475", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135822025-04-30T06:26:56.055ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
135832025-04-30T06:26:56.057ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1364db2c-9d35-4bcd-b968-3c5c34a494af", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135842025-04-30T06:26:56.058ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
135852025-04-30T06:26:56.059ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73af710b-3c86-4394-acf5-f3f66541c49e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135862025-04-30T06:26:56.059ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
135872025-04-30T06:26:56.060ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6db69a35-94e9-4f36-929f-534a9109b111", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135882025-04-30T06:26:56.061ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
135892025-04-30T06:26:56.062ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d96dbded-c3a8-44ec-8935-fb185612409d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135902025-04-30T06:26:56.062ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
135912025-04-30T06:26:56.064ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11b55725-619c-4582-bdb0-0c620bcac908", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135922025-04-30T06:26:56.064ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
135932025-04-30T06:26:56.066ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75f98666-521b-4e6a-b268-bade5340a7f1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135942025-04-30T06:26:56.066ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
135952025-04-30T06:26:56.067ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a76e9519-2752-4efd-8885-2c8de3505bde", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135962025-04-30T06:26:56.067ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
135972025-04-30T06:26:56.068ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b847f693-f562-473e-9370-1593b591c5e4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
135982025-04-30T06:26:56.068ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
135992025-04-30T06:26:56.069ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9c230d9-dee9-477b-a5c5-625392257184", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136002025-04-30T06:26:56.069ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
136012025-04-30T06:26:56.070ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36849132-3baf-4db5-9270-1247a4da5644", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136022025-04-30T06:26:56.070ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
136032025-04-30T06:26:56.072ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0e79f74-1642-42a9-bd3f-8fa0d18d9c85", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136042025-04-30T06:26:56.072ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
136052025-04-30T06:26:56.075ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57b5179b-3d4b-4bf1-b8fc-4f4d84c0986f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136062025-04-30T06:26:56.075ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
136072025-04-30T06:26:56.077ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aee661f-6978-4d7b-ad1a-f4a594603970", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136082025-04-30T06:26:56.077ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
136092025-04-30T06:26:56.079ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7315fbf1-52f5-455e-a601-5da286c83dde", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136102025-04-30T06:26:56.079ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
136112025-04-30T06:26:56.080ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdbebbe4-07c8-475a-9507-6056e759ec0f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136122025-04-30T06:26:56.081ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
136132025-04-30T06:26:56.083ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0aa06dd-4fb6-404c-aea1-c4dda1c762a8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136142025-04-30T06:26:56.083ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
136152025-04-30T06:26:56.085ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "609692b4-deef-4b6d-9442-0f9c2c520fd6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136162025-04-30T06:26:56.085ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
136172025-04-30T06:26:56.086ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0388efde-c0da-483b-ba52-e8926b015d1a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136182025-04-30T06:26:56.086ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
136192025-04-30T06:26:56.087ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c71a6eec-0061-4306-b6a4-1a637f01d968", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136202025-04-30T06:26:56.087ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
136212025-04-30T06:26:56.088ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31033584-0778-4cf1-9c2c-42de37231a4b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136222025-04-30T06:26:56.088ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
136232025-04-30T06:26:56.089ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f62997e-9b66-4ecc-8830-349f858ca31b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136242025-04-30T06:26:56.089ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
136252025-04-30T06:26:56.091ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "baaf7fba-1f1e-4822-9bed-0f535649527a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136262025-04-30T06:26:56.091ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
136272025-04-30T06:26:56.093ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20372e22-8b58-4a5e-a47c-d98b51ad09c9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136282025-04-30T06:26:56.094ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
136292025-04-30T06:26:56.096ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a4632b3-278b-4436-b3cf-46c6927716f1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136302025-04-30T06:26:56.096ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
136312025-04-30T06:26:56.097ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136322025-04-30T06:26:56.097ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136332025-04-30T06:26:56.097ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136342025-04-30T06:26:56.097ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
136352025-04-30T06:26:56.098ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136362025-04-30T06:26:56.100ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c6c9c87-2291-46ed-9604-378b6c20e13d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136372025-04-30T06:26:56.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
136382025-04-30T06:26:56.101ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136392025-04-30T06:26:56.107ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136402025-04-30T06:26:56.107ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136412025-04-30T06:26:56.107ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
136422025-04-30T06:26:56.107ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136432025-04-30T06:26:56.110ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136442025-04-30T06:26:56.110ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
136452025-04-30T06:26:56.110ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136462025-04-30T06:26:56.112ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32d448d9-dc11-4d09-b875-1178bd0ca411", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136472025-04-30T06:26:56.112ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
136482025-04-30T06:26:56.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86e43a42-ffb5-4065-b698-3438f7454d41", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136492025-04-30T06:26:56.115ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
136502025-04-30T06:26:56.117ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e71e82fc-d15e-4fe3-aa3a-8c00a418db85", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136512025-04-30T06:26:56.118ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
136522025-04-30T06:26:56.119ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e63a8c2-3f8c-4d6f-b3fb-5c32e77dbee6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136532025-04-30T06:26:56.119ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
136542025-04-30T06:26:56.120ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "709b7235-6e35-475c-9c6c-56cee356e929", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136552025-04-30T06:26:56.121ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
136562025-04-30T06:26:56.122ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1e7c876-a935-4608-8ec8-6ff9d00d9e11", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136572025-04-30T06:26:56.122ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
136582025-04-30T06:26:56.124ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1a560a5-fbd5-483b-8510-eb3dee0d4bb3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136592025-04-30T06:26:56.124ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136602025-04-30T06:26:56.124ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136612025-04-30T06:26:56.124ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136622025-04-30T06:26:56.124ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136632025-04-30T06:26:56.125ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136642025-04-30T06:26:56.128ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136652025-04-30T06:26:56.132ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136662025-04-30T06:26:56.132ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136672025-04-30T06:26:56.132ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136682025-04-30T06:26:56.132ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136692025-04-30T06:26:56.132ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136702025-04-30T06:26:56.133ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136712025-04-30T06:26:56.136ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136722025-04-30T06:26:56.141ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136732025-04-30T06:26:56.142ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136742025-04-30T06:26:56.142ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136752025-04-30T06:26:56.142ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136762025-04-30T06:26:56.221ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
136772025-04-30T06:26:56.221ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
136782025-04-30T06:26:56.223ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:32822 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
136792025-04-30T06:26:56.223ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136802025-04-30T06:26:56.223ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136812025-04-30T06:26:56.224ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
136822025-04-30T06:26:56.225ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
136832025-04-30T06:26:56.226ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
136842025-04-30T06:26:56.227ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:32822 req_id = a4579419-34b6-4e4c-a176-a79f5994105a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136852025-04-30T06:26:56.228ZINFOtest_disk_metrics (dropshot): request completed latency_us = 723 local_addr = [::1]:50989 method = GET remote_addr = [::1]:32822 req_id = a4579419-34b6-4e4c-a176-a79f5994105a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136862025-04-30T06:26:56.229ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4579419-34b6-4e4c-a176-a79f5994105a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:55 GMT"} })
136872025-04-30T06:26:56.232ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136882025-04-30T06:26:56.232ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136892025-04-30T06:26:56.232ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
136902025-04-30T06:26:56.234ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
136912025-04-30T06:26:56.234ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136922025-04-30T06:26:56.234ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136932025-04-30T06:26:56.234ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136942025-04-30T06:26:56.235ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136952025-04-30T06:26:56.236ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
136962025-04-30T06:26:56.236ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
136972025-04-30T06:26:56.238ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136982025-04-30T06:26:56.242ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136992025-04-30T06:26:56.242ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137002025-04-30T06:26:56.242ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137012025-04-30T06:26:56.243ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137022025-04-30T06:26:56.243ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
137032025-04-30T06:26:56.243ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137042025-04-30T06:26:56.274ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
137052025-04-30T06:26:56.274ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
137062025-04-30T06:26:56.274ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 390.397712ms iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
137072025-04-30T06:26:56.274ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
137082025-04-30T06:26:56.274ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137092025-04-30T06:26:56.274ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137102025-04-30T06:26:56.274ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137112025-04-30T06:26:56.274ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137122025-04-30T06:26:56.275ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137132025-04-30T06:26:56.278ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137142025-04-30T06:26:56.278ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137152025-04-30T06:26:56.278ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137162025-04-30T06:26:56.278ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
137172025-04-30T06:26:56.280ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137182025-04-30T06:26:56.280ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137192025-04-30T06:26:56.292ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
137202025-04-30T06:26:56.292ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137212025-04-30T06:26:56.292ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137222025-04-30T06:26:56.293ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137232025-04-30T06:26:56.295ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137242025-04-30T06:26:56.302ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137252025-04-30T06:26:56.302ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137262025-04-30T06:26:56.302ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
137272025-04-30T06:26:56.302ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137282025-04-30T06:26:56.303ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
137292025-04-30T06:26:56.303ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
137302025-04-30T06:26:56.304ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4c ac 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 L............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61557 req_id = 2c7b5606-8407-4d4b-8d11-a685cfd72fc9
137312025-04-30T06:26:56.304ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19628,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61557 req_id = 2c7b5606-8407-4d4b-8d11-a685cfd72fc9
137322025-04-30T06:26:56.304ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
137332025-04-30T06:26:56.304ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137342025-04-30T06:26:56.306ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61557 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2c7b5606-8407-4d4b-8d11-a685cfd72fc9
137352025-04-30T06:26:56.306ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137362025-04-30T06:26:56.306ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
137372025-04-30T06:26:56.307ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137382025-04-30T06:26:56.307ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3281, tv_nsec: 106342698 } })
137392025-04-30T06:26:56.307ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
137402025-04-30T06:26:56.308ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b6 e5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61181 req_id = 18730b95-b72c-4470-a749-cd93d7e2b323
137412025-04-30T06:26:56.308ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46821,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61181 req_id = 18730b95-b72c-4470-a749-cd93d7e2b323
137422025-04-30T06:26:56.308ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137432025-04-30T06:26:56.308ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61181 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 18730b95-b72c-4470-a749-cd93d7e2b323
137442025-04-30T06:26:56.373ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
137452025-04-30T06:26:56.375ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
137462025-04-30T06:26:56.375ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8c 7f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54948 req_id = ee535ddd-ede0-48ef-82e8-4c19f3cf5da8
137472025-04-30T06:26:56.376ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35967,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54948 req_id = ee535ddd-ede0-48ef-82e8-4c19f3cf5da8
137482025-04-30T06:26:56.376ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
137492025-04-30T06:26:56.376ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35967, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54948 req_id = ee535ddd-ede0-48ef-82e8-4c19f3cf5da8
137502025-04-30T06:26:56.377ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
137512025-04-30T06:26:56.377ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
137522025-04-30T06:26:56.377ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
137532025-04-30T06:26:56.381ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "28f795bc-c37c-44dd-be95-77bc822fd395", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137542025-04-30T06:26:56.381ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
137552025-04-30T06:26:56.383ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cdf5b95-62bb-4e76-bbfd-b4d15e411e9e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137562025-04-30T06:26:56.384ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
137572025-04-30T06:26:56.386ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf65f078-0502-4967-ac21-bf8436184c36", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137582025-04-30T06:26:56.387ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
137592025-04-30T06:26:56.391ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58accd9f-85e8-4454-ae43-8ce3acfd2984", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137602025-04-30T06:26:56.391ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
137612025-04-30T06:26:56.395ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf23c2f1-53b4-4343-a1f1-0956b9fa966a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137622025-04-30T06:26:56.395ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
137632025-04-30T06:26:56.400ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78236c11-42f4-45b1-889c-84767d949745", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137642025-04-30T06:26:56.400ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
137652025-04-30T06:26:56.402ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6087e0fc-6b19-4098-ad12-c87880c93f66", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137662025-04-30T06:26:56.403ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
137672025-04-30T06:26:56.407ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
137682025-04-30T06:26:56.408ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "588c4afe-dfd2-4250-ad01-748b0e75e1b2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137692025-04-30T06:26:56.408ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
137702025-04-30T06:26:56.409ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6102536b-1322-41cc-a1f0-f8eb785f953b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137712025-04-30T06:26:56.409ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
137722025-04-30T06:26:56.410ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13b5bae6-aab3-4ee6-8406-13e4624cef4c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137732025-04-30T06:26:56.410ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
137742025-04-30T06:26:56.412ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dfaecbd-15a8-42c0-b385-c3ccb4a04c93", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137752025-04-30T06:26:56.412ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
137762025-04-30T06:26:56.413ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61219c36-cbfd-4174-935e-de450ee877c5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137772025-04-30T06:26:56.414ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
137782025-04-30T06:26:56.416ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f7ef5f0-e29c-4bae-8ae2-ea777c974e37", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137792025-04-30T06:26:56.416ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
137802025-04-30T06:26:56.420ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45d452f2-1b61-4a34-b958-41ba153f1ad7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137812025-04-30T06:26:56.420ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
137822025-04-30T06:26:56.428ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00aac14d-3d94-4e8c-a8e6-78dc7a7871e7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137832025-04-30T06:26:56.428ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
137842025-04-30T06:26:56.432ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2a5306b-4429-4f65-bb7c-1f6a14a29096", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137852025-04-30T06:26:56.434ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
137862025-04-30T06:26:56.441ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37debf69-3f51-49b8-9f44-5ce0fd822b96", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137872025-04-30T06:26:56.441ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
137882025-04-30T06:26:56.448ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3280d7b-832a-44eb-b749-7faf89ce01f7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137892025-04-30T06:26:56.448ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
137902025-04-30T06:26:56.451ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6d45599-4c34-4bd2-92bf-58e52e176fa7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137912025-04-30T06:26:56.452ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
137922025-04-30T06:26:56.453ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "921f89d0-3716-4373-95c2-a21e0aeb8dce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137932025-04-30T06:26:56.454ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
137942025-04-30T06:26:56.456ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "327879ab-7d57-4bab-a985-94b4f2b29279", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
137952025-04-30T06:26:56.460ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
137962025-04-30T06:26:56.460ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137972025-04-30T06:26:56.461ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137982025-04-30T06:26:56.461ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137992025-04-30T06:26:56.461ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138002025-04-30T06:26:56.463ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138012025-04-30T06:26:56.467ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0977d29a-9d15-47e5-849d-42e82d6a4aef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138022025-04-30T06:26:56.468ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
138032025-04-30T06:26:56.470ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138042025-04-30T06:26:56.476ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138052025-04-30T06:26:56.476ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138062025-04-30T06:26:56.476ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138072025-04-30T06:26:56.477ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138082025-04-30T06:26:56.478ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0412085-1bc9-41c9-935d-ff04a0e99113", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138092025-04-30T06:26:56.478ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
138102025-04-30T06:26:56.480ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138112025-04-30T06:26:56.480ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138122025-04-30T06:26:56.480ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138132025-04-30T06:26:56.482ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1319889d-36ab-4a07-be4c-e157d21d0dce", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138142025-04-30T06:26:56.482ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
138152025-04-30T06:26:56.487ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dfdfcbd-74c6-40b4-8976-e9f6ae64ddda", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138162025-04-30T06:26:56.487ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
138172025-04-30T06:26:56.491ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d00156b5-a2e1-472a-91d4-bf54a23613a5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138182025-04-30T06:26:56.491ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
138192025-04-30T06:26:56.494ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7893d06-ec24-4480-8831-955d4f087a33", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138202025-04-30T06:26:56.495ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
138212025-04-30T06:26:56.498ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3622bf60-0641-4225-8e18-25497ec2b948", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138222025-04-30T06:26:56.498ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
138232025-04-30T06:26:56.500ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6736460-23bb-41e5-9842-40b31a7b5a27", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138242025-04-30T06:26:56.500ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
138252025-04-30T06:26:56.504ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50d48ba2-e718-4e6a-98b8-2ba718c034bd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138262025-04-30T06:26:56.504ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
138272025-04-30T06:26:56.505ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ff39f19-e1d7-4377-a839-b767ed3b446b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138282025-04-30T06:26:56.505ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
138292025-04-30T06:26:56.506ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33216b30-977a-431e-a832-1bf4be83459c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138302025-04-30T06:26:56.507ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
138312025-04-30T06:26:56.507ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88fd230e-e405-48cc-a54a-1a83202e8029", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138322025-04-30T06:26:56.508ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
138332025-04-30T06:26:56.510ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "601cea9e-66e3-48a6-b124-3261d1e0d852", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138342025-04-30T06:26:56.510ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
138352025-04-30T06:26:56.511ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
138362025-04-30T06:26:56.512ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b66528af-1891-4313-a2d6-6efc15754d02", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138372025-04-30T06:26:56.512ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
138382025-04-30T06:26:56.514ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f45d5959-b70a-41cb-afe9-13a3647f9104", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138392025-04-30T06:26:56.514ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
138402025-04-30T06:26:56.515ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6436751f-f4d0-4ee2-aaed-1a2d511fcf30", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138412025-04-30T06:26:56.516ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
138422025-04-30T06:26:56.517ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fac2b70-7d10-4b53-87ec-f4e03770d682", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138432025-04-30T06:26:56.517ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
138442025-04-30T06:26:56.519ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "656d1bbb-369c-473a-b951-2902c3ed703f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138452025-04-30T06:26:56.519ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
138462025-04-30T06:26:56.521ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4097ee1a-9f61-46b1-9dec-6eaf700a1e57", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138472025-04-30T06:26:56.521ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
138482025-04-30T06:26:56.524ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f29d146a-9a1d-4aa8-bf5e-dc0e5ecee576", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138492025-04-30T06:26:56.524ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
138502025-04-30T06:26:56.525ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d650962d-6164-428f-becc-0422204d9fa3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138512025-04-30T06:26:56.525ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
138522025-04-30T06:26:56.526ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49741c86-d515-487e-aa35-64634d3824bb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138532025-04-30T06:26:56.526ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
138542025-04-30T06:26:56.527ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e0f4918-e778-4a24-bf94-f3916cab14f6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138552025-04-30T06:26:56.527ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
138562025-04-30T06:26:56.528ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99226fa3-fba9-4473-9078-f67d04184831", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138572025-04-30T06:26:56.528ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
138582025-04-30T06:26:56.528ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138592025-04-30T06:26:56.528ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138602025-04-30T06:26:56.528ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138612025-04-30T06:26:56.529ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138622025-04-30T06:26:56.531ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138632025-04-30T06:26:56.536ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
138642025-04-30T06:26:56.536ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
138652025-04-30T06:26:56.536ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138662025-04-30T06:26:56.536ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138672025-04-30T06:26:56.536ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138682025-04-30T06:26:56.537ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138692025-04-30T06:26:56.539ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138702025-04-30T06:26:56.545ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
138712025-04-30T06:26:56.557ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138722025-04-30T06:26:56.557ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138732025-04-30T06:26:56.557ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138742025-04-30T06:26:56.660ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
138752025-04-30T06:26:56.660ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
138762025-04-30T06:26:56.662ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138772025-04-30T06:26:56.663ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138782025-04-30T06:26:56.663ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138792025-04-30T06:26:56.663ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138802025-04-30T06:26:56.663ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138812025-04-30T06:26:56.663ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:60171 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138822025-04-30T06:26:56.664ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:60171 req_id = ac4415bf-af95-4392-b4c1-46775860a76e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
138832025-04-30T06:26:56.672ZINFOtest_disk_metrics (dropshot): request completed latency_us = 580 local_addr = [::1]:50989 method = GET remote_addr = [::1]:60171 req_id = ac4415bf-af95-4392-b4c1-46775860a76e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
138842025-04-30T06:26:56.672ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac4415bf-af95-4392-b4c1-46775860a76e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
138852025-04-30T06:26:56.674ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
138862025-04-30T06:26:56.674ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
138872025-04-30T06:26:56.674ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
138882025-04-30T06:26:56.680ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
138892025-04-30T06:26:56.680ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138902025-04-30T06:26:56.680ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138912025-04-30T06:26:56.680ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138922025-04-30T06:26:56.680ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138932025-04-30T06:26:56.681ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138942025-04-30T06:26:56.690ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138952025-04-30T06:26:56.690ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138962025-04-30T06:26:56.690ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138972025-04-30T06:26:56.692ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138982025-04-30T06:26:56.715ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138992025-04-30T06:26:56.720ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139002025-04-30T06:26:56.720ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139012025-04-30T06:26:56.720ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139022025-04-30T06:26:56.721ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139032025-04-30T06:26:56.724ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139042025-04-30T06:26:56.724ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
139052025-04-30T06:26:56.724ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139062025-04-30T06:26:56.725ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
139072025-04-30T06:26:56.727ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c6 31 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .1..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49269 req_id = d2db44d7-6f91-4eba-9e86-14c04db1b911
139082025-04-30T06:26:56.727ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50737,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49269 req_id = d2db44d7-6f91-4eba-9e86-14c04db1b911
139092025-04-30T06:26:56.727ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
139102025-04-30T06:26:56.727ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139112025-04-30T06:26:56.727ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49269 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d2db44d7-6f91-4eba-9e86-14c04db1b911
139122025-04-30T06:26:56.727ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3281, tv_nsec: 526280126 } })
139132025-04-30T06:26:56.727ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
139142025-04-30T06:26:56.728ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d0 5a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .Z..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52983 req_id = 4f461a1b-661d-4ea1-ad97-a8a613c8ca0f
139152025-04-30T06:26:56.728ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53338,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52983 req_id = 4f461a1b-661d-4ea1-ad97-a8a613c8ca0f
139162025-04-30T06:26:56.728ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139172025-04-30T06:26:56.729ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52983 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4f461a1b-661d-4ea1-ad97-a8a613c8ca0f
139182025-04-30T06:26:56.729ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
139192025-04-30T06:26:56.778ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
139202025-04-30T06:26:56.778ZDEBGtest_disk_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
139212025-04-30T06:26:56.778ZDEBGtest_disk_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
139222025-04-30T06:26:56.778ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
139232025-04-30T06:26:56.780ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
139242025-04-30T06:26:56.839ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
139252025-04-30T06:26:56.843ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
139262025-04-30T06:26:56.843ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 569.069023ms iteration = 51 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
139272025-04-30T06:26:56.843ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 52 reason = Signaled
139282025-04-30T06:26:56.843ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139292025-04-30T06:26:56.843ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139302025-04-30T06:26:56.844ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139312025-04-30T06:26:56.844ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139322025-04-30T06:26:56.845ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139332025-04-30T06:26:56.846ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
139342025-04-30T06:26:56.851ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139352025-04-30T06:26:56.855ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139362025-04-30T06:26:56.855ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139372025-04-30T06:26:56.855ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139382025-04-30T06:26:56.856ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139392025-04-30T06:26:56.870ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
139402025-04-30T06:26:56.871ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139412025-04-30T06:26:56.872ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f1c0cdbf-4dbe-4384-8af0-d54897d95822 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139422025-04-30T06:26:56.872ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f1c0cdbf-4dbe-4384-8af0-d54897d95822 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139432025-04-30T06:26:56.872ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f1c0cdbf-4dbe-4384-8af0-d54897d95822 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139442025-04-30T06:26:56.873ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f1c0cdbf-4dbe-4384-8af0-d54897d95822 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139452025-04-30T06:26:56.883ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
139462025-04-30T06:26:56.885ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
139472025-04-30T06:26:56.886ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 14783 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = f1c0cdbf-4dbe-4384-8af0-d54897d95822 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139482025-04-30T06:26:56.887ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1c0cdbf-4dbe-4384-8af0-d54897d95822", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
139492025-04-30T06:26:56.887ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139502025-04-30T06:26:56.888ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 5aa3e4bf-8722-468b-91fa-c72f8d33f9b7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139512025-04-30T06:26:56.888ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a2 d4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60523 req_id = 12160c7b-5493-4614-a5d1-c86df89a1498
139522025-04-30T06:26:56.888ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41684,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60523 req_id = 12160c7b-5493-4614-a5d1-c86df89a1498
139532025-04-30T06:26:56.888ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
139542025-04-30T06:26:56.890ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139552025-04-30T06:26:56.890ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60523 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 12160c7b-5493-4614-a5d1-c86df89a1498
139562025-04-30T06:26:56.891ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 5aa3e4bf-8722-468b-91fa-c72f8d33f9b7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139572025-04-30T06:26:56.891ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 5aa3e4bf-8722-468b-91fa-c72f8d33f9b7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139582025-04-30T06:26:56.893ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 5aa3e4bf-8722-468b-91fa-c72f8d33f9b7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139592025-04-30T06:26:56.894ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3281, tv_nsec: 693188260 } })
139602025-04-30T06:26:56.894ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
139612025-04-30T06:26:56.895ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
139622025-04-30T06:26:56.895ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6b 7e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e k~..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63862 req_id = ed8fd9d0-c4d7-4ef6-9114-c06e6c4bef44
139632025-04-30T06:26:56.895ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27518,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63862 req_id = ed8fd9d0-c4d7-4ef6-9114-c06e6c4bef44
139642025-04-30T06:26:56.895ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139652025-04-30T06:26:56.896ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63862 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ed8fd9d0-c4d7-4ef6-9114-c06e6c4bef44
139662025-04-30T06:26:56.896ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 8038 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 5aa3e4bf-8722-468b-91fa-c72f8d33f9b7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139672025-04-30T06:26:56.941ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
139682025-04-30T06:26:56.941ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c0 95 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64439 req_id = 9fca1492-5a37-43c4-bb70-64b0b5d8dae2
139692025-04-30T06:26:56.941ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49301,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64439 req_id = 9fca1492-5a37-43c4-bb70-64b0b5d8dae2
139702025-04-30T06:26:56.942ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
139712025-04-30T06:26:56.942ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139722025-04-30T06:26:56.942ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64439 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9fca1492-5a37-43c4-bb70-64b0b5d8dae2
139732025-04-30T06:26:56.942ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aa3e4bf-8722-468b-91fa-c72f8d33f9b7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
139742025-04-30T06:26:56.942ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
139752025-04-30T06:26:56.942ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139762025-04-30T06:26:56.942ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
139772025-04-30T06:26:56.943ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139782025-04-30T06:26:56.943ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139792025-04-30T06:26:56.943ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139802025-04-30T06:26:56.943ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
139812025-04-30T06:26:56.944ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139822025-04-30T06:26:56.944ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3281, tv_nsec: 743704240 } })
139832025-04-30T06:26:56.944ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
139842025-04-30T06:26:56.947ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a5 f1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53982 req_id = cd5c2d16-56e0-463b-9be9-06136d01d2c5
139852025-04-30T06:26:56.947ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42481,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53982 req_id = cd5c2d16-56e0-463b-9be9-06136d01d2c5
139862025-04-30T06:26:56.947ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
139872025-04-30T06:26:56.947ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42481, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53982 req_id = cd5c2d16-56e0-463b-9be9-06136d01d2c5
139882025-04-30T06:26:56.949ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
139892025-04-30T06:26:56.949ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
139902025-04-30T06:26:56.949ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
139912025-04-30T06:26:56.950ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 41 19 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e A...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61454 req_id = eedae4a5-0153-4a9c-84e3-41051e324e9f
139922025-04-30T06:26:56.950ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16665,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61454 req_id = eedae4a5-0153-4a9c-84e3-41051e324e9f
139932025-04-30T06:26:56.950ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139942025-04-30T06:26:56.950ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61454 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = eedae4a5-0153-4a9c-84e3-41051e324e9f
139952025-04-30T06:26:56.951ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139962025-04-30T06:26:56.958ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139972025-04-30T06:26:56.958ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139982025-04-30T06:26:56.958ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
139992025-04-30T06:26:56.959ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140002025-04-30T06:26:56.960ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
140012025-04-30T06:26:57.016ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
140022025-04-30T06:26:57.016ZDEBGtest_disk_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
140032025-04-30T06:26:57.016ZDEBGtest_disk_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
140042025-04-30T06:26:57.016ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
140052025-04-30T06:26:57.019ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
140062025-04-30T06:26:57.020ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140072025-04-30T06:26:57.027ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140082025-04-30T06:26:57.029ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140092025-04-30T06:26:57.029ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140102025-04-30T06:26:57.032ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140112025-04-30T06:26:57.035ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140122025-04-30T06:26:57.035ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140132025-04-30T06:26:57.035ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140142025-04-30T06:26:57.035ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140152025-04-30T06:26:57.040ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "151e30fd-d440-4380-b79d-b88e2e6d9f3b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140162025-04-30T06:26:57.041ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
140172025-04-30T06:26:57.061ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a74b40-768b-4005-aba9-9da8cd2cc0a9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140182025-04-30T06:26:57.062ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
140192025-04-30T06:26:57.081ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71fc7d2f-3341-4d8b-b242-495c626b60fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140202025-04-30T06:26:57.082ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
140212025-04-30T06:26:57.087ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6dc1828-2f1e-407a-936c-b3c553916e6d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140222025-04-30T06:26:57.087ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
140232025-04-30T06:26:57.091ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33d9a826-8f27-492c-8fe6-63f969733b43", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140242025-04-30T06:26:57.091ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
140252025-04-30T06:26:57.095ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
140262025-04-30T06:26:57.097ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d80aa6a-d2e5-4ba9-9263-c8da0dda0e14", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140272025-04-30T06:26:57.097ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
140282025-04-30T06:26:57.101ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f707d57b-5f22-4114-aedd-5ae40146864f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140292025-04-30T06:26:57.101ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
140302025-04-30T06:26:57.103ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08beb93a-fa94-4e97-91f7-90e94d22a88a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140312025-04-30T06:26:57.103ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
140322025-04-30T06:26:57.104ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e6c2e05-4cac-46c7-91fd-eabaa721824b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140332025-04-30T06:26:57.104ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
140342025-04-30T06:26:57.106ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05563a95-5d1a-4ae4-9c9a-48085bedc813", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140352025-04-30T06:26:57.106ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
140362025-04-30T06:26:57.107ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c05573f8-b5ff-4248-a8c7-fcf973eb7b1c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140372025-04-30T06:26:57.107ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
140382025-04-30T06:26:57.108ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50cf8ad7-a741-48e8-b4c0-f82868bbedfc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140392025-04-30T06:26:57.108ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
140402025-04-30T06:26:57.110ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b17bdfd-b720-4495-861b-4c0b3170f863", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140412025-04-30T06:26:57.110ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
140422025-04-30T06:26:57.112ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "368d60e3-cb08-45aa-aba6-990d81e9a922", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140432025-04-30T06:26:57.112ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
140442025-04-30T06:26:57.114ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d72ca93-1b95-4eea-9055-4b6d67860c80", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140452025-04-30T06:26:57.114ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
140462025-04-30T06:26:57.116ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50ccb999-d528-4c28-9e27-b53774e95a73", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140472025-04-30T06:26:57.116ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
140482025-04-30T06:26:57.118ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2b89b59-3e37-48a8-9ee1-9914cb910cd2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140492025-04-30T06:26:57.118ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
140502025-04-30T06:26:57.121ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45925efa-9a36-4048-9059-4e570eca71e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140512025-04-30T06:26:57.121ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
140522025-04-30T06:26:57.123ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa5c4520-c6c8-425c-8466-3c09f08168f2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140532025-04-30T06:26:57.124ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
140542025-04-30T06:26:57.125ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f2ff670-af5b-4c21-835a-fa69bad81e17", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140552025-04-30T06:26:57.125ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
140562025-04-30T06:26:57.126ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "305f5110-24ce-4caf-ad15-a90552822b10", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140572025-04-30T06:26:57.126ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
140582025-04-30T06:26:57.127ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7c2c48f-c743-4f99-81a2-a70a90b71d8a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140592025-04-30T06:26:57.127ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
140602025-04-30T06:26:57.128ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13026b7e-6db6-4566-9245-d018d9d68269", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140612025-04-30T06:26:57.129ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
140622025-04-30T06:26:57.130ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0ecbeb2-097a-4d78-856a-be0f99e8b5dc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140632025-04-30T06:26:57.130ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
140642025-04-30T06:26:57.132ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6ec97b9-b96e-435a-8b3d-e3b52cef771e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140652025-04-30T06:26:57.132ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
140662025-04-30T06:26:57.135ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa09b7bb-9836-4d17-9dc5-d52b0d137926", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140672025-04-30T06:26:57.135ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
140682025-04-30T06:26:57.137ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e4d056c-f4ec-45ac-a5eb-dc04c91a6a87", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140692025-04-30T06:26:57.137ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
140702025-04-30T06:26:57.140ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec0fffc-bccb-4001-b998-70639e5dd9bc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140712025-04-30T06:26:57.140ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
140722025-04-30T06:26:57.143ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df0b9dd9-bf99-4b3a-8160-5c1fb098be75", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140732025-04-30T06:26:57.143ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
140742025-04-30T06:26:57.146ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1f6edeb-ae43-4c29-be69-90771e32dfcb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140752025-04-30T06:26:57.146ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
140762025-04-30T06:26:57.146ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140772025-04-30T06:26:57.146ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140782025-04-30T06:26:57.146ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140792025-04-30T06:26:57.146ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140802025-04-30T06:26:57.147ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140812025-04-30T06:26:57.148ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69f186de-7606-4ec6-a5a0-9a213d393dac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140822025-04-30T06:26:57.148ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
140832025-04-30T06:26:57.150ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcfd9fc3-9063-47cd-b3d4-7922e4fc4835", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140842025-04-30T06:26:57.150ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
140852025-04-30T06:26:57.151ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140862025-04-30T06:26:57.157ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140872025-04-30T06:26:57.157ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140882025-04-30T06:26:57.157ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140892025-04-30T06:26:57.158ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140902025-04-30T06:26:57.160ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc529fcd-676a-47f1-bc9d-6bacee1ae7e7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140912025-04-30T06:26:57.161ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
140922025-04-30T06:26:57.164ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9e7edf-3e8b-44e5-a0d8-f6ca4bde524c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140932025-04-30T06:26:57.164ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
140942025-04-30T06:26:57.164ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140952025-04-30T06:26:57.164ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140962025-04-30T06:26:57.165ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140972025-04-30T06:26:57.169ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a824e8a-dc53-4d9c-8ea6-08a1163dfd12", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
140982025-04-30T06:26:57.170ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
140992025-04-30T06:26:57.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141002025-04-30T06:26:57.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141012025-04-30T06:26:57.176ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
141022025-04-30T06:26:57.177ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
141032025-04-30T06:26:57.182ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "703ca548-b438-468d-8211-f9061f603e35", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141042025-04-30T06:26:57.182ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
141052025-04-30T06:26:57.183ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
141062025-04-30T06:26:57.183ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
141072025-04-30T06:26:57.189ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
141082025-04-30T06:26:57.189ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141092025-04-30T06:26:57.190ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86997bd8-3660-4eb6-bb6c-883253e3ab1e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141102025-04-30T06:26:57.190ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
141112025-04-30T06:26:57.197ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffae1117-6cb8-4d84-8849-315ecaea94fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141122025-04-30T06:26:57.197ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
141132025-04-30T06:26:57.199ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "947e520a-67ce-40d2-9dd6-c71f6f971287", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141142025-04-30T06:26:57.199ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
141152025-04-30T06:26:57.203ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
141162025-04-30T06:26:57.203ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bec64ea1-5cd9-42e9-afba-2bb148f5a112", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141172025-04-30T06:26:57.203ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
141182025-04-30T06:26:57.208ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eff5db4e-973e-4f98-b644-9de15e4f1237", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141192025-04-30T06:26:57.208ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
141202025-04-30T06:26:57.210ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51576002-7a42-4381-a070-7576d6b420df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141212025-04-30T06:26:57.210ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
141222025-04-30T06:26:57.211ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "468eb150-35f9-404a-921b-f32cb2dd6a85", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141232025-04-30T06:26:57.211ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
141242025-04-30T06:26:57.212ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa33d546-70a7-4262-99d6-fceb1dd955fd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141252025-04-30T06:26:57.212ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
141262025-04-30T06:26:57.213ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "492fa8da-7f9a-455f-adb1-5a473af2974b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141272025-04-30T06:26:57.213ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
141282025-04-30T06:26:57.213ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141292025-04-30T06:26:57.213ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141302025-04-30T06:26:57.214ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141312025-04-30T06:26:57.214ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141322025-04-30T06:26:57.217ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141332025-04-30T06:26:57.221ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
141342025-04-30T06:26:57.221ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
141352025-04-30T06:26:57.221ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141362025-04-30T06:26:57.221ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141372025-04-30T06:26:57.221ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141382025-04-30T06:26:57.222ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141392025-04-30T06:26:57.226ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141402025-04-30T06:26:57.229ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
141412025-04-30T06:26:57.229ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141422025-04-30T06:26:57.230ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141432025-04-30T06:26:57.230ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141442025-04-30T06:26:57.300ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
141452025-04-30T06:26:57.301ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
141462025-04-30T06:26:57.302ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
141472025-04-30T06:26:57.302ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141482025-04-30T06:26:57.302ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141492025-04-30T06:26:57.302ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
141502025-04-30T06:26:57.304ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141512025-04-30T06:26:57.304ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:59738 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
141522025-04-30T06:26:57.306ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
141532025-04-30T06:26:57.306ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:59738 req_id = 5251ba97-703f-4f58-8a80-4286f4dd1885 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141542025-04-30T06:26:57.307ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:36082 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
141552025-04-30T06:26:57.308ZINFOtest_disk_metrics (dropshot): request completed latency_us = 1492 local_addr = [::1]:50989 method = GET remote_addr = [::1]:59738 req_id = 5251ba97-703f-4f58-8a80-4286f4dd1885 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141562025-04-30T06:26:57.309ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:36082 req_id = 41db34f5-0748-4b51-a3f4-8445e20fbd1b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
141572025-04-30T06:26:57.310ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5251ba97-703f-4f58-8a80-4286f4dd1885", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:56 GMT"} })
141582025-04-30T06:26:57.312ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141592025-04-30T06:26:57.312ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141602025-04-30T06:26:57.312ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
141612025-04-30T06:26:57.316ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141622025-04-30T06:26:57.316ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141632025-04-30T06:26:57.316ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141642025-04-30T06:26:57.316ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141652025-04-30T06:26:57.316ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141662025-04-30T06:26:57.317ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 7925 local_addr = [::1]:35636 method = GET remote_addr = [::1]:36082 req_id = 41db34f5-0748-4b51-a3f4-8445e20fbd1b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
141672025-04-30T06:26:57.318ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141682025-04-30T06:26:57.327ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
141692025-04-30T06:26:57.327ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141702025-04-30T06:26:57.327ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
141712025-04-30T06:26:57.328ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141722025-04-30T06:26:57.330ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141732025-04-30T06:26:57.336ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141742025-04-30T06:26:57.337ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141752025-04-30T06:26:57.337ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141762025-04-30T06:26:57.338ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141772025-04-30T06:26:57.338ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141782025-04-30T06:26:57.339ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
141792025-04-30T06:26:57.339ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141802025-04-30T06:26:57.379ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
141812025-04-30T06:26:57.379ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
141822025-04-30T06:26:57.380ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 536.124814ms iteration = 52 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
141832025-04-30T06:26:57.380ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 53 reason = Signaled
141842025-04-30T06:26:57.380ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141852025-04-30T06:26:57.380ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141862025-04-30T06:26:57.380ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141872025-04-30T06:26:57.380ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141882025-04-30T06:26:57.380ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141892025-04-30T06:26:57.383ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141902025-04-30T06:26:57.396ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141912025-04-30T06:26:57.396ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141922025-04-30T06:26:57.396ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141932025-04-30T06:26:57.397ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141942025-04-30T06:26:57.397ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
141952025-04-30T06:26:57.400ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
141962025-04-30T06:26:57.400ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
141972025-04-30T06:26:57.400ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2a 1f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 *............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57572 req_id = d08d9d5b-fc06-4775-a93f-ec3abc14f40a
141982025-04-30T06:26:57.401ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10783,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57572 req_id = d08d9d5b-fc06-4775-a93f-ec3abc14f40a
141992025-04-30T06:26:57.401ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
142002025-04-30T06:26:57.401ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142012025-04-30T06:26:57.401ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57572 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d08d9d5b-fc06-4775-a93f-ec3abc14f40a
142022025-04-30T06:26:57.401ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3282, tv_nsec: 200760454 } })
142032025-04-30T06:26:57.401ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
142042025-04-30T06:26:57.402ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 84 b9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62475 req_id = 43b6b6ec-a83f-4b08-98ce-450327e87d09
142052025-04-30T06:26:57.402ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33977,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62475 req_id = 43b6b6ec-a83f-4b08-98ce-450327e87d09
142062025-04-30T06:26:57.402ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142072025-04-30T06:26:57.402ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62475 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 43b6b6ec-a83f-4b08-98ce-450327e87d09
142082025-04-30T06:26:57.460ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
142092025-04-30T06:26:57.460ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142102025-04-30T06:26:57.460ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142112025-04-30T06:26:57.460ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142122025-04-30T06:26:57.460ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
142132025-04-30T06:26:57.461ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142142025-04-30T06:26:57.461ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
142152025-04-30T06:26:57.462ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ff 12 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63520 req_id = 0464a664-7592-4dae-b9c2-5a530e8499ee
142162025-04-30T06:26:57.462ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65298,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63520 req_id = 0464a664-7592-4dae-b9c2-5a530e8499ee
142172025-04-30T06:26:57.462ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
142182025-04-30T06:26:57.462ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65298, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63520 req_id = 0464a664-7592-4dae-b9c2-5a530e8499ee
142192025-04-30T06:26:57.462ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
142202025-04-30T06:26:57.462ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
142212025-04-30T06:26:57.462ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
142222025-04-30T06:26:57.462ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 4c 00 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 L............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57336 req_id = 5991d2dc-8fe8-4ab2-9a3e-ed3d4cfb92d1
142232025-04-30T06:26:57.462ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19456,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57336 req_id = 5991d2dc-8fe8-4ab2-9a3e-ed3d4cfb92d1
142242025-04-30T06:26:57.462ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
142252025-04-30T06:26:57.462ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142262025-04-30T06:26:57.462ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57336 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5991d2dc-8fe8-4ab2-9a3e-ed3d4cfb92d1
142272025-04-30T06:26:57.464ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3282, tv_nsec: 263264409 } })
142282025-04-30T06:26:57.464ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
142292025-04-30T06:26:57.465ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142302025-04-30T06:26:57.475ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142312025-04-30T06:26:57.475ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142322025-04-30T06:26:57.475ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
142332025-04-30T06:26:57.476ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142342025-04-30T06:26:57.480ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ac 92 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50678 req_id = 7fc919b2-7036-48bd-86d3-99c683671566
142352025-04-30T06:26:57.480ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44178,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50678 req_id = 7fc919b2-7036-48bd-86d3-99c683671566
142362025-04-30T06:26:57.480ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142372025-04-30T06:26:57.480ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50678 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7fc919b2-7036-48bd-86d3-99c683671566
142382025-04-30T06:26:57.482ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
142392025-04-30T06:26:57.549ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
142402025-04-30T06:26:57.549ZDEBGtest_disk_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
142412025-04-30T06:26:57.549ZDEBGtest_disk_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
142422025-04-30T06:26:57.549ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
142432025-04-30T06:26:57.551ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
142442025-04-30T06:26:57.552ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142452025-04-30T06:26:57.552ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
142462025-04-30T06:26:57.553ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142472025-04-30T06:26:57.554ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "21624fe3-c37c-46dc-8144-e0fec2363d35", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142482025-04-30T06:26:57.554ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
142492025-04-30T06:26:57.557ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88e4cbb1-a0d5-4b59-ac08-fb10c8ded792", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142502025-04-30T06:26:57.557ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
142512025-04-30T06:26:57.560ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41827684-0ac8-44a8-85e2-6f540957f63d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142522025-04-30T06:26:57.560ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
142532025-04-30T06:26:57.562ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fc73551-969f-434f-9ca1-404a9cef72c2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142542025-04-30T06:26:57.562ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
142552025-04-30T06:26:57.564ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd52d07c-e66e-4f58-ab52-bf0f192ac742", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142562025-04-30T06:26:57.564ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
142572025-04-30T06:26:57.567ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa89d1ae-e41b-428e-a1bd-6d3bc0f5c863", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142582025-04-30T06:26:57.567ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
142592025-04-30T06:26:57.569ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd366430-b66b-4d9e-bb3c-e0f5d7c3b40d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142602025-04-30T06:26:57.569ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
142612025-04-30T06:26:57.572ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a432ada6-1816-4018-a6da-b3bf2d99055c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142622025-04-30T06:26:57.572ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
142632025-04-30T06:26:57.573ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6786dee-3673-4453-979e-c29d37c1721d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142642025-04-30T06:26:57.573ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
142652025-04-30T06:26:57.574ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72691677-9b26-4e6b-abb7-e0c5e91c0080", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142662025-04-30T06:26:57.574ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
142672025-04-30T06:26:57.576ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c1c0ea2-f3b8-4b8d-af64-f7c66774c4a1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142682025-04-30T06:26:57.576ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
142692025-04-30T06:26:57.578ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a138bb0-352b-4f37-a034-d52dfec6c766", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142702025-04-30T06:26:57.578ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
142712025-04-30T06:26:57.579ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aa9387b-e4f9-4dd0-9594-fcaed704d282", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142722025-04-30T06:26:57.580ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
142732025-04-30T06:26:57.582ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "059e19d9-aad6-44a2-89e7-9d883d09a485", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142742025-04-30T06:26:57.582ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
142752025-04-30T06:26:57.595ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1d8333c-e2ce-45fd-8f85-23f5030f19e8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142762025-04-30T06:26:57.596ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
142772025-04-30T06:26:57.597ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
142782025-04-30T06:26:57.600ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41d909fa-9d51-47c2-8a38-6a30a42676b4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142792025-04-30T06:26:57.600ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
142802025-04-30T06:26:57.610ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "349be0ac-f17e-46bb-b40d-4ae48f16747a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142812025-04-30T06:26:57.610ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
142822025-04-30T06:26:57.615ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d68497f-66e9-4bd1-8ccf-84ac132cd264", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142832025-04-30T06:26:57.615ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
142842025-04-30T06:26:57.623ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "539fe193-4f65-42d9-b936-12257017768c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142852025-04-30T06:26:57.624ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
142862025-04-30T06:26:57.625ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e008432-cf0a-46af-899d-cda5638912d0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142872025-04-30T06:26:57.626ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
142882025-04-30T06:26:57.629ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c8abc0e-72fb-4d53-aac6-e9a0d60eade5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142892025-04-30T06:26:57.629ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
142902025-04-30T06:26:57.631ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ac9bb30-5d3d-467c-9a9e-05e2591e6626", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142912025-04-30T06:26:57.631ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
142922025-04-30T06:26:57.633ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83c45235-fff2-4814-af11-0cf27623a442", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142932025-04-30T06:26:57.634ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
142942025-04-30T06:26:57.636ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c824b729-abfc-46fa-92fe-3de1edc2e78d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142952025-04-30T06:26:57.636ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
142962025-04-30T06:26:57.638ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ada9e86-424c-4721-825f-fc35d9667da3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142972025-04-30T06:26:57.639ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
142982025-04-30T06:26:57.641ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad4488c-d809-4dc1-80df-4dda0d7f0427", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
142992025-04-30T06:26:57.641ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
143002025-04-30T06:26:57.643ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cacb2b09-93e4-4033-ae50-d32e801795d9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143012025-04-30T06:26:57.644ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
143022025-04-30T06:26:57.646ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cb44411-d4bf-4f69-99a2-57cd20fb61d4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143032025-04-30T06:26:57.646ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
143042025-04-30T06:26:57.648ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143052025-04-30T06:26:57.648ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143062025-04-30T06:26:57.648ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143072025-04-30T06:26:57.648ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143082025-04-30T06:26:57.649ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143092025-04-30T06:26:57.650ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbbd041e-ef82-45d3-b200-671bb11063cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143102025-04-30T06:26:57.651ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
143112025-04-30T06:26:57.652ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143122025-04-30T06:26:57.656ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143132025-04-30T06:26:57.656ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143142025-04-30T06:26:57.656ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143152025-04-30T06:26:57.657ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143162025-04-30T06:26:57.661ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143172025-04-30T06:26:57.661ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143182025-04-30T06:26:57.661ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143192025-04-30T06:26:57.663ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "467fda2e-50a9-42f5-af5d-25d379cb2b65", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143202025-04-30T06:26:57.663ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
143212025-04-30T06:26:57.665ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7a49ac7-b052-4237-896f-4a3515e3f478", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143222025-04-30T06:26:57.665ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
143232025-04-30T06:26:57.666ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f04bb19-e159-4309-9970-db8ee19f8746", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143242025-04-30T06:26:57.666ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
143252025-04-30T06:26:57.669ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb8850e5-cd64-4642-9199-5fa99f5dc2b1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143262025-04-30T06:26:57.669ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
143272025-04-30T06:26:57.681ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b4e8794-5e57-4d3b-9ef9-dc21646a2761", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143282025-04-30T06:26:57.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
143292025-04-30T06:26:57.684ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f09b6ea0-f7f8-4015-99c7-c1a0db93078e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143302025-04-30T06:26:57.684ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
143312025-04-30T06:26:57.688ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dd86c67-c80c-4df8-9735-4ebf20394245", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143322025-04-30T06:26:57.688ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
143332025-04-30T06:26:57.689ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67a16f8c-aaf3-4281-88c3-30686c484a71", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143342025-04-30T06:26:57.689ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
143352025-04-30T06:26:57.692ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f02d12fd-6172-4882-8103-e8ba42dff704", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143362025-04-30T06:26:57.692ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
143372025-04-30T06:26:57.696ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c37cabd-799a-412d-b0f9-2d2d739b78fe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143382025-04-30T06:26:57.696ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
143392025-04-30T06:26:57.698ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f73e990c-d2c0-4366-bbb4-cae9da925339", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143402025-04-30T06:26:57.698ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
143412025-04-30T06:26:57.701ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "130a898d-4444-43de-b088-0754d0d9cdae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143422025-04-30T06:26:57.702ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
143432025-04-30T06:26:57.704ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:40905 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143442025-04-30T06:26:57.705ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:40905 req_id = fbd31c87-590c-4679-90fb-f394e8eb9262 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143452025-04-30T06:26:57.705ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 509 local_addr = [::1]:35636 method = GET remote_addr = [::1]:40905 req_id = fbd31c87-590c-4679-90fb-f394e8eb9262 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143462025-04-30T06:26:57.705ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
143472025-04-30T06:26:57.706ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b7f287f-88fd-44d3-95c8-52e918947bd6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143482025-04-30T06:26:57.706ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
143492025-04-30T06:26:57.707ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:52079 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143502025-04-30T06:26:57.720ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:52573 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143512025-04-30T06:26:57.720ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:42077 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143522025-04-30T06:26:57.720ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:52079 req_id = 30f090a3-c255-4c0f-8aaf-2d22d6b5a9c1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143532025-04-30T06:26:57.728ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 7998 local_addr = [::1]:35636 method = GET remote_addr = [::1]:52079 req_id = 30f090a3-c255-4c0f-8aaf-2d22d6b5a9c1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143542025-04-30T06:26:57.729ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:63462 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143552025-04-30T06:26:57.729ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:33248 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143562025-04-30T06:26:57.729ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:52573 req_id = fb749b83-d677-4b31-a8d4-826d0397039a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143572025-04-30T06:26:57.730ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:42077 req_id = 370e8002-3858-4771-abe3-9dad85c8737b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143582025-04-30T06:26:57.733ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb9b520c-dc6d-42db-ae69-f9cf2a1e8856", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143592025-04-30T06:26:57.733ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
143602025-04-30T06:26:57.735ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 6038 local_addr = [::1]:35636 method = GET remote_addr = [::1]:52573 req_id = fb749b83-d677-4b31-a8d4-826d0397039a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143612025-04-30T06:26:57.735ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 5833 local_addr = [::1]:35636 method = GET remote_addr = [::1]:42077 req_id = 370e8002-3858-4771-abe3-9dad85c8737b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143622025-04-30T06:26:57.736ZINFOtest_disk_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:35636 remote_addr = [::1]:33577 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143632025-04-30T06:26:57.737ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:63462 req_id = 705e24f6-a539-4519-b2b8-b91549a34d31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143642025-04-30T06:26:57.738ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:33248 req_id = eff318d1-cc92-4866-bd85-9ea5d019dda3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143652025-04-30T06:26:57.738ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 970 local_addr = [::1]:35636 method = GET remote_addr = [::1]:63462 req_id = 705e24f6-a539-4519-b2b8-b91549a34d31 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143662025-04-30T06:26:57.738ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 903 local_addr = [::1]:35636 method = GET remote_addr = [::1]:33248 req_id = eff318d1-cc92-4866-bd85-9ea5d019dda3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143672025-04-30T06:26:57.739ZTRACtest_disk_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:35636 method = GET remote_addr = [::1]:33577 req_id = e52f1e98-35b0-4033-a490-3f3923662619 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143682025-04-30T06:26:57.739ZINFOtest_disk_metrics (dropshot): request completed kind = pantry latency_us = 521 local_addr = [::1]:35636 method = GET remote_addr = [::1]:33577 req_id = e52f1e98-35b0-4033-a490-3f3923662619 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143692025-04-30T06:26:57.740ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ed450e0-2d19-4010-8c89-3889f7e80747", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143702025-04-30T06:26:57.740ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
143712025-04-30T06:26:57.741ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "086d3c69-bec4-4fee-8815-458dffb6efca", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
143722025-04-30T06:26:57.741ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143732025-04-30T06:26:57.742ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143742025-04-30T06:26:57.743ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143752025-04-30T06:26:57.743ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143762025-04-30T06:26:57.743ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143772025-04-30T06:26:57.747ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143782025-04-30T06:26:57.760ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143792025-04-30T06:26:57.760ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143802025-04-30T06:26:57.760ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143812025-04-30T06:26:57.760ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143822025-04-30T06:26:57.760ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143832025-04-30T06:26:57.762ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143842025-04-30T06:26:57.763ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143852025-04-30T06:26:57.763ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143862025-04-30T06:26:57.763ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143872025-04-30T06:26:57.763ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143882025-04-30T06:26:57.764ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143892025-04-30T06:26:57.766ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143902025-04-30T06:26:57.771ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143912025-04-30T06:26:57.771ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143922025-04-30T06:26:57.771ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143932025-04-30T06:26:57.772ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143942025-04-30T06:26:57.773ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143952025-04-30T06:26:57.781ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143962025-04-30T06:26:57.781ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143972025-04-30T06:26:57.781ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143982025-04-30T06:26:57.781ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143992025-04-30T06:26:57.861ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
144002025-04-30T06:26:57.861ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
144012025-04-30T06:26:57.862ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:59620 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
144022025-04-30T06:26:57.862ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:59620 req_id = 4dbf1fb6-188f-4349-aa63-61611a0801d8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
144032025-04-30T06:26:57.863ZINFOtest_disk_metrics (dropshot): request completed latency_us = 490 local_addr = [::1]:50989 method = GET remote_addr = [::1]:59620 req_id = 4dbf1fb6-188f-4349-aa63-61611a0801d8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
144042025-04-30T06:26:57.863ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dbf1fb6-188f-4349-aa63-61611a0801d8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144052025-04-30T06:26:57.864ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
144062025-04-30T06:26:57.864ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
144072025-04-30T06:26:57.864ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
144082025-04-30T06:26:57.868ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
144092025-04-30T06:26:57.868ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144102025-04-30T06:26:57.868ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
144112025-04-30T06:26:57.869ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
144122025-04-30T06:26:57.869ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
144132025-04-30T06:26:57.870ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144142025-04-30T06:26:57.870ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
144152025-04-30T06:26:57.870ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144162025-04-30T06:26:57.873ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144172025-04-30T06:26:57.878ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
144182025-04-30T06:26:57.879ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
144192025-04-30T06:26:57.879ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
144202025-04-30T06:26:57.881ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
144212025-04-30T06:26:57.942ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
144222025-04-30T06:26:57.945ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
144232025-04-30T06:26:57.945ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 565.600942ms iteration = 53 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
144242025-04-30T06:26:57.945ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 54 reason = Signaled
144252025-04-30T06:26:57.945ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
144262025-04-30T06:26:57.946ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144272025-04-30T06:26:57.946ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
144282025-04-30T06:26:57.946ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
144292025-04-30T06:26:57.947ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
144302025-04-30T06:26:57.948ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
144312025-04-30T06:26:57.951ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144322025-04-30T06:26:57.958ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
144332025-04-30T06:26:57.958ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
144342025-04-30T06:26:57.958ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
144352025-04-30T06:26:57.961ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
144362025-04-30T06:26:57.981ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
144372025-04-30T06:26:57.981ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
144382025-04-30T06:26:57.985ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c0 74 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .t..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58408 req_id = 02f1c73d-53ab-4334-8e74-4ea150d9f0bb
144392025-04-30T06:26:57.986ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49268,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58408 req_id = 02f1c73d-53ab-4334-8e74-4ea150d9f0bb
144402025-04-30T06:26:57.986ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
144412025-04-30T06:26:57.986ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
144422025-04-30T06:26:57.986ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58408 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 02f1c73d-53ab-4334-8e74-4ea150d9f0bb
144432025-04-30T06:26:57.987ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3282, tv_nsec: 786491034 } })
144442025-04-30T06:26:57.987ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
144452025-04-30T06:26:57.987ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144462025-04-30T06:26:57.988ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144472025-04-30T06:26:57.992ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 08 2e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53220 req_id = 40aa21cb-9428-4d33-a5a8-32e8836253b1
144482025-04-30T06:26:57.992ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2094,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53220 req_id = 40aa21cb-9428-4d33-a5a8-32e8836253b1
144492025-04-30T06:26:57.992ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
144502025-04-30T06:26:57.992ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53220 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 40aa21cb-9428-4d33-a5a8-32e8836253b1
144512025-04-30T06:26:57.993ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144522025-04-30T06:26:57.994ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144532025-04-30T06:26:57.994ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144542025-04-30T06:26:57.994ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144552025-04-30T06:26:58.070ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
144562025-04-30T06:26:58.074ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
144572025-04-30T06:26:58.074ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144582025-04-30T06:26:58.074ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144592025-04-30T06:26:58.074ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
144602025-04-30T06:26:58.075ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144612025-04-30T06:26:58.085ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3e 9f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c >............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60345 req_id = 53d5c72b-e9f1-4b24-a1aa-84f834019337
144622025-04-30T06:26:58.085ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16031,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60345 req_id = 53d5c72b-e9f1-4b24-a1aa-84f834019337
144632025-04-30T06:26:58.085ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
144642025-04-30T06:26:58.086ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16031, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60345 req_id = 53d5c72b-e9f1-4b24-a1aa-84f834019337
144652025-04-30T06:26:58.086ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
144662025-04-30T06:26:58.087ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
144672025-04-30T06:26:58.087ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
144682025-04-30T06:26:58.087ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144692025-04-30T06:26:58.094ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
144702025-04-30T06:26:58.094ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144712025-04-30T06:26:58.094ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
144722025-04-30T06:26:58.095ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144732025-04-30T06:26:58.111ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144742025-04-30T06:26:58.111ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
144752025-04-30T06:26:58.111ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144762025-04-30T06:26:58.116ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c2a36bb-84af-467e-a126-23b86e4f492c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144772025-04-30T06:26:58.116ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
144782025-04-30T06:26:58.121ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4da07d2e-f51b-4e84-9612-3904844f32c6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144792025-04-30T06:26:58.121ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
144802025-04-30T06:26:58.125ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5c1458b-9fe5-41b1-ad2b-831776bf95c7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144812025-04-30T06:26:58.125ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
144822025-04-30T06:26:58.127ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98c130e3-1b2c-4271-918f-f352f74113c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144832025-04-30T06:26:58.129ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
144842025-04-30T06:26:58.132ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4072018f-15a1-450f-8475-004494f05060", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144852025-04-30T06:26:58.132ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
144862025-04-30T06:26:58.134ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7270a7d8-86a9-4e9d-a49f-3fb623359527", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144872025-04-30T06:26:58.134ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
144882025-04-30T06:26:58.136ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17b8c8b5-fd63-40f7-a45d-443165af8edc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144892025-04-30T06:26:58.136ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
144902025-04-30T06:26:58.139ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8d864fc-5686-421d-bf78-30239e8def25", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144912025-04-30T06:26:58.139ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
144922025-04-30T06:26:58.140ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab1a9e89-07ca-42fc-be26-166823a6cea3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144932025-04-30T06:26:58.140ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
144942025-04-30T06:26:58.141ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "189d168d-5de5-4ac1-98f0-613f758bd0ba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144952025-04-30T06:26:58.142ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
144962025-04-30T06:26:58.145ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
144972025-04-30T06:26:58.148ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a70152d-6c67-4f5b-bbe9-b45a5c3a9919", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
144982025-04-30T06:26:58.149ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
144992025-04-30T06:26:58.150ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5c339c9-911b-499c-8cde-826a31526c8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145002025-04-30T06:26:58.150ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
145012025-04-30T06:26:58.151ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f470cca5-d570-4204-baa3-b94ca239e7a9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145022025-04-30T06:26:58.151ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
145032025-04-30T06:26:58.153ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1049c348-0a0c-404d-86dd-49b364ac201d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145042025-04-30T06:26:58.153ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
145052025-04-30T06:26:58.155ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b26d01e-8d87-40fa-a11a-29483e09502b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145062025-04-30T06:26:58.155ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
145072025-04-30T06:26:58.157ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1db8304d-e007-4511-b406-14050205a7bf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145082025-04-30T06:26:58.157ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
145092025-04-30T06:26:58.160ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d44d9fb-49ae-4301-ab76-46da31361b6d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145102025-04-30T06:26:58.169ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
145112025-04-30T06:26:58.172ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9b64cc5-1a29-48dd-a5df-a5e259dd6cab", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145122025-04-30T06:26:58.172ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
145132025-04-30T06:26:58.174ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b4df0f9-d556-46ba-a282-02f4f94eaf32", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145142025-04-30T06:26:58.174ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
145152025-04-30T06:26:58.175ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
145162025-04-30T06:26:58.175ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
145172025-04-30T06:26:58.175ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
145182025-04-30T06:26:58.176ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
145192025-04-30T06:26:58.177ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b87d41ac-42ce-4b8e-9264-c48aa95399e6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145202025-04-30T06:26:58.177ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
145212025-04-30T06:26:58.178ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "564f819d-eccf-4155-9c0e-5555daeeefb7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145222025-04-30T06:26:58.178ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
145232025-04-30T06:26:58.179ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfb60110-f655-4806-bf48-135b71143f1e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145242025-04-30T06:26:58.179ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
145252025-04-30T06:26:58.180ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
145262025-04-30T06:26:58.180ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
145272025-04-30T06:26:58.182ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6feb3fe7-5b46-4717-90d0-ae4744b00264", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145282025-04-30T06:26:58.182ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
145292025-04-30T06:26:58.183ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
145302025-04-30T06:26:58.184ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
145312025-04-30T06:26:58.184ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b01e0d-bd21-48ad-86f6-361ad2a8ad94", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145322025-04-30T06:26:58.184ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
145332025-04-30T06:26:58.188ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b393e17-4a93-46e9-bcbe-a627cbfeb1d1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145342025-04-30T06:26:58.188ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
145352025-04-30T06:26:58.188ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b7bdd55-3a2c-4cd9-9648-359aa231d892", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145362025-04-30T06:26:58.188ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
145372025-04-30T06:26:58.190ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db2be878-e5ce-45c1-9536-1e305b22c4b6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145382025-04-30T06:26:58.191ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
145392025-04-30T06:26:58.192ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e97b9a0-5d8c-4e97-b63e-f0d7393d5a53", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145402025-04-30T06:26:58.192ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
145412025-04-30T06:26:58.195ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a2e90c-672f-44a8-beb4-14f469fa980b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145422025-04-30T06:26:58.195ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
145432025-04-30T06:26:58.196ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
145442025-04-30T06:26:58.196ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145452025-04-30T06:26:58.196ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145462025-04-30T06:26:58.196ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
145472025-04-30T06:26:58.198ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145482025-04-30T06:26:58.200ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a6535c2-849b-4b37-b507-66445e447a87", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145492025-04-30T06:26:58.200ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
145502025-04-30T06:26:58.201ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145512025-04-30T06:26:58.207ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
145522025-04-30T06:26:58.207ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145532025-04-30T06:26:58.207ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
145542025-04-30T06:26:58.208ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145552025-04-30T06:26:58.209ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ad6f6f1-c553-451f-8a27-035043a75eaf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145562025-04-30T06:26:58.209ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
145572025-04-30T06:26:58.211ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e569aca0-cd7d-4102-802a-416e71672373", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145582025-04-30T06:26:58.212ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
145592025-04-30T06:26:58.212ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145602025-04-30T06:26:58.212ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
145612025-04-30T06:26:58.214ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145622025-04-30T06:26:58.215ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08af7fd9-828e-4b2d-92c4-af09d28c6c63", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145632025-04-30T06:26:58.216ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
145642025-04-30T06:26:58.218ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d926ac9a-8bf2-4612-b20b-9c2813bd1518", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145652025-04-30T06:26:58.220ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
145662025-04-30T06:26:58.221ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcc811b7-6b80-42ea-8e6d-0e51a408fb1f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145672025-04-30T06:26:58.222ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
145682025-04-30T06:26:58.224ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e19e6567-de63-46ae-9c94-14d30938e719", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145692025-04-30T06:26:58.224ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
145702025-04-30T06:26:58.228ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5298946-354a-4d84-b1b7-8a4b201e9a18", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145712025-04-30T06:26:58.229ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
145722025-04-30T06:26:58.231ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38853cf8-a5b4-4b74-acaf-abff97bd5046", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145732025-04-30T06:26:58.231ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
145742025-04-30T06:26:58.235ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e22bf50-9639-4592-b0d9-ac90f2f722a8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145752025-04-30T06:26:58.235ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
145762025-04-30T06:26:58.238ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6239237-781e-4611-a1b7-2fee1632367b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145772025-04-30T06:26:58.238ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
145782025-04-30T06:26:58.240ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "579a8b17-de3a-4e59-a680-14365c78a820", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145792025-04-30T06:26:58.245ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
145802025-04-30T06:26:58.247ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7678c2d2-1b20-4af7-a48d-044a3b1ffef9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145812025-04-30T06:26:58.247ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
145822025-04-30T06:26:58.247ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
145832025-04-30T06:26:58.247ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13a1bbda-6b9e-4890-a57b-2ce16d14046a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145842025-04-30T06:26:58.247ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
145852025-04-30T06:26:58.249ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28012509-5290-474b-9250-eed94b4fbbe6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145862025-04-30T06:26:58.249ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
145872025-04-30T06:26:58.251ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "798d9423-8b39-44fd-98c3-43d7819c73a3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
145882025-04-30T06:26:58.251ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145892025-04-30T06:26:58.251ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145902025-04-30T06:26:58.251ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145912025-04-30T06:26:58.251ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145922025-04-30T06:26:58.253ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145932025-04-30T06:26:58.256ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145942025-04-30T06:26:58.260ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145952025-04-30T06:26:58.261ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145962025-04-30T06:26:58.261ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145972025-04-30T06:26:58.261ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145982025-04-30T06:26:58.261ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145992025-04-30T06:26:58.262ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146002025-04-30T06:26:58.265ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146012025-04-30T06:26:58.269ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
146022025-04-30T06:26:58.269ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146032025-04-30T06:26:58.269ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146042025-04-30T06:26:58.270ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146052025-04-30T06:26:58.329ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
146062025-04-30T06:26:58.330ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
146072025-04-30T06:26:58.331ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
146082025-04-30T06:26:58.331ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146092025-04-30T06:26:58.331ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146102025-04-30T06:26:58.331ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
146112025-04-30T06:26:58.333ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146122025-04-30T06:26:58.333ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:51647 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
146132025-04-30T06:26:58.333ZWARNtest_disk_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
146142025-04-30T06:26:58.334ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:51647 req_id = 5d300b7a-cf89-4531-a6c2-b3ba54484f32 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
146152025-04-30T06:26:58.335ZINFOtest_disk_metrics (dropshot): request completed latency_us = 726 local_addr = [::1]:50989 method = GET remote_addr = [::1]:51647 req_id = 5d300b7a-cf89-4531-a6c2-b3ba54484f32 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
146162025-04-30T06:26:58.336ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d300b7a-cf89-4531-a6c2-b3ba54484f32", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:57 GMT"} })
146172025-04-30T06:26:58.339ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
146182025-04-30T06:26:58.339ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
146192025-04-30T06:26:58.339ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
146202025-04-30T06:26:58.341ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
146212025-04-30T06:26:58.341ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146222025-04-30T06:26:58.341ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146232025-04-30T06:26:58.341ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146242025-04-30T06:26:58.342ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146252025-04-30T06:26:58.342ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146262025-04-30T06:26:58.351ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
146272025-04-30T06:26:58.351ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146282025-04-30T06:26:58.351ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
146292025-04-30T06:26:58.351ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146302025-04-30T06:26:58.353ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146312025-04-30T06:26:58.357ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
146322025-04-30T06:26:58.357ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146332025-04-30T06:26:58.357ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146342025-04-30T06:26:58.358ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146352025-04-30T06:26:58.358ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146362025-04-30T06:26:58.358ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
146372025-04-30T06:26:58.359ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146382025-04-30T06:26:58.391ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
146392025-04-30T06:26:58.392ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
146402025-04-30T06:26:58.392ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 446.767786ms iteration = 54 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
146412025-04-30T06:26:58.393ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 55 reason = Signaled
146422025-04-30T06:26:58.393ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
146432025-04-30T06:26:58.393ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146442025-04-30T06:26:58.393ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146452025-04-30T06:26:58.393ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146462025-04-30T06:26:58.394ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146472025-04-30T06:26:58.398ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
146482025-04-30T06:26:58.399ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146492025-04-30T06:26:58.402ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
146502025-04-30T06:26:58.402ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146512025-04-30T06:26:58.402ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146522025-04-30T06:26:58.403ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146532025-04-30T06:26:58.406ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
146542025-04-30T06:26:58.406ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
146552025-04-30T06:26:58.407ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 67 fb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 g............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52468 req_id = 7e77c480-335c-4b2b-b3c0-eaf8e7033766
146562025-04-30T06:26:58.407ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26619,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52468 req_id = 7e77c480-335c-4b2b-b3c0-eaf8e7033766
146572025-04-30T06:26:58.408ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
146582025-04-30T06:26:58.408ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
146592025-04-30T06:26:58.408ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52468 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7e77c480-335c-4b2b-b3c0-eaf8e7033766
146602025-04-30T06:26:58.409ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3283, tv_nsec: 208338097 } })
146612025-04-30T06:26:58.409ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
146622025-04-30T06:26:58.409ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 01 c6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52600 req_id = 1a9d0659-e292-4e89-8c46-98ae17cbd966
146632025-04-30T06:26:58.410ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 454,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52600 req_id = 1a9d0659-e292-4e89-8c46-98ae17cbd966
146642025-04-30T06:26:58.410ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
146652025-04-30T06:26:58.410ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52600 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1a9d0659-e292-4e89-8c46-98ae17cbd966
146662025-04-30T06:26:58.467ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
146672025-04-30T06:26:58.468ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
146682025-04-30T06:26:58.468ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146692025-04-30T06:26:58.468ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146702025-04-30T06:26:58.468ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
146712025-04-30T06:26:58.469ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146722025-04-30T06:26:58.469ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7d 07 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c }............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61510 req_id = bd19889f-c81d-4d6d-8b6d-37c4a22a83cc
146732025-04-30T06:26:58.470ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32007,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61510 req_id = bd19889f-c81d-4d6d-8b6d-37c4a22a83cc
146742025-04-30T06:26:58.470ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
146752025-04-30T06:26:58.470ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32007, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61510 req_id = bd19889f-c81d-4d6d-8b6d-37c4a22a83cc
146762025-04-30T06:26:58.470ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
146772025-04-30T06:26:58.470ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
146782025-04-30T06:26:58.470ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
146792025-04-30T06:26:58.471ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146802025-04-30T06:26:58.476ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
146812025-04-30T06:26:58.476ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146822025-04-30T06:26:58.476ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
146832025-04-30T06:26:58.477ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146842025-04-30T06:26:58.478ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "db44c3d7-12c6-447d-bb71-3d81413f9029", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
146852025-04-30T06:26:58.478ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
146862025-04-30T06:26:58.479ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e9dd4a8-1b9d-4de4-b140-3b6701d49137", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
146872025-04-30T06:26:58.479ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
146882025-04-30T06:26:58.479ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146892025-04-30T06:26:58.479ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
146902025-04-30T06:26:58.480ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146912025-04-30T06:26:58.482ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09d9f91a-cc1c-4921-b870-aa534f89fe7a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
146922025-04-30T06:26:58.482ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
146932025-04-30T06:26:58.485ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f43ee9c-d2e4-42f5-b30e-71e651b44ca6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
146942025-04-30T06:26:58.485ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
146952025-04-30T06:26:58.488ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3222b357-a91f-4ab7-ba3e-1766c23c7815", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
146962025-04-30T06:26:58.488ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
146972025-04-30T06:26:58.491ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44c4e76d-c3fa-43d3-93a2-e449c74c242d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
146982025-04-30T06:26:58.491ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
146992025-04-30T06:26:58.494ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38c238a9-addc-4250-98fe-ebc5c8c5fef6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147002025-04-30T06:26:58.495ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
147012025-04-30T06:26:58.509ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9f82c1d-e350-463a-84b9-7ec1aaa6f646", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147022025-04-30T06:26:58.509ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
147032025-04-30T06:26:58.512ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a49f0efa-24bb-4c63-9c3d-317d50ebe609", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147042025-04-30T06:26:58.512ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
147052025-04-30T06:26:58.517ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6e9885f-efe4-4c74-bf78-0092810eb404", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147062025-04-30T06:26:58.517ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
147072025-04-30T06:26:58.519ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "804cb24f-449e-448a-8930-c28c0019034c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147082025-04-30T06:26:58.519ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
147092025-04-30T06:26:58.551ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
147102025-04-30T06:26:58.552ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1ffd0ac-bdb2-4134-82ea-d718e9ac5e9d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147112025-04-30T06:26:58.552ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
147122025-04-30T06:26:58.560ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81265cd5-bd26-4133-957c-a783f7d906e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147132025-04-30T06:26:58.560ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
147142025-04-30T06:26:58.565ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5d400c6-2f09-4ab6-99eb-b1be705969b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147152025-04-30T06:26:58.565ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
147162025-04-30T06:26:58.584ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24798733-3ddd-4266-9a22-4b2bb272b769", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147172025-04-30T06:26:58.585ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
147182025-04-30T06:26:58.588ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f0b62a-37e7-4d99-a7be-085a1a4971cc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147192025-04-30T06:26:58.588ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
147202025-04-30T06:26:58.592ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69d64713-b35b-44be-819a-c6ebff4f0f27", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147212025-04-30T06:26:58.593ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
147222025-04-30T06:26:58.614ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
147232025-04-30T06:26:58.614ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147242025-04-30T06:26:58.614ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147252025-04-30T06:26:58.614ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
147262025-04-30T06:26:58.615ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147272025-04-30T06:26:58.619ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147282025-04-30T06:26:58.628ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
147292025-04-30T06:26:58.628ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147302025-04-30T06:26:58.628ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
147312025-04-30T06:26:58.628ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147322025-04-30T06:26:58.629ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c0e03fb-6b86-4ac9-8506-864e5b86a6cb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147332025-04-30T06:26:58.629ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
147342025-04-30T06:26:58.632ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147352025-04-30T06:26:58.632ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
147362025-04-30T06:26:58.632ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147372025-04-30T06:26:58.634ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c8bff9f-5d30-4a8d-92ec-5cf37fdd4126", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147382025-04-30T06:26:58.634ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
147392025-04-30T06:26:58.638ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "875f696a-42c4-4743-84f5-4b860e34392b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147402025-04-30T06:26:58.638ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
147412025-04-30T06:26:58.641ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69fc754e-3089-4149-8bea-be32fd1a9ac2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147422025-04-30T06:26:58.641ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
147432025-04-30T06:26:58.645ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee8ef09f-d213-4644-b338-d38fdb97fdbe", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147442025-04-30T06:26:58.645ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
147452025-04-30T06:26:58.647ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e742148-a896-42f7-803e-55586f6f497b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147462025-04-30T06:26:58.647ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
147472025-04-30T06:26:58.649ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da4d71f3-6970-4fe6-8b34-203ae0ab9d9e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147482025-04-30T06:26:58.649ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
147492025-04-30T06:26:58.652ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d3a8c26-81f1-48d7-8df5-7093a2c25f2f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147502025-04-30T06:26:58.652ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1
147512025-04-30T06:26:58.655ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7690260c-b9a6-4b5c-b1b4-bbe3ae370928", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147522025-04-30T06:26:58.655ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0
147532025-04-30T06:26:58.657ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c18835e8-8638-41e5-9049-0eafcc9d6f7d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147542025-04-30T06:26:58.657ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1
147552025-04-30T06:26:58.660ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "131a43f5-d2aa-43f7-9765-bdc30ac5e64f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147562025-04-30T06:26:58.660ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0
147572025-04-30T06:26:58.662ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bfa1055-8fb0-44ae-a4db-0405292efa93", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147582025-04-30T06:26:58.662ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1
147592025-04-30T06:26:58.672ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
147602025-04-30T06:26:58.673ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "775619b9-2a83-4577-9d0e-9be060189d0d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147612025-04-30T06:26:58.673ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cmpa
147622025-04-30T06:26:58.674ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1be50ef5-4284-4d90-95d3-bceaa095d6a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147632025-04-30T06:26:58.674ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
147642025-04-30T06:26:58.676ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fe709d3-c98c-4112-9b99-15592cd89b28", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147652025-04-30T06:26:58.676ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
147662025-04-30T06:26:58.677ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ffd563b-df5a-487d-a700-22e2c78afe41", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147672025-04-30T06:26:58.677ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/rot/cfpa
147682025-04-30T06:26:58.679ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39c259c6-bd56-4657-9ace-d7f8350c7563", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147692025-04-30T06:26:58.679ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1
147702025-04-30T06:26:58.681ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f4cfd72-5d46-495d-9399-d4be94f53c97", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147712025-04-30T06:26:58.681ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0
147722025-04-30T06:26:58.683ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e168f65-8173-48f1-bd39-3f6ef67bfaca", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147732025-04-30T06:26:58.683ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1
147742025-04-30T06:26:58.687ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "324af6ee-f260-4e35-bfd7-658af3753d05", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147752025-04-30T06:26:58.687ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0
147762025-04-30T06:26:58.689ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b30d079b-512b-463d-ab83-d99aba02bd66", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147772025-04-30T06:26:58.689ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1
147782025-04-30T06:26:58.691ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "741ff50b-d2c8-485d-8503-48201f7415b5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147792025-04-30T06:26:58.691ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0
147802025-04-30T06:26:58.693ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db228f27-157c-4cd8-bf27-ddf815d5e133", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147812025-04-30T06:26:58.694ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1
147822025-04-30T06:26:58.699ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c556b232-1ad0-465b-af22-0ae3f8f7fcf2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147832025-04-30T06:26:58.699ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cmpa
147842025-04-30T06:26:58.700ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a12a684-4ab6-4199-95b0-ee2581728036", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147852025-04-30T06:26:58.701ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
147862025-04-30T06:26:58.702ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5779bed6-ffff-40b9-b99f-d295577b4bd5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147872025-04-30T06:26:58.702ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
147882025-04-30T06:26:58.703ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b63141a-1df1-4dc9-9ddc-72284165c0ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147892025-04-30T06:26:58.703ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/1/component/rot/cfpa
147902025-04-30T06:26:58.704ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65a6e3c0-3711-4156-919a-a1eb1a2460c6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
147912025-04-30T06:26:58.704ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
147922025-04-30T06:26:58.705ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147932025-04-30T06:26:58.705ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147942025-04-30T06:26:58.705ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147952025-04-30T06:26:58.706ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147962025-04-30T06:26:58.716ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147972025-04-30T06:26:58.721ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
147982025-04-30T06:26:58.721ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
147992025-04-30T06:26:58.721ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148002025-04-30T06:26:58.721ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148012025-04-30T06:26:58.721ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148022025-04-30T06:26:58.722ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148032025-04-30T06:26:58.723ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
148042025-04-30T06:26:58.723ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148052025-04-30T06:26:58.724ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148062025-04-30T06:26:58.724ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
148072025-04-30T06:26:58.725ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148082025-04-30T06:26:58.727ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148092025-04-30T06:26:58.734ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
148102025-04-30T06:26:58.734ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148112025-04-30T06:26:58.734ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148122025-04-30T06:26:58.736ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148132025-04-30T06:26:58.737ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148142025-04-30T06:26:58.745ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
148152025-04-30T06:26:58.745ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148162025-04-30T06:26:58.745ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
148172025-04-30T06:26:58.745ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148182025-04-30T06:26:58.762ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148192025-04-30T06:26:58.762ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
148202025-04-30T06:26:58.763ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148212025-04-30T06:26:58.826ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:50989
148222025-04-30T06:26:58.826ZDEBGtest_disk_metrics (BackgroundTasks): client request SledAgent = http://[::1]:50989 background_task = inventory_collection body = None method = GET uri = http://[::1]:50989/inventory
148232025-04-30T06:26:58.826ZINFOtest_disk_metrics (dropshot): accepted connection local_addr = [::1]:50989 remote_addr = [::1]:50363 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
148242025-04-30T06:26:58.827ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
148252025-04-30T06:26:58.830ZTRACtest_disk_metrics (dropshot): incoming request local_addr = [::1]:50989 method = GET remote_addr = [::1]:50363 req_id = 7313b190-8931-43bb-a065-317baf5782cf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
148262025-04-30T06:26:58.831ZINFOtest_disk_metrics (dropshot): request completed latency_us = 609 local_addr = [::1]:50989 method = GET remote_addr = [::1]:50363 req_id = 7313b190-8931-43bb-a065-317baf5782cf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
148272025-04-30T06:26:58.831ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 2e a7 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56262 req_id = 42612e34-1d91-4181-8dd3-f6699bcd3ce1
148282025-04-30T06:26:58.831ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11943,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56262 req_id = 42612e34-1d91-4181-8dd3-f6699bcd3ce1
148292025-04-30T06:26:58.831ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
148302025-04-30T06:26:58.831ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148312025-04-30T06:26:58.831ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56262 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 42612e34-1d91-4181-8dd3-f6699bcd3ce1
148322025-04-30T06:26:58.831ZDEBGtest_disk_metrics (BackgroundTasks): client response SledAgent = http://[::1]:50989 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:50989/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7313b190-8931-43bb-a065-317baf5782cf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
148332025-04-30T06:26:58.834ZDEBGtest_disk_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
148342025-04-30T06:26:58.834ZDEBGtest_disk_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
148352025-04-30T06:26:58.834ZDEBGtest_disk_metrics (BackgroundTasks): finished collection background_task = inventory_collection
148362025-04-30T06:26:58.836ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
148372025-04-30T06:26:58.836ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148382025-04-30T06:26:58.836ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148392025-04-30T06:26:58.836ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148402025-04-30T06:26:58.837ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148412025-04-30T06:26:58.839ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3283, tv_nsec: 638042865 } })
148422025-04-30T06:26:58.839ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
148432025-04-30T06:26:58.840ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 22 c1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e "...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60410 req_id = e47360be-b3dc-487f-ac10-70b45494e188
148442025-04-30T06:26:58.840ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8897,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60410 req_id = e47360be-b3dc-487f-ac10-70b45494e188
148452025-04-30T06:26:58.841ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148462025-04-30T06:26:58.841ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60410 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e47360be-b3dc-487f-ac10-70b45494e188
148472025-04-30T06:26:58.841ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148482025-04-30T06:26:58.845ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
148492025-04-30T06:26:58.845ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148502025-04-30T06:26:58.845ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148512025-04-30T06:26:58.845ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148522025-04-30T06:26:58.846ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
148532025-04-30T06:26:58.897ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
148542025-04-30T06:26:58.897ZDEBGtest_disk_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
148552025-04-30T06:26:58.898ZDEBGtest_disk_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
148562025-04-30T06:26:58.898ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
148572025-04-30T06:26:58.902ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
148582025-04-30T06:26:58.902ZINFOtest_disk_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
148592025-04-30T06:26:58.906ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148602025-04-30T06:26:58.909ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 8c8ddb3c-8c0a-41ea-a2bc-1faaf5d53e96 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148612025-04-30T06:26:58.909ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 8c8ddb3c-8c0a-41ea-a2bc-1faaf5d53e96 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148622025-04-30T06:26:58.909ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 8c8ddb3c-8c0a-41ea-a2bc-1faaf5d53e96 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148632025-04-30T06:26:58.911ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 8c8ddb3c-8c0a-41ea-a2bc-1faaf5d53e96 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148642025-04-30T06:26:58.915ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 7599 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 8c8ddb3c-8c0a-41ea-a2bc-1faaf5d53e96 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148652025-04-30T06:26:58.917ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c8ddb3c-8c0a-41ea-a2bc-1faaf5d53e96", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
148662025-04-30T06:26:58.918ZDEBGtest_disk_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148672025-04-30T06:26:58.919ZTRACtest_disk_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 797a179c-8770-4eb0-ac07-9ae45af719ba uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148682025-04-30T06:26:58.919ZTRACtest_disk_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 797a179c-8770-4eb0-ac07-9ae45af719ba resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148692025-04-30T06:26:58.919ZDEBGtest_disk_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 797a179c-8770-4eb0-ac07-9ae45af719ba roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148702025-04-30T06:26:58.920ZDEBGtest_disk_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 797a179c-8770-4eb0-ac07-9ae45af719ba resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148712025-04-30T06:26:58.923ZINFOtest_disk_metrics (dropshot_internal): request completed latency_us = 3819 local_addr = 127.0.0.1:39551 method = GET remote_addr = 127.0.0.1:49839 req_id = 797a179c-8770-4eb0-ac07-9ae45af719ba response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148722025-04-30T06:26:58.923ZDEBGtest_disk_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:39551/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "797a179c-8770-4eb0-ac07-9ae45af719ba", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
148732025-04-30T06:26:58.923ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 06ed8ede-6e59-42fb-a0cc-51ac048d6bbe
148742025-04-30T06:26:58.923ZTRACtest_disk_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
148752025-04-30T06:26:58.923ZINFOtest_disk_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
148762025-04-30T06:26:58.934ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
148772025-04-30T06:26:58.940ZWARNtest_disk_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
148782025-04-30T06:26:58.940ZWARNtest_disk_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
148792025-04-30T06:26:58.940ZDEBGtest_disk_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 547.803131ms iteration = 55 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
148802025-04-30T06:26:58.940ZDEBGtest_disk_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 56 reason = Signaled
148812025-04-30T06:26:58.940ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
148822025-04-30T06:26:58.940ZTRACtest_disk_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148832025-04-30T06:26:58.940ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148842025-04-30T06:26:58.940ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148852025-04-30T06:26:58.941ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148862025-04-30T06:26:58.948ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148872025-04-30T06:26:58.952ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
148882025-04-30T06:26:58.952ZTRACtest_disk_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148892025-04-30T06:26:58.952ZDEBGtest_disk_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148902025-04-30T06:26:58.953ZDEBGtest_disk_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148912025-04-30T06:26:58.958ZDEBGtest_disk_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(80cb6c5e-8eff-4092-8c0b-d0e2fead80c7, 0)]
148922025-04-30T06:26:58.958ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
148932025-04-30T06:26:58.975ZTRACtest_disk_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 30 ac 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 0............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60949 req_id = c8fe210d-e168-4ea8-b5df-22180a1a2c72
148942025-04-30T06:26:58.975ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12460,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60949 req_id = c8fe210d-e168-4ea8-b5df-22180a1a2c72
148952025-04-30T06:26:58.976ZDEBGtest_disk_metrics (store): query key key = _mgs._tcp
148962025-04-30T06:26:58.976ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148972025-04-30T06:26:58.976ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60949 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c8fe210d-e168-4ea8-b5df-22180a1a2c72
148982025-04-30T06:26:58.978ZDEBGtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 39213, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3283, tv_nsec: 777450976 } })
148992025-04-30T06:26:58.978ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
149002025-04-30T06:26:58.986ZTRACtest_disk_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
149012025-04-30T06:26:58.986ZTRACtest_disk_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
149022025-04-30T06:26:58.986ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149032025-04-30T06:26:58.986ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
149042025-04-30T06:26:58.987ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149052025-04-30T06:26:58.988ZDEBGtest_disk_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149062025-04-30T06:26:58.988ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 60 96 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e `...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54062 req_id = db68034e-b314-454b-a33a-e03091cd6adb
149072025-04-30T06:26:58.989ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24726,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54062 req_id = db68034e-b314-454b-a33a-e03091cd6adb
149082025-04-30T06:26:58.989ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
149092025-04-30T06:26:58.989ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54062 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = db68034e-b314-454b-a33a-e03091cd6adb
149102025-04-30T06:26:58.989ZDEBGtest_disk_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149112025-04-30T06:26:59.053ZTRACtest_disk_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
149122025-04-30T06:26:59.060ZTRACtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
149132025-04-30T06:26:59.061ZTRACtest_disk_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149142025-04-30T06:26:59.062ZDEBGtest_disk_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149152025-04-30T06:26:59.062ZTRACtest_disk_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149162025-04-30T06:26:59.062ZTRACtest_disk_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149172025-04-30T06:26:59.063ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
149182025-04-30T06:26:59.068ZDEBGtest_disk_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
149192025-04-30T06:26:59.068ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149202025-04-30T06:26:59.069ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
149212025-04-30T06:26:59.069ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149222025-04-30T06:26:59.069ZTRACtest_disk_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1c a5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55166 req_id = fb905f99-39b5-4233-945e-f0cb707968d4
149232025-04-30T06:26:59.069ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7333,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55166 req_id = fb905f99-39b5-4233-945e-f0cb707968d4
149242025-04-30T06:26:59.070ZDEBGtest_disk_metrics (store): query key key = _clickhouse-admin-keeper._tcp
149252025-04-30T06:26:59.070ZERROtest_disk_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7333, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55166 req_id = fb905f99-39b5-4233-945e-f0cb707968d4
149262025-04-30T06:26:59.070ZDEBGtest_disk_metrics (BackgroundTasks): begin collection background_task = inventory_collection
149272025-04-30T06:26:59.071ZDEBGtest_disk_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:39213
149282025-04-30T06:26:59.071ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/ignition
149292025-04-30T06:26:59.071ZTRACtest_disk_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: a3 5d 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .]..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50155 req_id = 4e27dfad-a5b9-4ab1-ae20-905885c38b63
149302025-04-30T06:26:59.071ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41821,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50155 req_id = 4e27dfad-a5b9-4ab1-ae20-905885c38b63
149312025-04-30T06:26:59.071ZDEBGtest_disk_metrics (store): query key key = _dendrite._tcp
149322025-04-30T06:26:59.071ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
149332025-04-30T06:26:59.071ZDEBGtest_disk_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50155 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4e27dfad-a5b9-4ab1-ae20-905885c38b63
149342025-04-30T06:26:59.071ZDEBGtest_disk_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40605, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3283, tv_nsec: 870631132 } })
149352025-04-30T06:26:59.071ZTRACtest_disk_metrics (DnsResolver): lookup_service_targets: looking up SRV target
149362025-04-30T06:26:59.072ZTRACtest_disk_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149372025-04-30T06:26:59.072ZDEBGtest_disk_metrics (InternalApi): roles roles = RoleSet { roles: {} }
149382025-04-30T06:26:59.073ZDEBGtest_disk_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149392025-04-30T06:26:59.073ZTRACtest_disk_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bf 38 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .8..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59038 req_id = 0e8386bd-cc50-4fb0-b9a8-409ea6eb673c
149402025-04-30T06:26:59.073ZDEBGtest_disk_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48952,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59038 req_id = 0e8386bd-cc50-4fb0-b9a8-409ea6eb673c
149412025-04-30T06:26:59.073ZDEBGtest_disk_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
149422025-04-30T06:26:59.073ZDEBGtest_disk_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59038 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0e8386bd-cc50-4fb0-b9a8-409ea6eb673c
149432025-04-30T06:26:59.074ZINFOtest_disk_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
149442025-04-30T06:26:59.135ZINFOtest_disk_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
149452025-04-30T06:26:59.136ZDEBGtest_disk_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
149462025-04-30T06:26:59.136ZDEBGtest_disk_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
149472025-04-30T06:26:59.136ZWARNtest_disk_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
149482025-04-30T06:26:59.138ZWARNtest_disk_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
149492025-04-30T06:26:59.139ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b92f62-4d1a-459c-b09c-42a752c6e7a3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149502025-04-30T06:26:59.139ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0
149512025-04-30T06:26:59.142ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d12bbd8f-dda2-4d08-a438-e1b12613d9f7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149522025-04-30T06:26:59.142ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0
149532025-04-30T06:26:59.144ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe73273-9fda-4f25-9715-b3780ac2e1da", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149542025-04-30T06:26:59.145ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1
149552025-04-30T06:26:59.148ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2ecd058-10f1-4642-acca-80f4c27bf93f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149562025-04-30T06:26:59.148ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0
149572025-04-30T06:26:59.157ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37900aea-bbad-4ca8-8519-80dd4c27f391", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149582025-04-30T06:26:59.157ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1
149592025-04-30T06:26:59.170ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6a4ff43-8a11-4b88-82d8-e65a2e85fc55", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149602025-04-30T06:26:59.171ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0
149612025-04-30T06:26:59.173ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb382e87-101b-4f4f-9c7d-67a35283a518", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149622025-04-30T06:26:59.173ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1
149632025-04-30T06:26:59.186ZDEBGtest_disk_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
149642025-04-30T06:26:59.186ZDEBGtest_disk_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
149652025-04-30T06:26:59.187ZDEBGtest_disk_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25
149662025-04-30T06:26:59.188ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
149672025-04-30T06:26:59.191ZDEBGtest_disk_metrics: attempt to fetch latest inventory collection result = Ok("found")
149682025-04-30T06:26:59.193ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 2 table_name = fields_string
149692025-04-30T06:26:59.193ZTRACtest_disk_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
149702025-04-30T06:26:59.194ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bbd7d9d-4ce6-479a-8416-7b9b936d5c60", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149712025-04-30T06:26:59.194ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cmpa
149722025-04-30T06:26:59.198ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "785a80a5-427d-4253-8c8b-ec8b01d6129a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149732025-04-30T06:26:59.198ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
149742025-04-30T06:26:59.199ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6072ec71-3bbc-4b45-a557-25b6da7652a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149752025-04-30T06:26:59.200ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
149762025-04-30T06:26:59.201ZDEBGtest_disk_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2c15bcd7-e6d9-4677-9d01-d955902bdb25 n_rows = 1 table_name = measurements_i64
149772025-04-30T06:26:59.201ZTRACtest_disk_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
149782025-04-30T06:26:59.201ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07875b85-270a-4c3e-ba01-7a6e4d2044a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149792025-04-30T06:26:59.201ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/0/component/rot/cfpa
149802025-04-30T06:26:59.203ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "435398e4-cd31-4599-8024-ebdf5a5191cf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149812025-04-30T06:26:59.203ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1
149822025-04-30T06:26:59.207ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcd39209-21dc-4798-a490-7d23787483c4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149832025-04-30T06:26:59.208ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0
149842025-04-30T06:26:59.214ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "361def29-62cc-4fa0-bfca-5c265ee1b990", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149852025-04-30T06:26:59.214ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1
149862025-04-30T06:26:59.217ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58973d6c-df25-4b06-8a66-ded7f9800731", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149872025-04-30T06:26:59.217ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0
149882025-04-30T06:26:59.220ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3699ddf7-3898-4c41-8c6b-9c841c4b5eab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149892025-04-30T06:26:59.220ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1
149902025-04-30T06:26:59.222ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c57797e3-33f7-41b8-b4d4-6cc1834b4706", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149912025-04-30T06:26:59.222ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0
149922025-04-30T06:26:59.224ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65e25bbd-91e3-4581-9fa8-4911a547468c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149932025-04-30T06:26:59.224ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1
149942025-04-30T06:26:59.227ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2712f65c-19dd-47be-88e2-16b97e2ba7ba", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149952025-04-30T06:26:59.227ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cmpa
149962025-04-30T06:26:59.228ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b82df07c-75b3-4535-b1cc-4c01ae64d9b3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149972025-04-30T06:26:59.229ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
149982025-04-30T06:26:59.231ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38862a03-c0ab-4466-9079-397da88866aa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
149992025-04-30T06:26:59.231ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
150002025-04-30T06:26:59.233ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf13e11f-2c40-4d57-8c54-8def9a92fef6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
150012025-04-30T06:26:59.234ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/switch/1/component/rot/cfpa
150022025-04-30T06:26:59.236ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9524293f-8c1c-4194-a24c-5213dd75c471", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
150032025-04-30T06:26:59.236ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0
150042025-04-30T06:26:59.238ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56ae91aa-e5d5-4f85-9597-4b3e3db09005", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
150052025-04-30T06:26:59.239ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0
150062025-04-30T06:26:59.242ZDEBGtest_disk_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:39213 result = Ok(Response { url: "http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f609d6ae-356b-477f-8af7-0bc61bd8416f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:58 GMT"} })
150072025-04-30T06:26:59.242ZDEBGtest_disk_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:39213 method = GET uri = http://[::1]:39213/sp/sled/0/component/sp/caboose?firmware_slot=1